Technical element analysis method, learning method, technical element analysis system, and program

Abstract

A computer 30 receives, for a limited document group 4B as a document group of a specific technical field, a setting operation for an annotation in which a main intrinsic representation tag T1 and a sub intrinsic representation tag T2 are associated with each other, generates annotation data 5 (training data), finely tunes a pre-learning model 6, which is learned in advance by using a large document group 3, according to the annotation data 5, and generates an intrinsic representation extraction model 8 for extracting information about intrinsic representations from documents. Furthermore, the computer 30 receives a target document 2 as an analysis target and extracts information about intrinsic representations from the target document 2 by using the specific representation extraction model 8.

Description

[Technical field]

[0001] The present invention relates to a method for analyzing technical elements of technical documents using machine learning. [Background technology]

[0002] Extracting the necessary technical elements from a huge collection of technical documents is a big challenge. For example, there are many intellectual property-related documents such as patent publications. In recent years, these intellectual property-related documents have been digitized and further classified, making it easier to extract the desired documents by machine search using such classification and free keywords.

[0003] Recently, analysis methods have been developed that use machine learning to organize and output the necessary information from documents. In particular, BERT (Bidirectional Encoder Representations, BERT) is a method that can improve the accuracy of learning by pre-training and even re-pre-training. The development of a technique called "Data mining from Transformers" has attracted attention as a technique that can learn various word placement positions and similar expressions used in sentences and assign appropriate named entity tags to phrases (e.g., Non-Patent Document 1). There has also been a report on PatentBERT, which focuses on patent data as an intellectual property-related document (e.g., Non-Patent Document 2). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Gururangan, S., Marasovic, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey,D., & Smith, NA (2020, July). Don't Stop Pretraining: Adapt LanguageModels to Domains and Tasks. In Proceedings of the 58th Annual Meeting of theAssociation for Computational Linguistics (pp. 8342-8360). [Non-Patent Document 2] Lee, J.-S. and Hsiang, J.: Patentbert: Patent classification with fine-tuning pre-trained bert model, arXiv preprint arXiv:1906.02124 (2019). Summary of the Invention [Problem to be solved by the invention]

[0005] The application of the above-mentioned analysis method is expected to reduce the risk of missing search results due to simple keyword searches, since appropriate named entities can be extracted by machine learning. Furthermore, it is expected that classification of intellectual property-related documents that has been done manually can be automated. In this way, recent machine learning methods seem to be all-purpose for analyzing the technical elements of technical documents. However, there are many cases in which specific expressions are used for each field in technical documents, and there is a need to further improve the accuracy of analysis. In addition, from a cost perspective, it is preferable to keep the time required for machine learning as short as possible, and it is desirable to consider both accuracy and efficiency.

[0006] Furthermore, for researchers, the goal is not achieved simply by extracting related documents, but they must read and understand the documents themselves to understand what technologies are disclosed. Also, those who wish to confirm intellectual property rights must quickly understand the scope of rights disclosed in intellectual property-related documents such as patent documents in order to prevent patent infringement and to define new research areas.

[0007] The present invention has been made in consideration of the above-mentioned problems, and has an object to provide a method for analyzing technical elements of technical documents with improved accuracy and efficiency. [Means for solving the problem]

[0008] A first aspect of the present invention for achieving the above-mentioned object is a computer comprising: Unexamined patent publication, patent gazette, or registered utility model publication Document In a group The system executes an annotation process for receiving an annotation setting operation that associates a master named entity tag with a subordinate named entity tag that is subordinate to the master named entity tag, and generating annotated teacher data, and a named entity extraction model training process for fine-tuning a pre-trained model that has been pre-trained using a large-scale document group with the teacher data, and generating a named entity extraction model that extracts named entity information from documents. A master named entity tag and a subordinate named entity tag are defined for each type of named entity, and a master named entity tag or a subordinate named entity tag can be set for named entities of the same type. the annotation step further includes receiving an operation for setting related named entity tags that serve as clue words for associating the main named entity tags and the subordinate named entity tags, the annotated teacher data being teacher data for generating the named entity extraction model in which the main named entity tags, the subordinate named entity tags, and the related named entity tags are assigned to the document group, the subordinate named entity tags including named entity tags that are set to the upper and lower limits of a numerical range, and the related named entity tags including named entity tags that are set to symbols or phrases that represent a numerical range; This is a learning method characterized by the above. In addition, fine tuning "tuning" is a technique in which a language model is trained using unsupervised data, and then the parameters of the language model are adjusted using supervised data for the specific task to be solved (in this invention, named entity extraction).

[0009] According to the first invention, Unexamined patent publication, patent gazette, or registered utility model publication Document In a groupOn the other hand, by setting annotations for named entity tags with a master-slave relationship and fine-tuning a pre-trained model that has been pre-trained on a large set of documents with annotated training data, it is possible to efficiently and accurately generate a named entity extraction model that is suited to the documents to be analyzed.

[0011] Also, Unexamined patent publication, patent gazette, or registered utility model publication Document The flock and a re-pre-training step of re-pre-training the pre-trained model using the training data to generate a re-pre-trained model, and the named entity extraction model training step preferably generates the named entity extraction model based on the re-pre-trained model and the teacher data. Unexamined patent publication, patent gazette, or registered utility model publication Narrow down to TA sentence By using a re-pre-trained model, which is a pre-trained model re-pre-trained using a collection of documents, as the basis for a named entity extraction model, it is possible to efficiently improve the learning accuracy.

[0012] Also, Unexamined patent publication, patent gazette, or registered utility model publication Document The flock A re-annotation process of accepting a tagging correction operation for the named entity tag that has been input to the named entity extraction model and extracted, and a previous process of setting the corrected named entity tag. record and re-fine-tuning the named entity extraction model using the document set. Since the named entity extraction model is re-trained after correcting the tagging errors of the named entities, the performance of the named entity extraction model is improved.

[0014] The first step to achieve the above objectives 2 The invention relates to a computer that: Unexamined patent publication, patent gazette, or registered utility model publication Document In a groupan annotation process for receiving an annotation setting operation that associates a master named entity tag with a subordinate named entity tag that is subordinate to the master named entity tag, and generating annotated teacher data; and a named entity extraction model training process for fine-tuning a pre-trained model that has been pre-trained using a large-scale document group with the teacher data, and generating a named entity extraction model that extracts named entity information from documents. Unexamined patent publication, patent gazette, or registered utility model publication and extracting named entity information from the document to be analyzed using the named entity extraction model. A main named entity tag and a subordinate named entity tag are defined for each type of named entity, and a main named entity tag or a subordinate named entity tag can be set for a named entity of the same type. the annotation step further includes receiving an operation for setting related named entity tags that serve as clue words for associating the main named entity tags and the subordinate named entity tags, the annotated teacher data being teacher data for generating the named entity extraction model in which the main named entity tags, the subordinate named entity tags, and the related named entity tags are assigned to the document group, the subordinate named entity tags including named entity tags that are set to the upper and lower limits of a numerical range, and the related named entity tags including named entity tags that are set to symbols or phrases that represent a numerical range; The present invention relates to a method for analyzing technical elements of a technical document. In addition, the input means may input a plurality of documents as analysis targets, and the means for extracting information about the named entities may obtain, for each of the plurality of documents, a specific main named entity and an upper limit and a lower limit of a numerical range for the subordinate named entities relative to the main named entity, and display the numerical ranges described in each document for comparison.

[0015] No. 2 According to the invention, Unexamined patent publication, patent gazette, or registered utility model publication Document In a group On the other hand, by annotating named entity tags that have a master-slave relationship and fine-tuning a pre-trained model that has been pre-trained on a large set of documents with annotated training data, it is possible to efficiently and accurately generate a named entity extraction model that is suited to the documents to be analyzed, and to perform document analysis.

[0016] The first step to achieve the above objectives 3 The invention is Unexamined patent publication, patent gazette, or registered utility model publication Document In a group On the other hand, the system includes an annotation means for receiving an annotation setting operation that associates a master named entity tag with a subordinate named entity tag that is subordinate to the master named entity tag, and generating annotated teacher data; a named entity extraction model training means for fine-tuning a pre-trained model that has been pre-trained using a large-scale document group with the teacher data, and generating a named entity extraction model that extracts named entity information from documents; Unexamined patent publication, patent gazette, or registered utility model publicationa means for inputting a document as an analysis target, and a means for extracting named entity information from the analysis target document using the named entity extraction model, wherein a master named entity tag and a slave named entity tag are defined for each type of named entity, and a master named entity tag or a slave named entity tag can be set for a named entity of the same type. the annotation means further accepts a setting operation of related named entity tags which serve as clue words for associating the main named entity tags and the subordinate named entity tags, the annotated teacher data being teacher data for generating the named entity extraction model in which the main named entity tags, the subordinate named entity tags, and the related named entity tags are assigned to the document group, the subordinate named entity tags including named entity tags set to the upper and lower limits of a numerical range, and the related named entity tags including named entity tags set to symbols or phrases representing a numerical range; The present invention relates to a system for analyzing technical elements of technical documents. In addition, the input means may input a plurality of documents as analysis targets, and the means for extracting information about the named entities may obtain, for each of the plurality of documents, a specific main named entity and an upper limit and a lower limit of a numerical range for the subordinate named entities relative to the main named entity, and display the numerical ranges described in each document for comparison.

[0017] No. 3 According to the invention, Unexamined patent publication, patent gazette, or registered utility model publication Document In a group On the other hand, by annotating named entity tags that have a master-slave relationship and fine-tuning a pre-trained model that has been pre-trained on a large set of documents with annotated training data, it is possible to efficiently and accurately generate a named entity extraction model that is suited to the documents to be analyzed, and to perform document analysis.

[0018] The first step to achieve the above objectives 4 The invention relates to a computer, Unexamined patent publication, patent gazette, or registered utility model publication Document In a group In response to this, the system functions as an annotation means that receives an annotation setting operation that associates a master named entity tag with a subordinate named entity tag that is subordinate to the master named entity tag, and generates annotated teacher data, and a named entity extraction model training means that fine-tunes a pre-trained model that has been pre-trained using a large-scale document group with the teacher data, and generates a named entity extraction model, in which master named entity tags and subordinate named entity tags are defined for each type of named entity, and master named entity tags or subordinate named entity tags can be set for named entities of the same type. the annotation means further accepts a setting operation of related named entity tags which serve as clue words for associating the main named entity tags and the subordinate named entity tags, the annotated teacher data being teacher data for generating the named entity extraction model in which the main named entity tags, the subordinate named entity tags, and the related named entity tags are assigned to the document group, the subordinate named entity tags including named entity tags set to the upper and lower limits of a numerical range, and the related named entity tags including named entity tags set to symbols or phrases representing a numerical range; The program is characterized by:

[0019] No. 4 According to the present invention, there is provided a program for causing a computer to execute the learning method of the first invention.

[0020] The first step to achieve the above objectives 5 The invention relates to a computer, Unexamined patent publication, patent gazette, or registered utility model publication Document In a group On the other hand, the present invention provides an annotation means for receiving an annotation setting operation that associates a master named entity tag with a slave named entity tag that is subordinate to the master named entity tag, and generating annotated teacher data; a named entity extraction model training means for fine-tuning a pre-trained model that has been pre-trained using a large-scale document group, with the teacher data, and generating a named entity extraction model that extracts named entity information from documents; Unexamined patent publication, patent gazette, or registered utility model publication a means for inputting a document as an analysis target, and a means for extracting named entity information from the analysis target document using the named entity extraction model, wherein a master named entity tag and a slave named entity tag are defined for each type of named entity, and a master named entity tag or a slave named entity tag can be set for a named entity of the same type. the annotation means further accepts a setting operation of related named entity tags which serve as clue words for associating the main named entity tags and the subordinate named entity tags, the annotated teacher data being teacher data for generating the named entity extraction model in which the main named entity tags, the subordinate named entity tags, and the related named entity tags are assigned to the document group, the subordinate named entity tags including named entity tags set to the upper and lower limits of a numerical range, and the related named entity tags including named entity tags set to symbols or phrases representing a numerical range; The program is characterized by: In addition, the input means may input a plurality of documents as analysis targets, and the means for extracting information about the named entities may obtain, for each of the plurality of documents, a specific main named entity and an upper limit and a lower limit of a numerical range for the subordinate named entities relative to the main named entity, and display the numerical ranges described in each document for comparison.

[0021] No. 5 According to the invention, 2 The present invention provides a program for causing a computer to execute the technical element analysis method of the present invention. Effect of the Invention

[0022] The present invention provides a method for analyzing technical elements of technical documents with improved accuracy and efficiency. [Brief description of the drawings]

[0023] [Figure 1] Diagram showing the learning function of the technology element analysis system 1 [Diagram 2] Diagram showing the learning function of the technology element analysis system 1 [Diagram 3] A diagram showing an example of annotation settings [Figure 4] Diagram showing the analysis functions of the Technology Element Analysis System 1 [Diagram 5] A diagram showing the document extraction DB 100. [Figure 6] FIG. 3 shows the hardware configuration of a computer 30. [Figure 7] Flowchart showing the flow of learning processing [Figure 8] Flowchart showing the flow of learning processing [Figure 9] Flowchart showing the flow of analysis processing [Figure 10] Example of analysis results [Figure 11] Diagram showing the search function of the technology element analysis system 1 [Figure 12] Flowchart showing the flow of search processing [Figure 13] Graph showing named entity extraction performance after re-pre-training [Figure 14] Graph showing named entity extraction performance of main named entity tag (atom) [Figure 15] Graph showing the named entity extraction performance of the main named entity tag (substance) [Figure 16] Graph showing the named entity extraction performance of the main named entity tag (fig_UL) [Figure 17] Graph showing the named entity extraction performance of the main named entity tag (fig_LL) DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

[0025] (1. Learning function) First, the learning function for learning the named entity extraction model 8 used in document analysis will be described with reference to Fig. 1. The learning function consists of "pre-learning", "re-pre-learning", and "actual learning", and the named entity extraction model 8 is generated by carrying out these learnings. Each learning function will be described.

[0026] (1-1. Preliminary learning) The pre-learning is executed by a pre-learning unit 14 included in the technological element analysis system 1. Specifically, as shown in FIG. 1, the pre-learning unit 14 performs unsupervised pre-learning using a large-scale document set 3 to generate a pre-learned model 6.

[0027] The large-scale document group 3 is a database (large-scale general-purpose corpus) in which texts written on the Web, in books, etc. are structured and accumulated on a large scale, and serves as training data for training the pre-training model 6. In this embodiment, a database (large-scale corpus) in which articles on the Japanese version of Wikipedia are structured and accumulated on a large scale is used as the large-scale document group 3.

[0028] The learning of the pre-learning model 6 performed by the pre-learning unit 14 is desirably deep learning based on a neural network. In this case, if a transformer is used as the deep learning model, parallel processing is possible and the learning time can be shortened. Such deep learning can be performed, for example, by using a known natural language processing technology known as BERT. In this embodiment, the pre-learning unit 14 uses the BERT technology to learn the pre-learning model 6 with a large amount of document data obtained from Japanese Wikipedia.

[0029] Since pre-learning is a process that involves a high computational load using a large amount of data, it is desirable to perform the pre-learning in advance. In other words, it is desirable to prepare the pre-learning model 6 in advance and store it in the storage device 32 (FIG. 6) of the computer 30 that executes the analysis process. In this case, the function and step of learning the pre-learning model 6 can be omitted from the present invention.

[0030] (1-2. Re-learning in advance) The re-pre-learning is executed by a re-pre-learning unit 15 included in the technological element analysis system 1. Specifically, as shown in Fig. 1, the re-pre-learning unit 15 re-pre-learns the pre-learning model 6 using a limited document group 4A, which is a document group in a specific technological field, to generate a re-pre-learning model 7.

[0031] The restricted document group 4A is a database (special purpose corpus) that structures and accumulates documents in a specific technical field to be analyzed, and serves as training data for re-pre-training the pre-training model 6. The restricted document group 4A is a group of documents in the same technical field as the target document 2 to be analyzed (see FIG. 4).

[0032] In addition, it is desirable that the restricted document group 4A be intellectual property-related documents. Intellectual property-related documents are documents that publish the contents of patent applications and utility model registration applications, such as published patent gazettes, patent gazettes, registered utility model gazettes, etc. When the restricted document group 4A is an intellectual property-related document, the specific technical field is, for example, a technical field classified by a patent classification symbol consisting of IPC, FI, F-term, etc.

[0033] Similar to pre-training, re-pre-training can be performed using a known technique known as BERT. By re-pre-training the pre-trained model 6 using the restricted document set 4A, a training model (re-pre-trained model 7) suitable for analysis of a specific technical field can be obtained. In addition, the use of the restricted document set 4A is useful for efficiently improving the training accuracy by large-scale pre-training.

[0034] Although it is possible to train a pre-training model by BERT using the limited document group 4A from the beginning without using the large-scale document group 3, this is inefficient because it requires preparation of many samples within a limited range. In this respect, it is effective to improve accuracy by re-pre-training based on a universally available pre-training model 6 as in this embodiment in terms of achieving both accuracy and efficiency.

[0035] The re-pre-learning may not be performed. That is, the pre-learning model 6 may be prepared in advance and stored in the storage device 32 (FIG. 6) of the computer 30. In this case, the function / step of learning the re-pre-learning model 7 may be omitted from the present invention.

[0036] (1-3. Practical learning) The actual training consists of a process of generating teacher data (annotation data) for training the named entity extraction model 8, and a process of fine-tuning the pre-trained model 6 or re-pre-trained model 7 with the teacher data (annotation data) to generate the named entity extraction model 8. Specifically, the actual learning is performed by an annotation unit (annotation means) 16 and a named entity extraction model learning unit (named entity extraction model learning means) 17 included in the technical element analysis system 1. The annotation unit 16 accepts from a user an annotation setting operation that associates a master named entity tag T1 with a subordinate named entity tag T2 that is subordinate to the master named entity tag for a limited document group 4B, which is a group of documents in a specific technical field, and generates annotated teacher data (hereinafter referred to as "annotation data 5").

[0037] The restricted document group 4B is a document group in the same technical field as the target document 2 to be analyzed and the restricted document group 4A. The restricted document group 4B is preferably intellectual property-related documents. The documents in the restricted document group 4B may be the same as or have some in common with the documents in the restricted document group 4A.

[0038] The master named entity tag T1 is a named entity tag that is set for a named entity in a document of the restricted document group 4B. The subordinate named entity tag T2 is one or more named entity tags that are set subordinate to the master named entity tag T1 for a named entity in a document of the restricted document group 4B. In the present invention, named entity tags having such a master-slave relationship (master named entity tag T1, subordinate named entity tag T2) are set. The set master named entity tag T1 and subordinate named entity tag T2 are associated with each other and stored in the annotation data 5.

[0039] Specifically, in the case where the limited document group 4B is the "claims," ​​which are intellectual property-related documents, a master named entity tag T1 can be set for a named entity relating to a specific invention-specific matter described in the claims, and a slave named entity tag T2 can be set for a named entity that more specifically expresses the nature, structure, etc. of the invention-specific matter to which the master named entity tag T1 is set.

[0040] For example, in the case of the chemical field, a main named entity tag T1 is set for a named entity (invention-specific matter) that represents a specific element or specific property, and a secondary named entity tag T2 is set for a named entity that represents the range or degree of the specific element or specific property in association with the main named entity tag T1. This allows information to be learned about, for example, the range or degree (secondary named entity) that is disclosed for a specific element or specific property (main named entity), enabling a detailed analysis of the target document 2. In intellectual property-related documents, the scope of rights is often expressed in a master-slave relationship, so tagging as described above is particularly effective.

[0041] The annotation unit 16 may also accept, from the user, a setting operation of a related named entity tag Tr, which serves as a clue word for associating the main named entity tag T1 and the subordinate named entity tag T2, for the limited document group 4B. In this case, annotation data 5 is generated in which the related named entity tag Tr is set in addition to the main named entity tag T1 and the subordinate named entity tag T2.

[0042] For example, if a primary named entity tag T1 is set for a named entity relating to a specific element, and a secondary named entity tag T2 is set for a named entity relating to a numerical range of the specific element, then a related named entity tag Tr is set for a named entity that supplements the range or degree, such as a symbol or phrase that indicates the upper or lower limit of the numerical range.By setting the related named entity tag Tr as training data in this way, further improvement in the performance of the prediction model for primary and secondary named entity tags (named entity extraction model 8) can be expected.

[0043] FIG. 3 is a diagram illustrating an example of setting an annotation. FIG. 3 shows an example of setting an annotation for the sentence "The metallic material according to claim 1, wherein the Ni content is 0.5 to 5.0 mass %" in the "Claims" section of an intellectual property-related document. In the example of FIG. 3, a main named entity tag T1 (element tag) is set to "Ni", and a sub named entity tag T2 (lower limit numeric tag, upper limit numeric tag, unit tag) is set to the lower limit numeric value "0.5", upper limit numeric value "5.0", and unit "mass%" of "Ni" in association with the main named entity tag T1 (element tag). In addition, a related named entity tag Tr (range tag) is set to "~", which indicates a range. In addition, a main named entity tag T1 (usage tag) is set to "metallic material". In the example of FIG. 3, an "element tag" set to the element name and "usage tags" set to the method of use, manufacturing method, and object are predefined as the main named entity tag T1, and a "lower limit numeric tag" set to the lower limit value of the numeric range, an "upper limit numeric tag" set to the lower limit value of the numeric range, and a "unit tag" set to the unit expression of the numeric value are predefined as the sub named entity tag T2. In addition, a "range tag" set to an expression indicating a range is predefined as the related named entity tag Tr.

[0044] In the example of FIG. 3, different types of tags are used for the main named entity tag T1 and the subordinate named entity tag T2, but the same type of tag may be used for the main named entity tag T1 and the subordinate named entity tag T2. For example, an "element tag" may be used for the main named entity tag T1 and the subordinate named entity tag T2. In this case, in order to distinguish the "element tag" used as the main named entity tag T1 from the "element tag" used as the subordinate named entity tag T2 in terms of data, for example, the "element tag" used as the main named entity tag T1 is defined as a "main-element tag" and the "element tag" defined as the subordinate named entity tag T2 is defined as a "subordinate-element tag", and tag setting is performed. In addition, in the example of FIG. 3, an "upper limit numeric tag" and a "lower limit numeric tag" are used as the subordinate named entity tag T2, but an "upper limit numeric tag" and a "lower limit numeric tag" may be used as the main named entity tag T1. In this case, for example, the "upper limit numeric tag" and "lower limit numeric tag" used as the master named entity tag T1 are defined as "master upper limit numeric tag" and "master lower limit numeric tag", and the "upper limit numeric tag" and "lower limit numeric tag" used as the slave named entity tag T2 are defined as "slave upper limit numeric tag" and "slave lower limit numeric tag", and the tags are set.

[0045] The named entity extraction model learning unit 17 generates a named entity extraction model 8 by fine-tuning the pre-trained model 6 or the re-pre-trained model 7 with the annotation data 5 (teacher data in which named entity tags are set in the limited document group 4B). Fine-tuning can be performed using a well-known technique known as BERT.

[0046] In this embodiment, a language model (pre-training model 6) is pre-trained using unsupervised data from Japanese Wikipedia, and the language model (re-pre-training model 7) is re-pre-trained using a limited document set 4A. Then, annotation data 5 (supervisor data), which is a dataset for the named entity extraction task, is used to adjust the parameters of the re-pre-training model 7 through fine tuning, and a named entity extraction model 8 is generated.

[0047] By the learning function described above, a named entity extraction model 8 that extracts named entity information from documents is generated.

[0048] (1-4. Re-learning) Next, re-learning for further improving the learning accuracy of the named entity extraction model 8 will be described with reference to Fig. 2. As shown in Fig. 2, the re-learning is executed by the document input unit 11, the document analysis unit 12, the re-annotation unit 18, and the named entity extraction model re-fine-tuning unit 19 provided in the technical element analysis system 1.

[0049] The document input unit 11 accepts input of a restricted document group 4C, which is a document group in a specific technical field. The restricted document group 4C is a document group in the same technical field as the target document 2 and the restricted document groups 4A and 4B. In addition, it is preferable that the restricted document group 4C is an intellectual property-related document.

[0050] The document analysis unit 12 analyzes the limited document group 4C using the named entity extraction model 8, extracts named entity information from each document, and outputs extracted data 9. The re-annotation unit 18 displays the extracted data 9 on the display device 34, accepts tagging correction operations for each extracted named entity tag (master named entity tag T1, subordinate named entity tag T2, related named entity tag Tr), and re-generates a limited document group 4C (hereinafter referred to as "re-annotation data 20") in which the corrected named entity tags are set as training data.

[0051] The named entity extraction model re-fine-tuning unit 19 re-fine-tunes the named entity extraction model 8 using the re-annotated data 20. The re-fine-tuning can be performed using a known technique known as BERT. The above-described re-fine-tuning can be repeated until a predetermined learning performance is obtained.

[0052] It is necessary to construct a model suitable for analyzing the target document 2 through re-pre-learning, actual learning, and re-actual learning, but over-learning may result in a decrease in the accuracy of extraction from the target document 2. For this reason, it is desirable to appropriately perform generalization performance evaluation by cross-validation in each learning process of re-pre-learning, actual learning, and re-actual learning.

[0053] (2. Analysis Function) Next, the analysis function of the technological element analysis system 1 will be described with reference to Fig. 4. The analysis function is mainly composed of a document input unit 11, a document analysis unit 12, and an analysis result display unit 13.

[0054] The document input unit 11 accepts input of one or more target documents 2 that are documents to be analyzed. The target documents 2 are documents in a specific technical field to be analyzed, and are documents in the same technical field as the limited document groups 4A, 4B, and 4C. In addition, it is preferable that the target documents 2 are intellectual property-related documents.

[0055] The document analysis unit 12 uses the named entity extraction model 8 trained by machine learning to extract named entity information from the target document 2. The document analysis unit 12 stores the extracted data 10 from which the named entity information has been extracted in the document extraction DB 100 in association with a document ID, document attributes, etc. that uniquely identify the document.

[0056] FIG. 5 is a diagram showing an example of the data structure of the document extraction DB. 5, extracted data 10 is stored for each document in association with document ID 41, document name 43, and document attribute 45. For example, in the case of document "Patent A" (document attribute: claims) with document ID "T001", "Ni" and "Cr" to which main named entity tag T1 (element tag) is set, and "0.1", "1.5", and "mass%" to which secondary named entity tag T2 (lower limit numeric tag, upper limit numeric tag, unit tag) are set are stored as extracted data 10 of the named entity.

[0057] The document analysis unit 12 performs a technical element analysis of the target document 2 based on the extracted data 10, and outputs the analysis result. For example, the document analysis unit 12 obtains necessary named entity information (e.g., information on main named entities and subordinate named entities) from the extracted data 10, and outputs it as the analysis result. Alternatively, when multiple target documents 2 are input to the document input unit 11 as analysis targets, the document analysis unit 12 may perform a comparative analysis of technical elements between each target document 2 based on the extracted data 10 of each target document 2. The comparative analysis can also be performed after the fact by obtaining the extracted data 10 of multiple documents that have already been extracted from the document extraction DB 100.

[0058] The analysis result display unit 13 displays the analysis results obtained by the document analysis unit 12 on a display device 34 (such as a display) of the computer 30 .

[0059] (3. Hardware Configuration) Next, the hardware configuration of the computer 30 to which the technology element analysis system 1 is applied will be described with reference to FIG.

[0060] As shown in Fig. 6, a computer 30 is configured by connecting a control unit 31, a storage device 32, an input device 33, a display device 34, a media input / output device 35, a communication I / F unit 36, a peripheral device I / F unit 37, etc. via a bus 39. However, the present invention is not limited to this, and various other configurations may be used as appropriate.

[0061] The control unit 31 is composed of a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The control unit 31 loads programs stored in the storage device 32, the ROM, a recording medium, etc. into a work memory area on the RAM, executes them, and drives and controls each unit connected via a bus 39. The ROM permanently stores programs such as the boot program and BIOS of the computer 30, data, etc. The RAM temporarily stores loaded programs and data, and also has a work area used by the control unit 31 to perform various processes described later.

[0062] Moreover, the control unit 31 executes the learning process shown in Figures 7 and 8, the analysis process shown in Figure 9, the search process shown in Figure 12, and the like, in accordance with the processing program stored in the storage device 32. The program for executing each process may be stored in advance in the storage device 32 or ROM of the computer 30, or may be downloaded via a network or the like and stored in the storage device 32, etc.

[0063] The storage device 32 is an HDD (hard disk drive) or the like, and stores the programs executed by the control unit 31, data necessary for executing the programs (pre-training model 6, re-pre-training model 7, named entity extraction model 8, document extraction DB 100, etc.), an OS (operating system), etc. These program codes are read out by the control unit 31 as necessary, transferred to the RAM, and read out by the CPU for execution.

[0064] The input device 33 is, for example, a keyboard, a mouse, a touch panel, a pointing device such as a tablet, a numeric keypad, etc., and outputs input data to the control unit 31.

[0065] The display device 34 is composed of a display device such as a liquid crystal panel or a CRT monitor, and a logic circuit (such as a video adapter) for executing display processing in cooperation with the display device, and displays the display information inputted under the control of the control unit 31 on the display device. Note that the input device 33 and the display device 34 may be integrated into a touch panel type input / output unit.

[0066] The media input / output device 35 is an input / output device for various recording media such as a CD / DVD drive, and performs input / output of data. The communication I / F unit 36 ​​has a communication control device, a communication port, etc., and is an interface that mediates communication with an external device connected for communication via a network, and performs communication control.

[0067] The peripheral device I / F unit 37 is a port for connecting a peripheral device to the computer 30, and the computer 30 transmits and receives data to and from the peripheral device via the peripheral device I / F unit 37. The peripheral device I / F unit 37 is configured with USB, IEEE1394, etc. The connection with the peripheral device may be wired or wireless. The bus 39 is a path that mediates the transmission of control signals, data signals, etc. between the devices.

[0068] (4. Processing of Technology Element Analysis System 1) Next, a description will be given of the processing (technical element analysis method) of the technical element analysis system 1. In the technical element analysis system 1, the control unit 31 of the computer 30 executes a learning process (FIGS. 7 and 8) for learning the pre-training model 6, the re-pre-training model 7, and the named entity extraction model 8. The control unit 31 also uses the named entity extraction model 8 learned by the learning process to extract named entity information from the target document 2 to be analyzed, and executes an analysis process (FIG. 9) of the target document 2 based on the extracted data.

[0069] (4-1. Learning process) First, the learning process (learning method) will be described with reference to FIGS.

[0070] The control unit 31 (pre-learning unit 14) of the computer 30 performs unsupervised pre-learning using the large-scale document set 3 to generate a pre-learning model 6 (step S11, pre-learning step).

[0071] Next, the control unit 31 (re-pre-training unit 15) of the computer 30 re-pre-trains the pre-training model 6 generated in step S11 using a limited document group 4A, which is a group of documents in a specific technical field, and generates the re-pre-training model 7 (step S12, re-pre-training process). Through the above processing, the pre-trained model 6 and the re-pre-trained model 7 are generated and stored in the storage device 32.

[0072] Next, we will explain the actual training process of generating teacher data (annotation data) for training the named entity extraction model 8, and fine-tuning the pre-training model 6 or re-pre-training model 7 using the teacher data (annotation data) to generate the named entity extraction model 8. First, the control unit 31 (annotation unit 16) of the computer 30 accepts from a user an annotation setting operation that associates a master named entity tag T1 with a subordinate named entity tag T2 that is subordinate to the master named entity tag for a limited document group 4B, which is a group of documents in a specific technical field, and generates annotated teacher data (annotation data 5) (step S13, annotation process).

[0073] At this time, the control unit 31 (annotation unit 16) of the computer 30 may accept, from the user, a setting operation of a related named entity tag Tr, which is a clue word for associating the main named entity tag T1 and the subordinate named entity tag T2, for the limited document group 4B. In this case, annotation data 5 (teaching data) is generated in which the related named entity tag Tr is set for the limited document group 4B in addition to the main named entity tag T1 and the subordinate named entity tag T2.

[0074] Next, the control unit 31 (named entity extraction model learning unit 17) of the computer 30 generates a named entity extraction model 8 by fine-tuning the pre-training model 6 generated in step S11 or the re-pre-training model 7 generated in step S12 with the annotation data 5 (step S14, named entity extraction model learning process). Through the above processing, a named entity extraction model 8 is generated and stored in the storage device 32.

[0075] With reference to FIG. 8, a re-learning process for further improving the learning performance of the named entity extraction model 8 will be described. The control unit 31 (document input unit 11) of the computer 30 accepts input of a limited document group 4C, which is a document group in a specific technical field (step S31). Next, the control unit 31 (document analysis unit 12) of the computer 30 analyzes the limited document group 4C using the named entity extraction model 8, extracts named entity information from each document, and outputs extracted data 9 (step S32).

[0076] Next, the control unit 31 (re-annotation unit 18) of the computer 30 displays the extracted data 9 on the display device 34, accepts tagging correction operations for each extracted named entity tag (master named entity tag T1, subordinate named entity tag T2, related named entity tag Tr), and regenerates a limited document group 4C (re-annotation data 20) in which the corrected named entity tags are set as teacher data (step S33, re-annotation process).

[0077] Then, the control unit 31 (named entity extraction model re-fine-tuning unit 19) of the computer 30 re-fine-tunes the named entity extraction model 8 using the re-annotation data 20 (step S34, named entity extraction model re-fine-tuning step). The above steps S31 to S34 may be repeatedly executed until a desired learning performance is obtained.

[0078] (4-2. Analysis processing) Next, the analysis process will be described with reference to FIG. First, the control unit 31 (document input unit 11) of the computer 30 accepts input of one or more target documents 2 which are documents to be analyzed (step S51).

[0079] Next, the control unit 31 (document analysis unit 12) of the computer 30 uses the named entity extraction model 8 to extract named entity information from the target document 2 (step S52). The control unit 31 (document analysis unit 12) links the extracted data 10 from which the named entity information has been extracted with a document ID and document attributes that uniquely identify the document, and stores the data in the document extraction DB 100 (FIG. 5). According to the data structure of Save.

[0080] Next, the control unit 31 (document analysis unit 12) of the computer 30 performs a technical element analysis of the target document 2 based on the extracted data 10 (step S53). For example, the control unit 31 (document analysis unit 12) obtains necessary named entity information (for example, information on main named entities and subordinate named entities) from the extracted data 10 and outputs it as the analysis result. Alternatively, when multiple target documents 2 are input as analysis targets in step S51, the control unit 31 (document analysis unit 12) may perform a comparative analysis of technical elements between each target document 2 based on the extracted data 10 of each target document 2. The comparative analysis can also be performed after the fact by obtaining the extracted data 10 of multiple documents that have already been extracted from the document extraction DB 100.

[0081] Then, the control unit 31 (analysis result display unit 13) of the computer 30 displays the analysis results obtained in step S53 on the display device 34 (such as a display) (step S54).

[0082] Fig. 10 is a diagram showing an example of the analysis result. In the example of Fig. 10, a comparative analysis is performed on the scope of claims of "Patent A" and "Patent B" in Fig. 5 (document extraction DB100). Specifically, for each of "Patent A" and "Patent B", the control unit 31 obtains the numerical ranges of "Ni" and "Cr" to which the main named entity tag T1 (element tag) is attached from the subordinate named entity tag T2 (lower limit numerical value tag, upper limit numerical value tag, unit tag), and plots the numerical ranges on a graph. This type of display makes it easy to grasp the overlapping relationship of the scope of rights of "Patent A" and "Patent B".

[0083] As described above, in this embodiment, first, a re-pre-trained model 7 adapted to a specific technical field is generated based on a pre-trained model 6 pre-trained using a large-scale document group 3. Then, named entity tags having a master-slave relationship (master named entity tag T1, slave named entity tag T2) are annotated for a limited document group 4B, which is a document group in the technical field, and the re-pre-trained model 7 is fine-tuned using the annotation data 5 to generate a named entity extraction model 8 that extracts master-slave named entity tags. This makes it possible to efficiently and accurately generate a named entity extraction model 8 adapted to a technical document to be analyzed, and to perform technical element analysis of the technical document.

[0084] In addition, by setting a related named entity tag Tr, which serves as a clue word for associating the tags, the main named entity tag T1 and the subordinate named entity tag T2, the performance of the named entity extraction model 8, which extracts main and subordinate named entity tags, can be further improved.

[0085] In addition, the performance of the named entity extraction model can be further improved by analyzing the limited document group 4C using the named entity extraction model 8, accepting tagging correction operations for each extracted named entity tag (primary named entity tag T1, secondary named entity tag T2, related named entity tag Tr), regenerating the limited document group 4C (re-annotation data 20) with the corrected named entity tag set as training data, and re-fine-tuning the named entity extraction model 8.

[0086] (5. Document Search) 9 is executed for many target documents 2, a large amount of information on named entities extracted from documents is accumulated in the document extraction DB 100. Here, document retrieval will be explained as an example of how the document extraction DB 100 is used.

[0087] 11 is a functional block diagram showing the document search function of the technological element analysis system 1. The document search function is mainly composed of a search condition setting section 21, a search section 22, and a search result display section 23.

[0088] The search condition setting unit 21 accepts inputs of search keywords, numerical conditions, unit conditions, etc. from the user, and sets search conditions. The search unit 22 searches the document extraction DB 100 for documents that satisfy the search conditions. The search result display unit 23 displays information about the searched documents.

[0089] The flow of the search process will be described with reference to FIG. First, the control unit 31 (search condition setting unit 21) of the computer 30 accepts input of search keywords, numerical conditions, unit conditions, etc. from the user, and sets search conditions (step S71). Next, the control unit 31 (search unit 22) of the computer 30 searches the document extraction DB 100 for documents that satisfy the search conditions set in step S71 (step S72). Then, the control unit 31 (search result display unit 23) of the computer 30 displays the document search results on the display device 34 (step S73).

[0090] For example, assume that the search conditions are set as follows in step S71: Search keyword: "Ni" or "Nickel" Number: Lower limit=5 Upper limit=10 Unit: at% In this case, the control unit 31 searches the document extraction DB100 for documents in which the master named entity tag T1 (element tag) is set to "Ni" or "nickel," the slave named entity tag T2 (lower limit numeric tag) is set to a number equal to or greater than "5," the slave named entity tag T2 (upper limit numeric tag) is set to a number equal to or less than "10," and the slave named entity tag T2 (unit tag) is set to "at%," and displays the search results. The above provides additional information about document retrieval as an example of how the document extraction DB 100 can be used.

[0091] <Example> The technical element analysis system 1 was used to train the named entity extraction model 8, and its performance was evaluated.

[0092] (About learning pre-learning model 6) In this embodiment, an existing model that has been trained and prepared in advance was used as the pre-trained model 6. Specifically, the BERT model (hereinafter referred to as the "Tohoku University model") created by Tohoku University and published at "https: / / github.com / cl-tohoku / bert-japanese" [searched on January 31, 2022] was used as the pre-trained model 6. The Tohoku University model is a pre-trained model that has been trained in advance by BERT using a database (large-scale corpus) in which Japanese Wikipedia articles are structured and accumulated on a large scale as a large-scale document group 3.

[0093] (About re-learning pre-learning model 7) In addition, as a limited document group 4A (a limited document group limited to a specific technical field to be analyzed) to be used for re-pre-learning, the following patent documents (in this embodiment, the “claims” are the subject of the study) with the International Patent Classification C22C were prepared. Document group A: 10,000 registered patents that were extracted based on the name of a certain alloy from among the registered patents that are classified by the Patent Office under the International Patent Classification (IPC) C22C category. Document group B: 31,055 registered patents extracted from category C22C since 2000 (some of the data overlaps with data in A) Then, the computer 30 re-pre-trained the pre-trained model 6 (Tohoku University model) using the above-mentioned limited document set 4A, and generated the re-pre-trained model 7.

[0094] (About learning named entity extraction model 8) In this embodiment, a group of documents containing compositions was selected from the group of documents B as the limited document group 4B (a limited document group limited to a specific technical field to be analyzed) for setting annotations (master named entity tag T1, subordinate named entity tag T2, related named entity tag Tr).

[0095] In addition, the main named entity tag T1 and the subordinate named entity tag T2 are atom,fig, fig_LL, and fig_UL. , s ubstance, Defined use. In addition, limitation was defined as a related named entity tag Tr, and used as a clue word for the subordinate named entity tag T2 and the main named entity tag T1, fig_LL and fig_UL. In addition, selection was defined as a related named entity tag Tr, and was used as a clue word for the subordinate named entity tag T2 and the main named entity tag T1, which are atom, substance, fig_LL, and fig_UL. The contents of each tag are listed in Table 1.

[0096] [Table 1] TIFF0007632858000001.tif88137

[0097] Then, the computer 30 accepted the annotation setting operation of the above-mentioned master named entity tag T1, subordinate named entity tag T2, and related named entity tag Tr for the limited document group 4B, and generated annotation data 5 (teaching data).

[0098] Specifically, the annotation tool doccano was used to create the annotation data (teaching data). The annotation data was written using the IOB2 method, which tags the first word of a named entity with B (Begin), subsequent named entities of the same type with I (Inside), and other words that are not named entities with O (Outside). As a result, annotation data5 (teaching data) of 978 sentences was generated.

[0099] Then, the computer 30 fine-tuned the re-pre-training model 7 using the annotation data 5 (teacher data) to generate a named entity extraction model 8.

[0100] Next, we evaluated the performance of the generated named entity extraction model 8. In evaluating the named entity extraction model, 10 sets of annotation data 5 (teacher data) were randomly created so that the training:validation:test ratio was 8:1:1, and the model was evaluated using the average evaluation value of the 10 sets of test data.

[0101] (Experiment 1: Performance results after re-pre-training) First, we present the results of a comparison of the performance of named entity extraction using re-pre-training. Fig. 13 is a graph showing the results of comparing performance when no re-pre-training was performed ("baseline" in the figure), when re-pre-training was performed using document group A ("A" in the figure), and when re-pre-training was performed using document group B ("B" in the figure). The right vertical axis represents the F1-score (evaluation result), and the left vertical axis represents the number of sentences in the corpus. As a result, we confirmed that the re-pre-trained model 7 ("A" and "B" in Figure 13), which was re-pre-trained with the limited document set 4A, showed improved named entity extraction performance compared to the pre-trained model 6 ("baseline" in Figure 13). In particular, it was confirmed that the best named entity extraction performance could be achieved by using intellectual property-related documents (limited document group 4A) that are closely related to the annotation data 5 (teaching data) as the corpus for the re-pre-training model 7.

[0102] (Experiment 2: Named entity extraction performance results using clue words) Next, the effectiveness of named entity extraction by setting the related named entity tag Tr to the clue word as the named entity tag is confirmed for each individual tag. From the named entity tags of the training data and test data, we selected the named entity extraction model 8 that had the best overall performance when fine-tuning the re-pre-training model 7 for each of the patterns of "no exclusion" using all tags, "limitation exclusion" excluding the limitation tag, and "selection exclusion" excluding the selection tag. In this experiment, we selected the named entity extraction model 8 fine-tuned with document group A for no exclusion and limitation exclusion, and the named entity extraction model 8 fine-tuned with document group B for selection exclusion, and evaluated the evaluation value for each individual tag.

[0103] 14 to 17, the named entity extraction performance of the main named entity tag T1 will be individually confirmed.

[0104] (Comparison of named entity extraction performance between atom and substance) Figure 14 shows the named entity extraction performance for atom, and Figure 15 for substance. For atom and substance, selection is set as the clue word (related named entity tag Tr). The results show that "excluding limitation" and "excluding selection" have better F1-scores (F-value: harmonic mean of precision and recall) than "no exclusion." In particular, "excluding limitation" achieved the best performance, with good results being achieved when only selection was used as the clue word. For atom and substance, it is effective to not set limitation as the clue word, and to select selection as the clue word.

[0105] (Comparison of named entity extraction performance between fig_UL and fig_LL) Figure 16 shows the named entity extraction performance of fig_UL, and Figure 17 shows that of fig_LL. For fig_UL and fig_LL, both selection and limitation are set as clue words (related named entity tags Tr). The results show that "no exclusion" has high extraction performance, and the effect of using both selection and limitation as clue words improved extraction performance. In both cases, the F1-score was better than that of "exclusion of limitation," confirming that the effect of the clue word "selection" was large. As described above, the performance of extracting main named entity tags was improved by appropriately using related named entity tags as clue words.

[0106] Although the preferred embodiments of the technical element analysis system 1 and the like according to the present invention have been described above with reference to the attached drawings, the present invention is not limited to such examples. It is clear that a person skilled in the art can come up with various modified or revised examples within the scope of the technical ideas disclosed in this application, and it is understood that these also naturally fall within the technical scope of the present invention. [Explanation of symbols]

[0107] 1. Technology element analysis system 2. Target document 3. Large-scale document collection 4A~4C... Limited Document Group 5. Annotation data 6. Pre-learning model 7. Re-pre-training model 8. Named entity extraction model 9, 10………Extracted data 20. Re-annotated data 11.....Document input section 12. Document analysis section 13…………Analysis result display section 14.…………Preparatory Study Section 15. Re-learning section 16...Annotation section 17……………Named entity extraction model training section 18. Re-annotation section 19. Named entity extraction model re-fine-tuning part 21...Search condition setting section 22...Search section 23...Search result display section 30……………Computer 31...Control unit 32………Storage device 33...Input device 34………………display device 35...Media input / output device 36.....Communication I / F section 37...Peripheral device I / F section 39……………Bus 100....Document extraction DB T1: Primary named entity tag T2: Sub-named entity tag Tr…Related named entity tag

Claims

1. The computer an annotation process for receiving an annotation setting operation that associates a master named entity tag with a subordinate named entity tag that is subordinate to the master named entity tag for a group of documents of published patent publications, patent publications, or registered utility model publications, and generating annotated teacher data; a named entity extraction model training step of fine-tuning a pre-trained model, which has been pre-trained using a large-scale document group, with the training data to generate a named entity extraction model that extracts named entity information from documents; Run A main named entity tag and a subordinate named entity tag are defined for each type of named entity, and a main named entity tag or a subordinate named entity tag can be set for a named entity of the same type; The annotation step further includes accepting a setting operation of a related named entity tag that serves as a clue word for associating the main named entity tag and the subordinate named entity tag with each other; The annotated training data is training data for generating the named entity extraction model, in which the main named entity tag, the subordinate named entity tag, and the related named entity tag are assigned to the document group, The subordinate named entity tag includes a named entity tag set at an upper limit and a lower limit of a numerical range, The related named entity tag includes a named entity tag set to a symbol or phrase representing a numerical range. A learning method comprising:

2. A re-pre-training step of re-pre-training the pre-trained model using a group of documents of published patent publications, patent publications, or registered utility model publications to generate a re-pre-trained model, The named entity extraction model training step generates the named entity extraction model based on the re-pre-training model and the training data.

2. The learning method according to claim 1 .

3. A re-annotation process for inputting documents of published patent publications, patent publications, or registered utility model publications into the named entity extraction model and accepting tagging correction operations for the extracted named entity tags; a named entity extraction model re-fine-tuning step of re-fine-tuning the named entity extraction model by using the document group to which the corrected named entity tag is set; 3. The learning method according to claim 1, further comprising:

4. The computer an annotation process for receiving an annotation setting operation that associates a master named entity tag with a subordinate named entity tag that is subordinate to the master named entity tag for a group of documents of published patent publications, patent publications, or registered utility model publications, and generating annotated teacher data; a named entity extraction model training step of fine-tuning a pre-trained model, which has been pre-trained using a large-scale document group, with the training data to generate a named entity extraction model that extracts named entity information from documents; A step of inputting a document such as a published patent application, a patent gazette, or a registered utility model application as an analysis target; extracting named entity information from the document to be analyzed using the named entity extraction model; Run A main named entity tag and a subordinate named entity tag are defined for each type of named entity, and a main named entity tag or a subordinate named entity tag can be set for a named entity of the same type; The annotation step further includes accepting a setting operation of a related named entity tag that serves as a clue word for associating the main named entity tag and the subordinate named entity tag with each other; The annotated training data is training data for generating the named entity extraction model, in which the main named entity tag, the subordinate named entity tag, and the related named entity tag are assigned to the document group, The subordinate named entity tag includes a named entity tag set at an upper limit and a lower limit of a numerical range, The related named entity tag includes a named entity tag set to a symbol or phrase representing a numerical range. A method for analyzing technical elements of a technical document.

5. In the inputting step, a plurality of documents are input as analysis targets, In the step of extracting the named entity information, a specific main named entity and an upper limit and a lower limit of a numerical range as a subordinate named entity for the main named entity are obtained for each of the plurality of documents, and the numerical ranges described in each document are comparatively displayed.

5. The method for analyzing technical elements of technical documents according to claim 4.

6. An annotation means for receiving an annotation setting operation that associates a master named entity tag with a subordinate named entity tag that is subordinate to the master named entity tag for a group of documents of published patent publications, patent publications, or registered utility model publications, and generating annotated teacher data; a named entity extraction model training means for fine-tuning a pre-trained model, which has been pre-trained using a large-scale document group, with the training data to generate a named entity extraction model that extracts named entity information from documents; A means for inputting a document such as a published patent application, a patent gazette, or a registered utility model application as an analysis target; means for extracting named entity information from the document to be analyzed using the named entity extraction model; Equipped with A main named entity tag and a subordinate named entity tag are defined for each type of named entity, and a main named entity tag or a subordinate named entity tag can be set for a named entity of the same type; The annotation means further receives a setting operation of a related named entity tag serving as a clue word for associating the main named entity tag and the subordinate named entity tag; The annotated training data is training data for generating the named entity extraction model, in which the main named entity tag, the subordinate named entity tag, and the related named entity tag are assigned to the document group, The subordinate named entity tag includes a named entity tag set at an upper limit and a lower limit of a numerical range, The related named entity tag includes a named entity tag set to a symbol or phrase representing a numerical range. A technical element analysis system for technical documents.

7. In the inputting means, a plurality of documents are input as analysis targets, In the means for extracting information on named entities, a specific main named entity and the upper and lower limits of a numerical range of a subordinate named entity for the main named entity are obtained for each of the plurality of documents, and the numerical ranges described in each document are comparatively displayed.

7. The technology element analysis system according to claim 6.

8. Computer, An annotation means for receiving an annotation setting operation that associates a master named entity tag with a subordinate named entity tag that is subordinate to the master named entity tag for a group of documents of published patent publications, patent publications, or registered utility model publications, and generating annotated teacher data; A named entity extraction model training means for fine-tuning a pre-trained model, which has been pre-trained using a large-scale document group, with the training data to generate a named entity extraction model; Function as a A main named entity tag and a subordinate named entity tag are defined for each type of named entity, and a main named entity tag or a subordinate named entity tag can be set for a named entity of the same type; The annotation means further receives a setting operation of a related named entity tag serving as a clue word for associating the main named entity tag and the subordinate named entity tag; The annotated training data is training data for generating the named entity extraction model, in which the main named entity tag, the subordinate named entity tag, and the related named entity tag are assigned to the document group, The subordinate named entity tag includes a named entity tag set at an upper limit and a lower limit of a numerical range, The related named entity tag includes a named entity tag set to a symbol or phrase representing a numerical range. A program characterized by:

9. Computer, An annotation means for receiving an annotation setting operation that associates a master named entity tag with a subordinate named entity tag that is subordinate to the master named entity tag for a group of documents of published patent publications, patent publications, or registered utility model publications, and generating annotated teacher data; a named entity extraction model training means for fine-tuning, with the training data, a pre-trained model that has been pre-trained using a large-scale document group, to generate a named entity extraction model that extracts named entity information from documents; A means for inputting a document such as a published patent application, a patent gazette, or a registered utility model application as an analysis target; means for extracting named entity information from the document to be analyzed using the named entity extraction model; Function as a A main named entity tag and a subordinate named entity tag are defined for each type of named entity, and a main named entity tag or a subordinate named entity tag can be set for a named entity of the same type; The annotation means further receives a setting operation of a related named entity tag serving as a clue word for associating the main named entity tag and the subordinate named entity tag; The annotated training data is training data for generating the named entity extraction model, in which the main named entity tag, the subordinate named entity tag, and the related named entity tag are assigned to the document group, The subordinate named entity tag includes a named entity tag set at an upper limit and a lower limit of a numerical range, The related named entity tag includes a named entity tag set to a symbol or phrase representing a numerical range. A program characterized by:

10. In the inputting means, a plurality of documents are input as analysis targets, In the means for extracting information on named entities, a specific main named entity and the upper and lower limits of a numerical range for a subordinate named entity relative to the main named entity are obtained for each of the plurality of documents, and the numerical ranges described in each document are comparatively displayed.

10. The program according to claim 9 .

Images