Document sentence concept annotation system and its training method and annotation method
Through the file statement concept labeling system, the pre-trained language model and file analysis model are used to process unstructured files, which solves the problem of low labeling accuracy under file structure variation, and realizes high accuracy and tolerance file structure analysis, which is suitable for Relational extraction and file search.
Patent Information
- Application Number
- CN202011554995.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-24
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2040-12-24
AI Technical Summary
When handling unstructured files, existing file structure analysis technology is difficult to maintain high accuracy and tolerance under file structure variation, resulting in unstable labeling results.
The file statement concept labeling system is adopted, and the language model and file analysis model are pre-trained, and the position generation unit and the data generation unit are combined, the statement set order is adjusted and the starting and end positions are updated. The generated word vector set is used to train the file analysis model. Implement concept labeling of statements for unlabeled files.
It improves the accuracy and tolerance of file structure analysis, enhances the ability to adapt to file structure variation, and achieves significant results when applied to relationship extraction and file search.
Smart Images

Figure CN114580410B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to a document sentence concept tagging system and a training method and a tagging method thereof. Background Art
[0002] Document structure analysis is an important technology for deep document understanding and information extraction. Its purpose is to expand from the level of words and named entities to the level of larger contexts such as multiple sentences. It has a wide range of applications, including dividing document content into smaller units of document fragments and giving corresponding category labels to different blocks. For example, the test standard of many performance competitions is to automatically analyze and divide the sentences in the abstracts of scientific papers in the biomedical field into different sentence concept combinations such as background, purpose, method, conclusion, and contribution.
[0003] After the unstructured document content is annotated through document structure analysis, in addition to obtaining semantic category annotations (i.e., sentence concepts) for multiple sentence groups, it can also be used as feature information for higher-level application analysis. However, in practical applications, in addition to improving the accuracy, sufficient tolerance for document structure variations is also required. Summary of the invention
[0004] The present disclosure discloses a document sentence concept tagging system and its training method and tagging method.
[0005] According to one embodiment of the present disclosure, a training method for a document sentence concept tagging system is provided. The training method for a document sentence concept tagging system comprises the following steps. Provide a plurality of tagged documents. Each tagged document has been tagged with one or more sentence sets of one or more sentence concepts. Generate each sentence set at the starting position and the ending position of these tagged documents. Adjust the order of the one or more sentence sets for these tagged documents, and update each starting position and each ending position to obtain a plurality of generated documents. Each generated file has been tagged with the one or more sentence sets. Input these generated documents into a pre-trained language model to obtain a word vector set for each generated document. Input the word vector sets, each starting position and each ending position of these generated documents into a document analysis model to train the document analysis model. The document analysis model is used to tag untagged documents with the one or more sentence concepts.
[0006] According to another embodiment of the present disclosure, a labeling method for a document sentence concept labeling system is proposed. The labeling method for a document sentence concept labeling system includes the following steps. An unlabeled document and one or more sentence concepts are input into a pre-trained language model to obtain a word vector set of the unlabeled document. The word vector set of the unlabeled document is input into a document analysis model to obtain a sentence set of each sentence concept at the starting position and the ending position of the unlabeled document. Based on each starting position and each ending position, each sentence set is obtained.
[0007] According to another embodiment of the present disclosure, a document sentence concept tagging system is proposed. The document sentence concept tagging system includes a position generating unit, an enhanced data generating unit, a pre-trained language model and a document analysis model. The position generating unit is used to receive a plurality of tagged documents. Each tagged document has been tagged with one or more sentence sets of one or more sentence concepts. The position generating unit generates each sentence set at the starting position and the ending position of each tagged document. The enhanced data generating unit is used to adjust the order of the one or more sentence sets for these tagged documents, and update each starting position and each ending position to obtain a plurality of generated documents. Each generated document has been tagged with the one or more sentence sets. The pre-trained language model is used to obtain a word vector set for each generated document based on these generated documents. The document analysis model receives the word vector sets, each starting position and each ending position of these generated documents for training. The document analysis model is used to tag each sentence concept for untagged documents.
[0008] In order to better understand the above and other aspects of the present disclosure, embodiments are given below and described in detail with reference to the accompanying drawings: BRIEF DESCRIPTION OF THE DRAWINGS
[0009] Figure 1 A schematic diagram of annotating file statement concepts for an unlabeled file according to an embodiment;
[0010] Figure 2A to Figure 2E An execution annotation method of a file statement concept annotation system according to an embodiment;
[0011] Figure 3 is a block diagram of a file statement concept tagging system according to an embodiment;
[0012] Figure 4 is a flow chart of a tagging method of a document statement concept tagging system according to one embodiment;
[0013] Figure 5 is a flow chart of a training method of a document sentence concept tagging system according to one embodiment;
[0014] Figure 6A schematic diagram of using a document sentence concept annotation system to perform relationship extraction according to an embodiment;
[0015] Figure 7 A schematic diagram of using a file statement concept annotation system to perform file search according to an embodiment;
[0016] In the figure:
[0017] 100: Document statement concept annotation system;
[0018] 110: Pre-trained language model;
[0019] 120: Document analysis model;
[0020] 121: initial estimation unit;
[0021] 122: End the estimation unit;
[0022] 130: sentence set selection unit;
[0023] 140: position generating unit;
[0024] 150: Enhanced data generation unit;
[0025] 200: Relation extraction system;
[0026] 210: Sentence segmentation unit;
[0027] 220: named entity recognition unit;
[0028] 230: entity relationship extraction unit;
[0029] 300: File search system;
[0030] 310: index unit;
[0031] 320: search condition processing unit;
[0032] 330: sorting unit;
[0033] 340: search result presentation unit;
[0034] CL1, CL2, CL3, CL4, CL5, CLi, CLj: sentence concepts;
[0035] DC0: marked files;
[0036] DC0': Generate file;
[0037] DC1, DC2, DC3: unlabeled files;
[0038] E01, ..., E05, E01', ..., E05', E12, E13, E14: end position;
[0039] NEi: entity;
[0040] q: query condition;
[0041] S01, ..., S05, S01', ..., S05', S12, S13, S14: starting position;
[0042] S410, S420, S430, S510, S520, S530, S540, S550: steps;
[0043] SC01,…,CS05,SC12,SC13,SC14,SCi,SCj: statement set;
[0044] Si: sentence;
[0045] V0', V1: word vector set. DETAILED DESCRIPTION
[0046] In order to make the purpose, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.
[0047] Please refer to Figure 1 , Figure 2A to Figure 2E , Figure 1 FIG. 1 is a schematic diagram of annotating file statement concepts for an unlabeled file DC1 according to an embodiment. Figure 2A to Figure 2E 1 is an execution annotation method of the document sentence concept annotation system 100 according to an embodiment. For example, for an unannotated document DC1, the document sentence concept annotation system 100 may perform annotation for the sentence concept CL1 of “background” (e.g. Figure 2A ), the sentence concept CL2 of "purpose" (such as Figure 2B ), the sentence concept CL3 of "method" (as shown in 2C), the sentence concept CL4 of "conclusion" (as shown in Figure 2D ), the sentence concept CL5 of "contribution" (such as Figure 2E As shown), etc., to perform file structure analysis.
[0048] like Figure 1 and Figure 2A As shown, after the unlabeled document DC1 and the sentence concept CL1 of "background" are input into the document sentence concept labeling system 100, the document sentence concept labeling system 100 does not find any sentence concept CL1 related to "background", so the sentence set of "background" cannot be labeled.
[0049] Then, if Figure 1 and Figure 2B As shown, after the unannotated document DC1 and the sentence concept CL2 of "purpose" are input into the document sentence concept annotation system 100, the document sentence concept annotation system 100 can annotate the sentence set SC12 of "purpose", that is, annotate the sentence set SC12 at a starting position (start token) S12 and an ending position (endtoken) E12 in the unannotated document DC1.
[0050] Then, if Figure 1 and Figure 2C As shown, after the unannotated file DC1 and the statement concept CL3 of "method" are input into the file statement concept annotation system 100, the file statement concept annotation system 100 can annotate the statement set SC13 of "method", that is, annotate the statement set SC13 at a starting position S13 and an ending position E13 of the unannotated file DC1.
[0051] Then, if Figure 1 and Figure 2D As shown, after the unannotated document DC1 and the sentence concept CL4 of "conclusion" are input into the document sentence concept annotation system 100, the document sentence concept annotation system 100 can annotate the sentence set SC14 of "conclusion", that is, annotate the sentence set SC14 at a starting position S14 and an ending position E14 of the unannotated document DC1.
[0052] Then, if Figure 1 and Figure 2E As shown, after the unlabeled document DC1 and the sentence concept CL5 of "contribution" are input into the document sentence concept labeling system 100, the document sentence concept labeling system 100 does not find any sentence concept CL5 related to "contribution", so the sentence set of "contribution" cannot be labeled.
[0053] In this embodiment, the sentence sets SC12, SC13, and SC14 analyzed by the document sentence concept annotation system 100 may include more than one sentence. Moreover, the content input to the document sentence concept annotation system 100 is not a single sentence, but the full text of the unannotated document DC1. In addition, the document sentence concept annotation system 100 does not classify the sentence concepts for individual sentences, but finds the starting positions S12, S13, and S14 and the ending positions E12, E13, and E14 of the sentence sets SC12, SC13, and SC14 from the entire unannotated document DC1.
[0054] Please refer to Figure 3, which illustrates a block diagram of a document sentence concept tagging system 100 according to an embodiment. The document sentence concept tagging system 100 includes a pre-trained language model 110, a document analysis model 120, a sentence set selection unit 130, a position generation unit 140, and an enhanced data generation unit 150. The pre-trained language model 110, the document analysis model 120, the sentence set selection unit 130, the position generation unit 140, and the enhanced data generation unit 150 are, for example, a circuit, a chip, a circuit board, or a storage device storing program codes.
[0055] The document sentence concept annotation system 100 can use the pre-trained language model 110, the document analysis model 120 and the sentence set selection unit 130 to annotate the unannotated document DC1 with the sentence set SC12 (the same applies to the sentence sets SC13 and SC14). During the annotation process, only one sentence concept is annotated at a time. To make this point clear, Figure 3 Only the statement set SC12 of the statement concept CL2 is used as an example for description.) The operation of the above-mentioned components in the annotation method is described in detail below with a flow chart.
[0056] Please refer to Figure 3 and Figure 4 , Figure 4 A flow chart of a tagging method of a document sentence concept tagging system 100 according to one embodiment is depicted. In step S410, an unlabeled document DC1 and one or more sentence concepts (for example, sentence concept CL2) are input into the pre-trained language model 110 to obtain a word vector set V1 of the unlabeled document DC1. The pre-trained language model 110 is, for example, a BERT model, an ALBERT model, an XLNet model, a RoBERTa model, a DeBERTa model, or a compressed, deleted, or simplified version of the above models. The full name of BERT is Bidirectional Encoder Representations from Transformers. The pre-trained language model 110 can extract high-quality language features from text data.
[0057] Taking the BERT model as an example, the input of the BERT model is generally a sentence, which is marked with [CLS] and [SEP] as the beginning and end of the sentence, such as [CLS]sentence1[SEP] or [CLS]sentence1[SEP]sentence2[SEP]…[SEP]. Sentence1 and sentence2 are sentences respectively.
[0058] In this embodiment, the input to the BERT model is the full text of the unlabeled file DC1 and the sentence concept CL2 (the same applies to sentence concepts CL3 and CL4. During the labeling process, only one sentence concept is labeled at a time. To make this point clear, only sentence concept CL2 is used as an example for explanation). The input to the BERT model is, for example, [CLS]label[SEP]text[SEP]. Where label is one of the sentence concepts CL1, CL2, ..., CL5, and text is the full text of the unlabeled file DC1.
[0059] Next, in step S420, the word vector set V1 of the unlabeled document DC1 is input into the document analysis model 120 to obtain the starting position (for example, the starting position S12) and the ending position (for example, the ending position E12) of one or more sentence sets (for example, the sentence set SC12) of each sentence concept (for example, the sentence concept CL2) in the unlabeled document DC1. The document analysis model 120 includes a starting estimation unit 121 and an ending estimation unit 122. The starting estimation unit 121 is used to estimate the starting position S12; the ending estimation unit 122 is used to estimate the ending position E12. In this step, the document analysis model 120 includes a dense layer and a normalized exponential function (Softmax) layer. After the word vector set V1 is input into the dense layer and the normalized exponential function layer, the starting position distribution probability and the ending position distribution probability are generated, and the starting position S12 and the ending position E12 are obtained accordingly.
[0060] In this embodiment, the document analysis model 120 receives the word vector set V1 generated from the full text of the unlabeled document DC1.
[0061] Then, in step S430, the sentence set selection unit 130 obtains each sentence set (for example, sentence set SC12) of each sentence concept (for example, sentence concept CL2) according to each starting position (for example, starting position S12) and each ending position (for example, ending position E12).
[0062] In this embodiment, the annotation method of the document sentence concept annotation system 100 is based on the analysis of the full text of the unannotated document DC1, inferring the scope of the sentence set included in a sentence concept rather than the classification of a single sentence, so the analysis process can identify the sentence set SC12 that best matches the sentence concept CL2 from all sentences. Therefore, the annotation accuracy of the document sentence concept annotation system 100 can be greatly improved.
[0063] For example, please refer to Table 1 below. Compared with HSLN-CNN, HSLIN-RNN, AI2 and other labeling methods, the labeling method disclosed in the present invention can achieve the highest accuracy in three types of unlabeled documents.
[0064] Table 1
[0065]
[0066] The above embodiment is described by taking sentence concept CL2 as an example. The remaining sentence concepts CL1, CL3, CL4, and CL5 can also be labeled according to the above steps.
[0067] The above is a tagging method of the document sentence concept tagging system 100. Before executing the tagging method, the document sentence concept tagging system 100 must be properly trained. Referring to FIG. 3 , the document sentence concept tagging system 100 can train the document analysis model 120 with the assistance of the position generation unit 140, the enhanced data generation unit 150, and the pre-trained language model 110. The following is a flowchart to explain in detail how the above components operate in the training method.
[0068] Please refer to Figure 3 and Figure 5 , Figure 5 A flowchart of a training method of the document sentence concept tagging system 100 according to an embodiment is shown. In step S510, a plurality of annotated documents DC0 are provided to the position generating unit 140. Each annotated document DC0 has annotated one or more sentence sets SC01, ..., SC05 of one or more sentence concepts CL1, ..., CL5.
[0069] Next, in step S520 , the position generating unit 140 generates the starting positions S01 , . . . , S05 and the ending positions E01 , . . . , E05 of the sentence sets SC01 , . . . , SC05 in the annotated documents DC0 .
[0070] Then, in step S530, the enhanced data generation unit 150 adjusts the order of the one or more statement sets SC01, ..., SC05 for these annotated files DC0, and updates the starting positions S01, ..., S05 and the ending positions E01, ..., E05 to the starting positions S01', ..., S05' and the ending positions E01', ..., E05', to obtain several generated files DC0'. Each generated file DC0' has been annotated with one or more statement sets SC01, ..., SC05. The generated file DC0' is no longer the original annotated file DC0, but retains these statement sets SC01, ..., SC05.
[0071] Next, in step S540, these generated files DC0' are input into the pre-trained language model 110 to obtain a word vector set V0' of each generated file DC0'.
[0072] Then, in step S550, the word vector set V0', each starting position S01', ..., S05' and each ending position E01', ..., E05' of these generated files DC0' are input into the file analysis model 120 to train the file analysis model 120. In other words, these annotated files DC0 are not input into the file analysis model 120 for training, but the generated files DC0' are input into the file analysis model 120 for training. In the generated file DC0', the order of the sentence set SC01, ..., SC05 has been disrupted, and there is no sequential feature. Therefore, the file sentence concept annotation system 100 has a relatively high tolerance and robustness for various file structure variations.
[0073] For example, please refer to Table 2 below. Compared with the AI2 tagging method, the tagging method disclosed herein can maintain a close accuracy rate for shuffled unlabeled files as for unshuffled unlabeled files. On the other hand, the AI2 tagging method will significantly reduce the accuracy rate for shuffled unlabeled files and is not highly tolerant and robust to file structure variations.
[0074]
[0075] Table 2
[0076] According to the above-mentioned embodiment, the document sentence concept tagging system 100 has a relatively high accuracy and robustness in document structure analysis, and can achieve relatively good results when applied to relation extraction and document retrieval.
[0077] Traditionally, when performing relationship extraction, words such as disease A, gene B, and drug C can be searched from the full text of certain documents, and then it is determined that disease A, gene B, and drug C are highly correlated. However, when drug C is a set of sentences with the concept of "background" of these documents, and disease A and gene B are a set of sentences with the concept of "contribution" of these documents, drug C is not actually highly correlated with disease A and gene B, and the correlation is incorrectly determined.
[0078] According to the method of this embodiment, the search can be limited to the sentence set of the sentence concept of "contribution". If it is found that the words A disease, B gene and C drug often exist in the sentence set of the sentence concept of "contribution" in several documents, it can be truly confirmed that A disease, B gene and C drug are highly correlated.
[0079] Please refer to Figure 6, which illustrates a schematic diagram of using a document sentence concept tagging system 100 to perform relation extraction according to an embodiment. The relation extraction system 200 includes a sentence segmentation unit 210, a named entity recognition unit 220, and an entity relation extraction unit 230. The sentence segmentation unit 210, the named entity recognition unit 220, and the entity relation extraction unit 230 are, for example, a circuit, a chip, a circuit board, or a storage device for storing array code.
[0080] After the unlabeled document DC2 and the sentence concept CLi are input into the document sentence concept labeling system 100, a sentence set SCi can be obtained. The named entity recognition unit 220 generates several entities NEi of the unlabeled document DC2. The sentence segmentation unit 210 generates all sentences Si in the unlabeled document DC2. The entity relationship extraction unit 230 generates entity relationship pairs according to whether there is a relationship between these entities NEi in the sentence set SCi of the sentence concept CLi. For example, medical researchers want to know the entity relationship pairs of disease A, gene B and drug C. The entity relationship extraction unit 230 will observe whether entities NEi such as disease A, gene B and drug C often appear together from the sentence set SCi of a specific sentence concept CLi to generate the correct entity relationship pairs. In other words, the relationship extraction system 200 can identify whether there are entity relationship pairs in the sentences of the sentence set SCi of the sentence concept SLi in the unlabeled document DC2.
[0081] In addition, when searching for files traditionally, the word D virus is searched from certain files, and then it is determined that these files are research papers on D virus. However, when the word D virus is a set of sentence concepts in the "background" of some files, these files are likely not research papers on D virus, resulting in file search errors.
[0082] According to the method of this embodiment, the search can be limited to the sentence set of the sentence concept of "contribution", or the sentence set of the sentence concept of "contribution" can be given a higher search priority weight, so that research papers on the D virus can be correctly found.
[0083] Please refer to Figure 7, which illustrates a schematic diagram of using the document sentence concept annotation system 100 to perform document search according to an embodiment. The document search system 300 includes an indexing unit 310, a query processing unit 320, a ranking unit 330, and a result representation unit 340. The indexing unit 310, the query processing unit 320, the ranking unit 330, and the result representation unit 340 are, for example, a circuit, a chip, a circuit board, or a storage device storing array code.
[0084] In the indexing stage, one or more unlabeled files DC3 are input to the indexing unit 310. The indexing unit creates a file index for these unlabeled files DC3.
[0085] The document sentence concept tagging system 100 extracts a plurality of sentence sets SCj belonging to the sentence concept CLj from each un-tagged document DC3. The indexing unit 310 creates a sub-document and a sub-document index for the sentence set SCj.
[0086] In the search phase, the search condition processing unit 320 receives a query condition q and a sentence concept CLj. After the search condition processing unit 320 generates the search condition, the index unit 310 searches in the sub-file index, and the sub-files that meet the condition are sorted by the sorting unit 330. The search result presentation unit 340 gives weighted scores to these sub-files according to the search condition and returns the search result. In other words, the file search system 300 can identify whether a certain query condition is met in the sentence set SCj corresponding to the sentence concept CLi in the unlabeled file DC3.
[0087] According to the above embodiment, the document sentence concept annotation system 100 has a high accuracy and robustness in document structure analysis, and its application in relationship extraction and document search can achieve quite good results. It is particularly helpful in the fields of technical document analysis, bid document analysis, academic text analysis, community comment text analysis, etc.
[0088] It should be understood that the above description is only a specific embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.
Claims
1. A training method for a document sentence concept annotation system, comprising: Providing a plurality of annotated documents, each of which has annotated one or more sentence sets of one or more sentence concepts; Generate each of the statement sets at the start position and the end position of each of the marked files; Adjusting the order of the one or more statement sets in the marked files and updating each of the starting position and each of the ending positions to obtain a plurality of generated files, each of which has been marked with the one or more statement sets; Input the generated files into the pre-trained language model to obtain a word vector set for each generated file; as well as The word vector set, the starting position and the ending position of each generated file are input into a file analysis model to train the file analysis model, and the file analysis model is used to annotate the one or more sentence concepts of the unlabeled file. 2 . The training method of the document sentence concept tagging system as claimed in claim 1 , wherein one of the one or more sentence sets includes more than one sentence. 3 . The training method of the document sentence concept tagging system as claimed in claim 1 , wherein the document analysis model receives the full text of the generated documents. 4 . The training method of the document sentence concept tagging system as claimed in claim 1 , wherein the document analysis model estimates each of the starting positions and each of the ending positions of each of the sentence sets in the unlabeled document. 5 . The training method of the document sentence concept tagging system as claimed in claim 1 , wherein the tagged documents are not input into the document analysis model for training.
6. The training method of the document sentence concept tagging system as described in claim 1, wherein the pre-trained language model is a BERT model, an ALBERT model, an XLNet model, a RoBERTa model, a DeBERTa model, or a compressed, deleted or simplified version of the above models.
7. The training method of the document sentence concept tagging system as claimed in claim 1, wherein the document analysis model comprises a dense layer and a normalized exponential function layer.
8. A method for annotating a document statement concept annotation system, comprising: Inputting an unlabeled document and one or more sentence concepts into a pre-trained language model to obtain a word vector set of the unlabeled document; Inputting the word vector set of the unlabeled document into a document analysis model to obtain a sentence set of each sentence concept at a starting position and an ending position of the unlabeled document, wherein the document analysis model is trained using the method of any one of claims 1 to 7; and According to each of the starting positions and each of the ending positions, each of the statement sets is obtained. 9 . The tagging method of the document sentence concept tagging system as claimed in claim 8 , wherein one of the one or more sentence sets includes more than one sentence. 10 . The annotation method of the document sentence concept annotation system as claimed in claim 8 , wherein the document analysis model receives the full text of the unannotated document.
11. The tagging method of the document sentence concept tagging system as claimed in claim 8, wherein the pre-trained language model is a BERT model, an ALBERT model, an XLNet model, a RoBERTa model, a DeBERTa model, or a compressed, deleted or simplified version of the above models. 12 . The tagging method of the document sentence concept tagging system as claimed in claim 8 , wherein the document analysis model comprises a dense layer and a normalized exponential function layer.
13. A training device for a document sentence concept annotation system, comprising: A position generating unit is used to receive a plurality of annotated documents, each of which has been annotated with one or more sentence sets of one or more sentence concepts, and the position generating unit generates each sentence set at a starting position and an ending position of each annotated document; An enhanced data generating unit is used to adjust the order of the one or more statement sets in the annotated files and update each of the starting positions and each of the ending positions to obtain a plurality of generated files, each of which is annotated with the one or more statement sets; A pre-trained language model is used to obtain a word vector set of each generated file based on the generated files; as well as The file analysis model receives the word vector set, the starting position and the ending position of each generated file for training. The file analysis model is used to annotate the concepts of each sentence in the unlabeled file.
14. The training device of claim 13, wherein The pre-trained language model also receives the unlabeled file to obtain the word vector set of the unlabeled file; The document analysis model also receives the word vector set and each of the sentence concepts of the unlabeled document to obtain each of the sentence sets of each of the sentence concepts at each of the starting positions and each of the ending positions of the unlabeled document.
15. The training device of claim 14, wherein one of the one or more sentence sets comprises more than one sentence.
16. The training device as claimed in claim 14, wherein the pre-trained language model receives full texts of the generated documents.
17. The training device of claim 14, wherein the pre-trained language model receives a full text of the unlabeled document.
18. The training device of claim 14, wherein the annotated documents are not input into the document analysis model for training.
19. The training device of claim 14, wherein the pre-trained language model is a BERT model, an ALBERT model, an XLNet model, a RoBERTa model, a DeBERTa model, or a compressed, deleted, or simplified version of the above models.
20. The training device of claim 14, wherein the document analysis model comprises a dense layer and a normalized exponential function layer.
21. The training device of claim 13, wherein the unlabeled document is input into a relation extraction system to identify whether there is an entity relationship pairing among the statements of each of the statement sets of the one or more statement concepts in the unlabeled document.
22. The training device of claim 13, wherein the unlabeled document is input into a document search system to identify whether a query condition is satisfied in the unlabeled document based on each of the sentence sets corresponding to the one or more sentence concepts.
Citation Information
Patent Citations
Semantic extraction method and device for natural language
CN108287858A
System and method for facilitating affective-state-based artificial intelligence
CN111954889A
