Element extraction method for document-level low-resource scene based on self-label and prompt

Abstract

The application provides an element extraction method for a document-level low-resource scene based on self-labeling and a prompt, adopts a self-labeling method to construct label core words for named entity recognition, adopts a prompt method for relation extraction, and solves the problems of difficulty in constructing label core words for element extraction in a document-level low-resource scene, slow decoding speed and poor performance; and the method is suitable for the fields of justice, tourism, e-commerce, electric games, social interaction and finance.

Description

Technical Field

[0001] This invention belongs to the field of document-level low-resource domain data processing technology, and specifically designs a feature extraction method for document-level low-resource scenarios based on self-labeling and prompt. Background Technology

[0002] In the information age, the volume of internet data is growing exponentially. With the rapid development of internet technology, information on the internet is experiencing explosive growth, not only in scale but also in variety. Simultaneously, the successful application of massive amounts of data in various fields heralds the arrival of the big data era. Big data plays an increasingly important role in social development, and its value has been widely recognized. For example, in recent years, with the deepening of my country's legal system, the adjudication of judicial cases has become increasingly transparent, with the online publication of judgments being a prime example. Judgments, as "judicial products" that carry the court's case adjudication process and results, contain rich judicial information, including the court that issued the judgment, case number, parties' claims, case name, judgment result, and applicable law. These precisely represent the core elements of court "big data." In-depth analysis of this information allows for the summarization of case adjudication patterns, prediction of adjudication trends, and enhancement of judicial credibility, providing technical support for achieving judicial fairness and building a society governed by the rule of law. However, court judgments are a type of semi-structured text, containing both formal legal terminology and everyday language. Furthermore, the writing of court judgments is largely determined by the judge, giving them a range of characteristics including polymorphism, heterogeneity, and arbitrariness. Therefore, extracting valuable information from these unique texts is a topic of significant importance.

[0003] There is a significant need for extracting key indicators from a large volume of judicial documents in the judicial field, for evaluating the effectiveness of judicial procedures or judicial trial reforms. With the development of my country's rule-of-law society and information technology, the number of judicial documents in the judicial field is growing exponentially. The situation of too many cases and too few personnel makes it difficult to meet the needs of case handling, and the differences in the professional competence of judicial personnel also have a certain impact on case judgments. Therefore, the intelligent extraction of key elements from judicial documents can provide references for judicial personnel, assist in case handling, improve work efficiency, and is also crucial for subsequent in-depth analysis and efficient judgment.

[0004] Currently, the field of judicial documents suffers from scarce and difficult-to-annotate data, resulting in a shortage of available data resources. This exemplifies a low-resource application scenario at the document level. The task of extracting indicator elements from judicial documents requires both high-quality and timely feedback. However, current methods not only fail to guarantee extraction accuracy under limited resources but also lack the ability to provide simple, effective, and rapid feedback. In summary, judicial document indicator element extraction models face challenges such as difficulty in training convergence, low extraction accuracy, poor generalization, and slow document-level element extraction and decoding speeds in low-resource application scenarios. Therefore, such models cannot meet the goal of assisting judicial personnel in handling cases and improving work efficiency.

[0005] Therefore, this invention proposes a prompt-based method for feature extraction in low-resource document-level scenarios, which effectively solves the above problems. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a document-level feature extraction method for low-resource scenarios based on self-labeling and prompts. First, a web crawler is used to acquire professional domain data of the document. TF-IDF statistics are used to construct a basic dictionary for the professional domain data. Entities in the professional domain data dictionary are queried using a professional knowledge graph to retrieve their neighboring edges (relationships) and nodes (entities). Entities and relationships not yet in the dictionary are added, resulting in a final professional domain data dictionary. A Bert-NER teacher model is obtained through unsupervised pre-training on large-scale data using this final professional domain data dictionary. Then, a Bert model is selected as the Bert-NER student model. This Bert-NER teacher model and the Bert-NER student model are used to self-train on large-scale unlabeled data to overcome the low-resource problem in the professional domain, iteratively obtaining the final Bert-NER model. Next, remote supervision from a remote knowledge base is used as annotation for the unlabeled data. The word frequencies of the document and the generated results of the BERT auxiliary model (entity and relation_trigger) are statistically analyzed, and these are fused to obtain a comprehensive word frequency. The top-k elements of the comprehensive word frequency for each tag are encoded and fused to obtain the word frequency token for that tag. Then, addressing the significant hierarchical nature of domain-specific tags, a domain-level hierarchical tagging system is designed. This system transforms document entity and relation_trigger tag data into natural language form, and then further transforms it into natural language annotations based on annotation styles, obtaining self-label tokens. These tags are then added to the Bert-NER model vocabulary, and the self-label tokens and term frequency tokens are fused together, serving as the corresponding tokens for the tags added to the Bert-NER vocabulary and fed into the model. Next, for document-level applications in professional domains, the NER task is redefined as an element-oriented language model classification task, significantly accelerating decoding speed and effectively absorbing cross-sentence entity relation semantics. The Bert-NER model is used to perform NER extraction on documents, obtaining entity sets and relation_trigger sets. Finally, for resource-constrained applications in professional domains, a prompt template for RE-MRC tasks is designed. The entity and relation_trigger sets are used to construct the template based on the prompt to obtain RE-MRC task training corpora. The prompt template is then used to significantly expand the data to overcome the resource constraints in professional domains. Finally, the Bert-RE model is obtained by training the corpus using the RE-MRC task. The Bert-RE model is then used to perform RE extraction on the documents to obtain the relation set, thus completing the feature extraction task for document-level low-resource scenarios.

[0007] To address the shortcomings of existing technologies, this invention provides a document-level feature extraction method for low-resource scenarios based on self-tags and prompts.

[0008] The prompt method is a model training method. It adds extra prompt information to the text as input, transforms downstream tasks such as prediction into language model tasks, and converts the prediction results of the language model into the prediction results of the original downstream tasks.

[0009] For the input text x, there is a function f prompt (x), transform x into the prompt form x′, that is: x′=f prompt (x).

[0010] This function typically performs two steps:

[0011] (1) Use a template, which is usually a piece of natural language and contains two empty positions: the position [X] for filling in the input x and the position [Z] for generating the answer text z;

[0012] (2) Fill the input x into the position of [X]: The model predicts the text at [X], and the language model finds the text z that will give the highest score when filled in at [X].

[0013] According to one embodiment of the present invention, a method for feature extraction in low-resource document-level scenarios based on self-tags and prompts is provided, the method comprising the following steps:

[0014] 1) Construction of the Bert-NER model: Web crawling is used to acquire professional domain data, and TF-IDF statistics are used to construct a basic dictionary of professional domain data. Entities in the professional domain data dictionary are queried through a professional knowledge graph to retrieve their neighboring edges (relationships) and nodes (entities). Entities and relationships not in the dictionary are added to the dictionary to obtain the final dictionary of professional domain data. The final dictionary of professional domain data and large-scale unsupervised pre-training are used to obtain the Bert-NER teacher model. Then, a Bert model is taken as the Bert-NER student model. This Bert-NER teacher model and the Bert-NER student model are used to self-train on large-scale unlabeled data to overcome the problem of low resources in the professional domain, and the final Bert-NER model is obtained iteratively.

[0015] 2) Obtaining the word frequency token: Using remote supervision from a remote knowledge base as annotation for unlabeled data, the word frequencies of the document and the entity and relation_trigger generated by the BERT-assisted model are counted separately, and they are fused to obtain the comprehensive word frequency. The top-k elements of the comprehensive word frequency of each tag are encoded and fused to obtain the word frequency token of that tag.

[0016] 3) Obtaining self-labeled tokens: In view of the significant hierarchical nature of professional domain labels, a professional domain hierarchical labeling method is designed. The label data of document entity and relation_trigger are converted into natural language form, and then converted into natural language annotation form according to the annotation style to obtain self-labeled tokens.

[0017] 4) Incorporating the corresponding token of the tag into the above model: The tag is added to the Bert-NER model vocabulary, and then the tag token and the term frequency token are merged and used as the tag token. The corresponding token of the tag added to the Bert-NER vocabulary is then incorporated into the model.

[0018] 5) Obtaining the Entity set and Relation_trigger set: For professional domain document-level application scenarios, the NER task is redefined as a feature-oriented language model classification task, which significantly speeds up the decoding speed and effectively absorbs cross-sentence entity relation semantics. The Bert-NER model is used to extract NER from the document to obtain the Entity set and Relation_trigger set.

[0019] 6) Large-scale expansion of corpus data: For application scenarios with low resources in professional fields, a prompt template for RE-MRC tasks is designed. The RE-MRC task training corpus is obtained by constructing the template based on the prompt using the Entity set and Relation_trigger set. The (corpus) data is expanded on a large scale using the prompt template to overcome the problem of low resources in professional fields. The Bert-RE model is obtained by using the RE-MRC task training corpus.

[0020] 7) Feature extraction: Using the Bert-RE model obtained in step 6), RE extraction is performed on the document to obtain the relation set, thus completing the feature extraction task for document-level low-resource scenarios.

[0021] Preferably, in step 1), the Bert-NER teacher model and the Bert-NER student model are used to perform self-training and adversarial training on large-scale unlabeled data to overcome the problem of low resources in the professional field, and the final Bert-NER model is obtained iteratively.

[0022] Specifically, the adversarial training employs the FGM adversarial training method, with the following perturbations added:

[0023] r adv =ε*g / ||g||2

[0024]

[0025] The newly added adversarial examples are:

[0026] x adv =x+r ad

[0027] Where: x is the original sample, r adv The added perturbation, ε is set to 1, L is the model loss function, g is the backpropagation gradient of the loss, and ||g||² represents the L2 norm of g. θ represents the parameters of the FGM adversarial training model, y represents the prediction result of the FGM adversarial training model, and X... adv For input samples with added perturbations, the L2 norm is the square root of the sum of the squares of all elements in the data vector.

[0028] In this invention, the data used in the self-training method is noisy data. Through self-training and adversarial training, data augmentation is achieved, and the robustness of the model is enhanced by using noisy data.

[0029] Preferably, in step 1), web crawlers are used to acquire professional domain data. TF-IDF statistics are used to construct a basic dictionary for the professional domain data. Entities in the professional domain data dictionary are queried using a professional knowledge graph to retrieve their neighboring edge "relationships" and node "entities." Entities and relationships not yet in the dictionary are added, resulting in the final professional domain data dictionary. A Bert-NER teacher model is obtained using this final professional domain data dictionary and large-scale unsupervised pre-training. Another Bert model is then used as the Bert-NER student model. This Bert-NER teacher model and the Bert-NER student model are used to self-train on large-scale unlabeled data to overcome the problem of low resources in the professional domain. The process of iteratively obtaining the final Bert-NER model more specifically includes:

[0030] Step 1.1) The professional field of the documents to be extracted is to crawl available network resources using a big data framework based on Hadoop, while complying with laws and crawling protocols.

[0031] 1.2) Use the BERT-assisted model to segment the unlabeled data and calculate the TF-IDF of the characters in the data:

[0032] TF i，j =(ni，j / Σ k n k，j );

[0033] IDF i =log(|D| / |j;t) i ∈d j |);

[0034] TF-IDF i =TF i，j ×IDF i ;

[0035] Among them: TF i，j Indicates the term t i In document d j Frequency of occurrence in; n i，j Document d represents document d j The number of terms i in n; k，j Document d represents document d j Number of terms k in the middle; IDF i Indicates the term t i The inverse document frequency; D represents all documents; j represents the index of the j-th document; t i d represents the i-th term; j This represents the j-th document; TF-IDF i This represents the document-inverse document frequency of the i-th term;

[0036] 1.3) Sort the TF-IDF calculation results and select a certain number of words (e.g., 10-5000, preferably 15-4500, preferably 20-4000, preferably 30-3000, such as 40, 50, 60, 70, 80, 90, 100, 150, 200, 300, 400, 500, 600, 700, 800, 900, 1000, 1200, 1500, 1800, 2000, 2200, 2500, 2800) to form a professional domain data foundation dictionary.

[0037] 1.4) For each word in the professional domain data foundation dictionary, remote query is performed through the knowledge graph. The related relationships and nearby entity nodes of the word in the knowledge graph are queried. If it is not in the professional domain data foundation dictionary, it is added to the professional domain data foundation dictionary to form the final professional domain data dictionary.

[0038] 1.5) The processed data is fed into two Bert models respectively. The final dictionary is used with professional domain data and pre-trained using MLM and NSP tasks to obtain a Bert-NER teacher model and a Bert-NER student model.

[0039] The MLM task involves randomly selecting a certain percentage (typically 5-40%, such as 10-30% or 15-20%) of tokens from the sentences in the input corpus, replacing these tokens with "[MASK]", and then using a BERT model to predict what word "[MASK]" represents. The NSP task, on the other hand, involves labeling the sentences in the input corpus. For each sample, two sentences A and B are selected, and the question of whether B is the next sentence of A is predicted. That is, in each sample, two sentences A and B are selected sequentially, and the question of whether B is the next sentence of A is predicted, and this process continues until the prediction of the entire sample is completed. The labeling refers to recording the order of the sentences.

[0040] 1.6) Use the Bert-NER teacher model to predict unlabeled data, and use the prediction results as pseudo-labels.

[0041] 1.7) Then, the labeled and unlabeled data are combined with the labeled and pseudo-labeled data to train the Bert-NER student model.

[0042] 1.8) After this, the Bert-NER student model is used as the new Bert-NER teacher model, and the Bert-NER teacher model is used as the Bert-NER student model for the next stage.

[0043] 1.9) Repeat the iteration a certain number of times (generally 3-300 times, preferably 4-100 times, preferably 5-10 times, for example 3, 4, 5, 6, 7, 8, 9, 10, 20, 30, 40, 50, 60, 70, 80 times) to obtain the final Bert-NER model.

[0044] Preferably, step 2) involves separately calculating the word frequencies of the document and the generated results of the BERT-assisted model (entity and relation_trigger), fusing them to obtain a comprehensive word frequency, and encoding and fusing the top-k elements of the comprehensive word frequency for each tag to obtain the word frequency token for that tag. More specifically, this includes:

[0045] Step 2.1) For document data with n documents, containing m feature label categories, the set of feature label categories is: {C1, C2, C3, ..., C...} m-1 C m};

[0046] Perform word frequency statistics on the document data to obtain the word frequency mapping of element label categories:

[0047] F1(C i ) = arg m axα(x=w,y=C i );

[0048] Where x represents a character in the document, w represents a specific character content, y represents the element label category, and C... i This represents the label category of the i-th element.

[0049] The Bert auxiliary model directly uses the open-source Bert-base model, which consists of 12 layers of transformer-encoders.

[0050] 2.2) Perform word frequency statistics on the prediction results of all feature labels for n documents to obtain the generated word frequency mapping of feature label categories: F2(C i ) = argmaxβ(x = w, y = C i ); where x is a character in the document, w represents a specific character content, and y represents the element label category.

[0051] 2.3) Obtain the comprehensive term frequency using data term frequency mapping and generated term frequency mapping: F(C i )=η1*F1(C i )+η2*F2(C i ); where η1 and η2 are the set weights, with values ​​ranging from 0.2 to 0.8, for example, 0.5.

[0052] 2.4) Using the Bert-NER model to analyze label C i The elements E that rank in the top k by overall word frequency i ={e1, e2, e3, ..., e k-1 e k Encode the tokens, where k is the set quantity, ranging from 1 to 10, for example, 5; then average all the encodings to obtain the word frequency token.

[0053] T(C i ) = Avg(Bert-NER(E t ));

[0054] Note: Due to the large number of formulas in this application, there is not enough lowercase English letters. Therefore, for convenience, different formulas can use i, j, k, l, m, n, o, p, q, r, s, t, u, v, w, etc., which are assigned independent values ​​in the formula calculation. For example, if i is used in different formulas, then i is assigned a separate value in each formula, and so on.

[0055] Preferably, in step 3), considering the significant hierarchical nature of professional domain tags, a hierarchical tagging system for professional domains is designed. This involves converting the tag data of document entity and relation_trigger into natural language form, and then converting it into natural language annotation form according to the annotation style. The process of obtaining the tag token more specifically includes:

[0056] Step 3.1) For the document data, the element label categories C = {C1, C2, C3, ..., C...} m-1 C m}:

[0057] Given the significant hierarchical nature of professional domain labels, a hierarchical labeling system for professional domains is designed. For the parent label C... i It contains multiple sub-tags (C1, C2, C3, ..., C...). k-1 C k If we set its tag to Pos_tag+Sup_Type_tag+Type_tag, then we will set it to Pos_tag+Sup_Type_tag+Type_tag.

[0058] The Sup_Type_tag can be zero layers, one layer, or multiple layers (e.g., 1-20 layers, preferably 1-10 layers, preferably 2-5 layers).

[0059] 3.2) After decomposing the element label category, we get three parts: Pos_tag, Sup_Type_tag, and Type_tag.

[0060] 3.3) First, perform natural language conversion on Pos_tag, Sup_Type_tag and Type_tag respectively to obtain Pos_pre_word, Sup_Type_pre_word and Type_pre_word.

[0061] 3.4) Then, convert the annotation style into natural language annotation form to obtain Pos_word, Sup_Type_word and Type_word.

[0062] 3.5) Input Pos_word, Sup_Type_word, and Type_word into the BERT auxiliary model to obtain the self-labeled token:

[0063] G(C)=(Bert[Pos_word]+Bert[Sup_Type_word]+Bert[Type_word]) / n;

[0064] Where n is the total number of Pos_tag, Sup_Type_tag, and Type_tag.

[0065] Specifically, natural language transformation is performed on Pos_tag, Sup_Type_tag, and Type_tag. For Pos_tag, obtained from feature tag decomposition, the abbreviated position tag is converted to its full name or a synonym, resulting in a natural language word combination, Pos_pre_word. For Sup_Type_tag, obtained from feature tag decomposition, the abbreviated subtype tag is converted to its full name or a synonym, resulting in a natural language word combination, Sup_Type_pre_word. For Type_tag, obtained from feature tag decomposition, the abbreviated type tag is converted to its full name or a synonym, resulting in a natural language word combination, Type_pre_word. For example, the natural language form of the pos tag is position, and the natural language form of the loc tag is position.

[0066] Preferably, step 4) involves adding tags to the Bert-NER model vocabulary, then fusing the tag tokens and term frequency tokens, and using them as tag tokens. More specifically, this process of adding the corresponding tokens of the tags added to the Bert-NER vocabulary to the model includes:

[0067] Step 4.1) For the label category C = {C1, C2, C3, ..., C...} m-1 C m}, its word frequency token is T(C i ), with the tag token being G(C i ), and perform weighted fusion: Final(C i )=α1*T(C i )+α2*G(C i ); where α1 and α2 are the weights set with values ​​ranging from 0.2 to 0.8, for example, 0.5.

[0068] Get the dictionary mapping tags to tag tokens: Dic = {C i Final(C) i )};

[0069] 4.2) Import the dictionary into the Bert-NER model.

[0070] Preferably, step 5) for document-level application scenarios in professional domains, the NER task is redefined as a feature-oriented language model classification task, which significantly accelerates the decoding speed and effectively absorbs cross-sentence entity relation semantics. The process of using the Bert-NER model to perform NER extraction on the document and obtain the Entity set and Relation_trigger set more specifically includes:

[0071] Step 5.1) For a dataset with n documents, extract each document sequentially. The longest document has a length of MAX_L, and the shortest document has a length of MIN_L. The length of the extracted documents is L∈[MIN_L, MAX_L]. Each document contains C types of features, where C∈[MIN_C, MAX_C], where MIN_C represents the minimum number of document feature types and MAX_C represents the maximum number of document feature types. For a document L={x1, x2, x3, ..., x... m-1 x m}, where x represents a character element in the document.

[0072] 5.2) First, the tokenizer of the Bert-NER model is used to segment L, followed by vector embedding, which includes token embedding, segment embedding, and position embedding. Token embedding involves inserting a [CLS] token at the beginning of each sentence and a [SEP] token at the end, where the [CLS] token represents the vector of the current sentence, and the [SEP] token represents the sentence segmentation used to divide the text. Segment embedding is used to distinguish between two sentences, with A and B tokens preceding the different sentences respectively. Therefore, the input sentence is represented as (E A E B E A E B Position embedding adds positional information based on the relative position of the sentence's index, such as [0,1,2,3…50,…100,…150,…200,….500,….].

[0073] 5.3) After embedding, it passes through 12 cascaded transformer layers, and the hidden layer output vector is calculated using a multi-head attention mechanism: h = {h1, h2, h3, ..., h...} m-1 h m};

[0074] The entity tag set is: EL = {l1, l2, l3, ..., l w-1 , l w};

[0075] The set of Relation_trigger tags is: RL = {l1, l2, l3, ..., l...}q-1 , l q};

[0076] Its set of all labels is: l = EL + RL = {l1, l2, l3, ..., l m-1 , l m};

[0077] Calculate its loss:

[0078] Where P(X) i =l i |X)=softmax(h*w), where h is the output vector of the hidden layer and w is the weight of the LM linear layer (e.g., 0.2-0.8, such as 0.5).

[0079] 5.4) The Bert-NER model is obtained by iteratively updating the model parameters using backpropagation of the loss.

[0080] 5.5) Use the trained Bert-NER model to perform NER extraction on the documents to obtain the entity set EG = {e1, e2, e3, ..., e...} p-1 e p} and the Relation_trigger set RG = {r1, r2, r3, ..., r o-1 r o}

[0081] Preferably, in step 6), for application scenarios with low resources in specialized fields, a prompt template for the RE-MRC task is designed. The RE-MRC task training corpus is obtained by constructing the template based on the prompt using the Entity set and the Relation_trigger set. The process of using the prompt template to massively expand the data to overcome the problem of low resources in specialized fields more specifically includes:

[0082] p1: "What is the relationship between [Entity_1] and [Entity_2]?"

[0083] p2: "What relationship does [Relation_trigger_1] represent?"

[0084] p3: "Do [Entity_1] and [Entity_2] have a relationship?"

[0085] p4: "Are [Entity_1] and [Relation_trigger_1] related?"

[0086] p5: "Are [Entity_2] and [Relation_trigger_1] related?";

[0087] The prompt template P = {p1, p2, p3, p4, p5} is obtained;

[0088] 6.2) Construction of the prompt answer space mapping:

[0089] Let the Entity set EG = {e1, e2, e3, ..., e p-1 e p} and the Relation_trigger set RG = {r1, r2, r3, ..., r o-1 r o The values ​​are filled into the prompt template P respectively to obtain the training corpus; [Z] is manually filled into the training corpus in the prompt template P as the answer space mapping to obtain the answer set [Z] = {z1, z2, z3, z4, z5}, thus obtaining the complete training corpus for the RE-MRC task;

[0090] Where: [z1], [z2], [z3], [z4], and [z5] are the answers manually judged from the training corpora p1, p2, p3, p4, and p5, respectively;

[0091] 6.3) Construction of the Bert-RE model:

[0092] For the Relation_trigger set RG = {r1, r2, r3, ..., r...} o-1 r o} and the set of entities EG = {e1, e2, e3, ..., e p-1 e p}, taking r and e as text and putting them into the prompt template as input, and using a pre-trained language model to predict the input text;

[0093] The predicted result is used as the label and compared with the answer set [Z]. The loss is calculated iteratively and backpropagated until the predicted result label matches the answer set [Z], thus obtaining the Bert-RE model.

[0094] Preferably, the construction of the Bert-RE model in step 6.3) is as follows:

[0095] Step 6.3.1) For a dataset with n documents, extract each document sequentially. The longest document length is MAX_L, the shortest document length is MIN_L, and the extracted document length is L∈[MIN_L, MAX_L]. Each document contains C types of features, where C∈[MIN_C, MAX_C]. MIN_C represents the minimum number of document feature types, and MAX_C represents the maximum number of document feature types.

[0096] For a document L = {x1, x2, x3, ..., x...} m-1 x m}, where x represents a character element in the document.

[0097] For the question Q = {q1, q2, q3, ..., q...} z-1 q z}

[0098] Take out q from Q in sequence, and each q and L together form the input corpus.

[0099] 6.3.2) First, the tokenizer of the Bert-RE model is used to segment L and q, and then vector embedding is performed, which includes token embedding, segment embedding and position embedding.

[0100] Token embedding refers to inserting a [CLS] token at the beginning of each sentence and a [SEP] token at the end of each sentence. The [CLS] token represents the vector of the current sentence, and the [SEP] token represents the clause used to segment sentences in the text.

[0101] Segment embedding is used to distinguish between two sentences, with A and B tags preceding the different sentences respectively. Therefore, the input sentence is represented as (E A E B E A E B ,……).

[0102] Position embedding adds positional information based on the relative position of the sentence's index, such as [0,1,2,3…50,…100,…200,…].

[0103] 6.3.3) After embedding, it passes through 12 cascaded transformer layers, and the hidden layer output vector is calculated using a multi-head attention mechanism: h = {h1, h2, h3, ..., h...} n-1 h n};

[0104] Its set of all labels is: l = {l1, l2, l3, ..., l n-1 , l n};

[0105] Calculate its loss:

[0106] Where P(X) i =l i |X)=softmax(h*w), where h is the output vector of the hidden layer and w is the weight of the LM linear layer (e.g., 0.2-0.8, such as 0.5).

[0107] The model consists of a hidden layer followed by a linear layer. The output probability p is obtained through the linear layer, and then a prediction is made based on p. h is the output vector of the hidden layer, and w is the model weight of the linear layer (e.g., 0.2-0.8, such as 0.5). The two are multiplied by a matrix and then softmax is performed to obtain the final prediction probability.

[0108] 6.3.4) The Bert-RE model is obtained by iteratively updating the model parameters using backpropagation of the loss.

[0109] Currently, the field of professional documents suffers from a scarcity of available labeled data and significant labeling challenges, resulting in a shortage of usable data resources. This falls under the category of document-level low-resource application scenarios. The task of extracting key metrics for professional documents requires both high-quality and high-quantity extraction, along with timely feedback. However, current methods not only fail to guarantee extraction accuracy under limited training resources but also lack the ability to provide simple, effective, and rapid feedback. Therefore, there is an urgent need for a document-level low-resource-situation metric extraction method to address the challenges faced by professional document metrics in low-resource application scenarios, including difficulty in convergence, low extraction accuracy, poor generalization, and slow document-level metric extraction and decoding speeds.

[0110] In this invention, web crawlers are used to acquire professional domain data. TF-IDF statistics are used to construct a basic dictionary for the professional domain data. Entities in the professional domain data dictionary are queried using a professional knowledge graph to retrieve their neighboring edges (relationships) and nodes (entities). Entities and relationships not in the dictionary are added to the dictionary to obtain the final professional domain data dictionary. The final professional domain data dictionary and large-scale unsupervised pre-training are used to obtain a Bert-NER teacher model. A Bert model is then taken as the Bert-NER student model. This Bert-NER teacher model and the Bert-NER student model are used to self-train on large-scale unlabeled data to overcome the problem of low resources in the professional domain, and the final Bert-NER model is obtained iteratively.

[0111] The documents to be extracted belong to a professional field that uses a Hadoop-based big data framework to crawl available web resources while complying with laws and web crawling protocols.

[0112] The unlabeled data is segmented using the BERT-assisted model, and the TF-IDF of the characters in the data is calculated:

[0113] TF i，j =(n i，j , / Σ k n k，j );

[0114] IDF i =lo g (|D| / |j:t i ∈d j |);

[0115] TF-IDF i =TF i，j ×IDF i ;

[0116] Among them: TF i，j Indicates the term t i In document d j Frequency of occurrence in; n i，j Document d represents document d j The number of terms i in n; k，j Document d represents document d j Number of terms k in the middle; IDF i Indicates the term t i The inverse document frequency; D represents all documents; j represents the index of the j-th document; t i d represents the i-th term; j This represents the j-th document; TF-IDF i This represents the document-inverse document frequency of the i-th term.

[0117] The TF-IDF calculation results are sorted, and a certain number of words (e.g., 4000) are selected to form a professional domain data foundation dictionary.

[0118] For each word in the professional domain data foundation dictionary, a remote query is performed through the knowledge graph. The related relationships and nearby entity nodes of the word in the knowledge graph are queried. If it is not in the professional domain data foundation dictionary, it is added to the professional domain data foundation dictionary to form the final professional domain data dictionary.

[0119] The processed data is fed into two Bert models. The final dictionary is used to pre-train the model using MLM and NSP tasks, resulting in a Bert-NER teacher model and a Bert-NER student model.

[0120] The MLM task involves randomly selecting a certain percentage (typically 5-40%, for example 15%) of tokens from the sentences in the input corpus, replacing these tokens with [MASK], and then using the BERT model to predict what word [MASK] is.

[0121] The NSP task involves labeling sentences in the input corpus. For each sample, two sentences A and B are selected, and the task is to predict whether B is the next sentence after A.

[0122] The Bert-NER teacher model is used to predict unlabeled data, and the prediction results are used as pseudo-labels.

[0123] Then, the labeled and unlabeled data are combined with labeled and pseudo-labeled data to train the Bert-NER student model.

[0124] After that, the Bert-NER student model was used as the new Bert-NER teacher model, and the Bert-NER teacher model was used as the Bert-NER student model for the next stage.

[0125] Repeat the iteration a certain number of times (e.g., 30 times) to obtain the final Bert-NER model.

[0126] In this invention, remote supervision using a remote knowledge base is used as annotation for unlabeled data. The word frequencies of the document and the entity and relation_trigger generated by the BERT-assisted model are counted separately and fused to obtain a comprehensive word frequency. The top-k elements of the comprehensive word frequency of each tag are encoded and fused to obtain the word frequency token of that tag.

[0127] For document data containing n documents and m feature label categories, the set of feature label categories is:

[0128] {C1, C2, C3, ..., C m-1 C m};

[0129] Perform word frequency statistics on the document data to obtain the word frequency mapping of element label categories:

[0130] F1(C i ) = argmaxα(x = w, y = C i );

[0131] Where x represents a character in the document, w represents a specific character content, y represents the element label category, and C... i This represents the label category of the i-th element.

[0132] The Bert auxiliary model directly uses the open-source Bert-base model, which consists of 12 layers of transformer-encoders.

[0133] Calculate the word frequency of all feature label prediction results for n documents to obtain the generated word frequency mapping of feature label categories:

[0134] F2(C i ) = argmaxβ(x = w, y = C i );

[0135] Where x represents a character in the document, w represents a specific character content, and y represents the element label category.

[0136] The comprehensive word frequency is obtained by using data word frequency mapping and generated word frequency mapping:

[0137] F(C i )=η1*F1(C i )+η2*F2(C i );

[0138] η1 and η2 are the weights set, with values ​​ranging from 0.2 to 0.8, for example, 0.5.

[0139] Using the Bert-NER model to analyze label C i The elements E that rank in the top k by overall word frequency i ={e1, e2, e3, ..., e k-1 e k Encode the tokens, where k is the set quantity, ranging from 1 to 10, for example, 5; then average all the encodings to obtain the word frequency token.

[0140] T(C i ) = Avg(Bert-NER(E i )).

[0141] In this invention, taking into account the significant hierarchical nature of professional domain tags, a professional domain hierarchical tagging system is designed. The tag data of document entity and relation_trigger are converted into natural language form, and then converted into natural language annotation form according to the annotation style to obtain the self-tag token.

[0142] The document data has element label categories C = {C1, C2, C3, ..., C...} m-1 C m}:

[0143] Given the significant hierarchical nature of professional domain labels, a hierarchical labeling system for professional domains is designed. For the parent label C... i It contains multiple sub-tags (C1, C2, C3, ..., C...). m-1 C m If we set its tag to Pos_tag+Sup_Type_tag+Type_tag, then we will set it to Pos_tag+Sup_Type_tag+Type_tag.

[0144] The Sup_Type_tag can be zero layers, one layer, or multiple layers (1-20 layers, preferably 1-10 layers, more preferably 2-5 layers).

[0145] After decomposing the element tag category, we get three parts: Pos_tag, Sup_Type_tag, and Type_tag.

[0146] First, natural language conversion is performed on Pos_tag, Sup_Type_tag, and Type_tag respectively to obtain Pos_pre_word, Sup_Type_pre_word, and Type_pre_word.

[0147] Then, based on the annotation style, it is converted into natural language annotation form to obtain Pos_word, Sup_Type_word, and Type_word.

[0148] Input Pos_word, Sup_Type_word, and Type_word into the Bert auxiliary model to obtain a self-labeled token:

[0149] G(C)=(Bert[Pos_word]+Bert[Sup_Type_word]+Bert[Type_word]) / n

[0150] Where n is the total number of Pos_tag, Sup_Type_tag, and Type_tag.

[0151] In this invention, tags are added to the Bert-NER model vocabulary, and then the tag token and the word frequency token are fused together and used as the tag token. The corresponding token of the tag added to the Bert-NER vocabulary is put into the model.

[0152] For label categories C = {C1, C2, C3, ..., C...} m-1 C m}, its word frequency token is T(C i ), with the tag token being G(C i _) weighted fusion: Final(C i )=α1*T(C i )+α2*G(C i ); where α1 and α2 are the weights set, with values ​​ranging from 0.2 to 0.8, for example, 0.5.

[0153] Get the dictionary mapping tags to tag tokens: Dic = {C i Final(C) i )};

[0154] Import the dictionary into the Bert-NER model.

[0155] In this invention, for document-level application scenarios in professional fields, the NER task is redefined as a feature-oriented language model classification task, which significantly accelerates the decoding speed and effectively absorbs cross-sentence entity relation semantics. The Bert-NER model is used to perform NER extraction on the document to obtain the Entity set and the Relation_trigger set.

[0156] For a dataset with n documents, each document is extracted sequentially. The longest document has a length of MAX_L, and the shortest document has a length of MIN_L. The length of the extracted documents is L∈[MIN_L, MAX_L]. Each document contains C types of features, where C∈[MIN_C, MAX_C]. MIN_C represents the minimum number of document feature types, and MAX_C represents the maximum number of document feature types.

[0157] For a document L = {x1, x2, x3, ..., x...} m-1 x m}, where x represents a character element in the document.

[0158] First, the tokenizer of the Bert-NER model is used to segment L, and then vector embedding is performed, including token embedding, segment embedding, and position embedding.

[0159] Token embedding refers to inserting a [CLS] token at the beginning of each sentence and a [SEP] token at the end of each sentence. The [CLS] token represents the vector of the current sentence, and the [SEP] token represents the clause used to segment sentences in the text.

[0160] Segment embedding is used to distinguish between two sentences, with A and B tags preceding the different sentences respectively. Therefore, the input sentence is represented as (E A E B E A E B ,……).

[0161] Position embedding adds positional information based on the relative position of the sentence's index, such as [0,1,2,3…50,…100,…150,…200,…300,…].

[0162] After embedding, it passes through 12 cascaded transformer layers, and the hidden layer output vector is calculated using a multi-head attention mechanism: h = {h1, h2, h3, ..., h...} m-1 h m};

[0163] The entity tag set is: EL = {l1, l2, l3, ..., l w-1 , l w};

[0164] The set of Relation_trigger tags is: RL = {l1, l2, l3, ..., l...} q-1 , l q};

[0165] Its set of all labels is: l = EL + RL = {l1, l2, l3, ..., l m-1 , l m};

[0166] Calculate its loss:

[0167] Where P(X) i =l i |X)=softmax(h*w), where h is the output vector of the hidden layer and w is the weight of the LM linear layer (e.g., 0.2-0.8, such as 0.5).

[0168] The Bert-NER model is obtained by iteratively updating the model parameters using backpropagation of the loss function.

[0169] The trained Bert-NER model is used to perform NER extraction on the documents, obtaining the entity set EG = {e1, e2, e3, ..., e...}. p-1 e p} and the Relation_trigger set RG = {r1, r2, r3, ..., r o-1 r o}

[0170] In this invention, a prompt template for RE-MRC tasks is designed for application scenarios with low resources in professional fields. The RE-MRC task training corpus is obtained by constructing the prompt template based on the Entity set and Relation_trigger set. The prompt template is used to expand the data on a large scale to overcome the problem of low resources in professional fields.

[0171] The prompt template designed for RE-MRC tasks is as follows:

[0172] p1: "What is the relationship between [Entity_1] and [Entity_2]?"

[0173] p2: "What relationship does [Relation_trigger_1] represent?"

[0174] p3: "Do [Entity_1] and [Entity_2] have a relationship?"

[0175] p4: "Are [Entity_1] and [Relation_trigger_1] related?"

[0176] p5: "Are [Entity_2] and [Relation_trigger_1] related?";

[0177] The prompt template P = {p1, p2, p3, p4, p5} is obtained;

[0178] Construction of the prompt answer space mapping:

[0179] Let the Entity set EG = {e1, e2, e3, ..., e p-1 e p} and the Relation_trigger set RG = {r1, r2, r3, ..., r o-1 r o The values ​​are filled into the prompt template P respectively to obtain the training corpus; [Z] is manually filled into the training corpus in the prompt template P as the answer space mapping to obtain the answer set [Z] = {z1, z2, z3, z4, z5}, thus obtaining the complete training corpus for the RE-MRC task;

[0180] Where: [z1], [z2], [z3], [z4], and [z5] are the answers manually judged from the training corpora p1, p2, p3, p4, and p5, respectively;

[0181] Construction of the Bert-RE model:

[0182] For the Relation_trigger set RG = {r1, r2, r3, ..., r...} o-1 r o} and the set of entities EG = {e1, e2, e3, ..., e p-1 e p}, taking r and e as text and putting them into the prompt template as input, and using a pre-trained language model to predict the input text;

[0183] The predicted result is used as the label and compared with the answer set [Z]. Loss is calculated iteratively, and the Bert-NER model is updated through backpropagation of the loss until the predicted result label matches the answer set [Z], thus obtaining the Bert-RE model.

[0184] In this invention, the Bert-RE model is obtained by training the corpus using the RE-MRC task, and the document is extracted using the Bert-RE model to obtain the relation set, thus completing the feature extraction task in the low-resource scenario at the document level.

[0185] For a dataset with n documents, each document is extracted sequentially. The longest document has a length of MAX_L, and the shortest document has a length of MIN_L. The length of the extracted documents is L∈[MIN_L, MAX_L]. Each document contains C types of features, where C∈[MIN_C, MAX_C]. MIN_C represents the minimum number of document feature types, and MAX_C represents the maximum number of document feature types.

[0186] For a document L = {x1, x2, x3, ..., x...} m-1 x m}, where x represents a character element in the document.

[0187] For the question Q = {q1, q2, q3, ..., q...} z-1 q z}

[0188] Take out q from Q in sequence, and each q and L together form the input corpus.

[0189] First, the tokenizer of the Bert-RE model is used to segment L and q, and then vector embedding is performed, including token embedding, segment embedding, and position embedding.

[0190] Token embedding refers to inserting a [CLS] token at the beginning of each sentence and a [SEP] token at the end of each sentence. The [CLS] token represents the vector of the current sentence, and the [SEP] token represents the clause used to segment sentences in the text.

[0191] Segment embedding is used to distinguish between two sentences, with A and B tags preceding the different sentences respectively. Therefore, the input sentence is represented as (E A E B E A E B ,……).

[0192] Position embedding adds positional information based on the relative position of the sentence's index, such as [0,1,2,3…50,…100,…150,…200,…300,…].

[0193] After embedding, it passes through 12 cascaded transformer layers, and the hidden layer output vector is calculated using a multi-head attention mechanism: h = {h1, h2, h3, ..., h...} n-1 h n};

[0194] Its set of all labels is: l = {l1, l2, l3, ..., l n-1 , l n};

[0195] Calculate its loss:

[0196] Where P(X) i =l i |X)=softmax(h*w), where h is the output vector of the hidden layer and w is the weight of the LM linear layer (e.g., 0.2-0.8, such as 0.5).

[0197] The model consists of a hidden layer followed by a linear layer. The output probability p is obtained through the linear layer, and then a prediction is made based on p. h is the output vector of the hidden layer, and w is the model weight of the linear layer (e.g., 0.2-0.8, such as 0.5). The two are multiplied by a matrix and then softmax is performed to obtain the final prediction probability.

[0198] The Bert-RE model is obtained by iteratively updating the model parameters using backpropagation of the loss.

[0199] In this invention, the relevant professional fields can include the judicial field, tourism field, e-commerce field, video game field, social networking field, and financial field. The method proposed in this invention is applicable to professional fields such as the judicial field, tourism field, e-commerce field, video game field, social networking field, and financial field.

[0200] In this invention, the self-labeled token is the average word vector obtained by embedding the self-labeled phrases obtained from previous labels through a pre-trained model embedding layer. The self-labeled token is used as the core word of the label in the named entity recognition task.

[0201] In this invention, the remote knowledge base and knowledge graph are both existing knowledge graphs, such as CASIA-KB knowledge graph, Galaxybase knowledge graph, TechKG knowledge graph knowledge base, Wiki knowledge base system, China Legal Knowledge Resource Database, China Tourist Attractions Knowledge Graph, EasyDL, Daguan knowledge graph, Baidu knowledge graph, etc.

[0202] In this invention, web crawlers are used to obtain data in the professional field, including all text data such as papers, journals, reports, and documents.

[0203] In this invention, [Z] is manually filled into the training corpus in the prompt template P as an answer space mapping. Based on basic common sense, those skilled in the art (even ordinary people) can understand the answer space mapping [Z] of the training corpus. For example, a teacher is an employee of a school, and the relationship between a teacher and a school is one of employment; a father is the elder of his son, and the relationship between a father and son is one of father and son; a case is the work of a judge, and the relationship between a case and a judge is one of trial; a plaintiff is the plaintiff of a defendant, and the relationship between a plaintiff and a defendant is one of litigation, etc.

[0204] Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects:

[0205] 1. In this invention, the core words of the tag are constructed using self-tags and prompts, which solves the problems of difficulty in constructing core words of tags in low-resource document-level scenarios, slow decoding speed, and poor performance.

[0206] 2. In this invention, for the NER method, a method of self-training using a teacher-student model is proposed to automatically construct labels for a large amount of unlabeled data, thereby alleviating the problem of low resources in professional fields.

[0207] 3. In this invention, the semantic similarity training method of NER is improved. A hierarchical label for professional domains with significant hierarchical labeling is designed. Self-label semantics and pre-trained auxiliary models are used to solve the problems of overly complex construction of core word token representation and over-reliance on remote supervision database in the training method.

[0208] 4. In this invention, for the NER method, a `relation_trigger` tag is added. The extracted `relation_trigger` will be used as a prompt auxiliary construction tag to solve the pipette problem. i The ne method causes a problem of information not being shared between NER tasks and RE tasks.

[0209] 5. In this invention, for the RE method, it is proposed to use the prompt MRC task for training, and the RE task is redefined as an MRC task adapted to the low-resource environment of the professional field.

[0210] 6. In this invention, for the RE method, a prompt-based MRC template is designed, using the entity and relation_trigger tags extracted by NER to fill the MRC data template, thus solving the problem of poor relation extraction performance in low-resource environments at the document level. Attached Figure Description

[0211] Figure 1A schematic diagram of the feature extraction method for document-level low-resource scenarios based on prompt in this invention;

[0212] Figure 2 The NER module structure diagram of the document-level feature extraction method for low-resource scenarios based on prompt in this invention;

[0213] Figure 3 The RE module structure diagram of the document-level low-resource scenario feature extraction method based on prompt in this invention. Detailed Implementation

[0214] The technical solution of the present invention will be illustrated below with examples. The scope of protection sought by the present invention includes, but is not limited to, the following embodiments.

[0215] A feature extraction method for document-level low-resource scenarios based on self-tags and prompts, comprising the following steps:

[0216] 1) Use web crawlers to acquire professional domain data, and construct a basic dictionary of professional domain data by statistically analyzing the data using TF-IDF. Query entities in the professional domain data dictionary through professional knowledge graphs to retrieve their neighboring edges (relationships) and nodes (entities). Add entities and relationships not in the dictionary to obtain the final dictionary of professional domain data. Use the final dictionary of professional domain data and large-scale unsupervised pre-training to obtain the Bert-NER teacher model. Then take a Bert model as the Bert-NER student model. Use this Bert-NER teacher model and the Bert-NER student model to self-train on large-scale unlabeled data to overcome the problem of low resources in professional domains, and iterate to obtain the final Bert-NER model.

[0217] 2) Using remote supervision from a remote knowledge base as annotation for unlabeled data, the word frequencies of entity and relation_trigger generated by the document and the BERT-assisted model are counted separately, and they are fused to obtain the comprehensive word frequency. The top-k elements of the comprehensive word frequency of each tag are encoded and fused to obtain the word frequency token of that tag.

[0218] 3) In view of the significant hierarchical nature of professional domain tags, a professional domain hierarchical tagging system is designed. The tag data of document entity and relation_trigger are converted into natural language form, and then converted into natural language annotation form according to the annotation style to obtain the self-tag token.

[0219] 4) Add the tags to the Bert-NER model vocabulary, then merge the tag tokens and term frequency tokens, and use them as tag tokens. The corresponding tokens of the tags added to the Bert-NER vocabulary are then put into the model.

[0220] 5) For document-level application scenarios in professional fields, the NER task is redefined as a feature-oriented language model classification task, which significantly speeds up the decoding speed and effectively absorbs cross-sentence entity relation semantics. The Bert-NER model is used to extract NER from the document to obtain the Entity set and the Relation_trigger set.

[0221] 6) For application scenarios with low resources in professional fields, a prompt template for RE-MRC tasks is designed. The RE-MRC task training corpus is obtained by constructing the template based on the prompt using the Entity set and Relation_trigger set. The data is expanded on a large scale using the prompt template to overcome the problem of low resources in professional fields. The Bert-RE model is obtained using the RE-MRC task training corpus.

[0222] 7) Use the Bert-RE model obtained in step 6) to perform RE extraction on the document to obtain the relation set, thus completing the feature extraction task for document-level low-resource scenarios.

[0223] Preferably, step 1) involves performing word frequency statistics on the document data to obtain the data word frequency mapping, including:

[0224] Step 1.1) Use web crawlers to acquire domain-specific data, perform TF-IDF analysis on the data to construct a basic domain-specific data dictionary, and use a professional knowledge graph to query entities in the domain-specific data dictionary, retrieving their neighboring edges (relationships) and nodes (entities). Add entities and relationships not in the dictionary to obtain the final domain-specific data dictionary. Use the final domain-specific data dictionary and large-scale unsupervised pre-training to obtain a Bert-NER teacher model. Then, take a Bert model as the Bert-NER student model. Use this Bert-NER teacher model and the Bert-NER student model to self-train on large-scale unlabeled data to overcome the problem of low resources in the domain-specific data, and iteratively obtain the final Bert-NER model.

[0225] The documents to be extracted belong to a professional field that uses a Hadoop-based big data framework to crawl available web resources while complying with laws and web crawling protocols.

[0226] The unlabeled data is segmented using the BERT-assisted model, and the TF-IDF of the characters in the data is calculated:

[0227] TF i，j =(n i，j / ∑ k n k，j );

[0228] IDF i =log(|D| / |j:t) i ∈d j |);

[0229] TF-IDF i =TF i，j ×IDF i ;

[0230] Among them: TF i，j Indicates the term t i In document d j Frequency of occurrence in; n i，j Document d represents document d j The number of terms i in n; k，j Document d represents document d j Number of terms k in the middle; IDF i Indicates the term t i The inverse document frequency; D represents all documents; j represents the index of the j-th document; t i d represents the i-th term; j This represents the j-th document; TF-IDF i This represents the document-inverse document frequency of the i-th term;

[0231] The TF-IDF calculation results are sorted, and a certain number of words (e.g., 2000) are selected to form a professional domain data foundation dictionary.

[0232] For each word in the professional domain data foundation dictionary, a remote query is performed through the knowledge graph. The related relationships and nearby entity nodes of the word in the knowledge graph are queried. If it is not in the professional domain data foundation dictionary, it is added to the professional domain data foundation dictionary to form the final professional domain data dictionary.

[0233] The processed data is fed into two Bert models. The final dictionary is used to pre-train the model using MLM and NSP tasks, resulting in a Bert-NER teacher model and a Bert-NER student model.

[0234] The MLM task involves randomly selecting a certain percentage (e.g., 15%) of tokens from the sentences in the input corpus, replacing these tokens with [MASK], and then using a BERT model to predict what word [MASK] is.

[0235] The NSP task involves labeling sentences in the input corpus. For each sample, two sentences A and B are selected, and the task is to predict whether B is the next sentence after A.

[0236] The Bert-NER teacher model is used to predict unlabeled data, and the prediction results are used as pseudo-labels.

[0237] Then, the labeled and unlabeled data are combined with labeled and pseudo-labeled data to train the Bert-NER student model.

[0238] After that, the Bert-NER student model was used as the new Bert-NER teacher model, and the Bert-NER teacher model was used as the Bert-NER student model for the next stage.

[0239] Repeat the iteration a certain number of times (e.g., 20 times) to obtain the final Bert-NER model.

[0240] Preferably, in step 1), the Bert-NER teacher model and the Bert-NER student model are used to perform self-training and adversarial training on large-scale unlabeled data to overcome the problem of low resources in the professional field, and the final Bert-NER model is obtained iteratively.

[0241] Specifically, the adversarial training employs the FGM adversarial training method, with the following perturbations added:

[0242] r adv =ε*g / ||g||2

[0243]

[0244] The newly added adversarial examples are:

[0245] x adv =x+r adv

[0246] Where: x is the original sample, r adv The added perturbation, ε is set to 1, L is the model loss function, g is the backpropagation gradient of the loss, and ||g||² represents the L2 norm of g. θ represents the parameters of the FGM adversarial training model, y represents the prediction result of the FGM adversarial training model, and X... adv For input samples with added perturbations, the L2 norm is the square root of the sum of the squares of all elements in the data vector.

[0247] Preferably, in step 2), the word frequencies of the document and the generated results of the BERT auxiliary model, namely entity and relation_trigger, are counted separately, and then fused to obtain the comprehensive word frequency. The top-k elements of the comprehensive word frequency for each tag are encoded and fused to obtain the word frequency token for that tag, including:

[0248] Step 2.1) For document data with n documents, containing m feature label categories, the set of feature label categories is: {C1, C2, C3, ..., C...} m-1 C m};

[0249] Perform word frequency statistics on the document data to obtain the word frequency mapping of element label categories:

[0250] F1(C i ) = argmaxα(x = w, y = C i );

[0251] Where x represents a character in the document, w represents a specific character content, y represents the element label category, and C... i This represents the label category of the i-th element.

[0252] The Bert auxiliary model directly uses the open-source Bert-base model, which consists of 12 layers of transformer-encoders.

[0253] Calculate the word frequency of all feature label prediction results for n documents to obtain the generated word frequency mapping of feature label categories:

[0254] F2(C i ) = argmaxβ(x = w, y = C i );

[0255] Where x represents a character in the document, w represents a specific character content, and y represents the element label category.

[0256] The comprehensive word frequency is obtained by using data word frequency mapping and generated word frequency mapping:

[0257] F(C i )=η1,F1(C i )+η2,F2(C i );

[0258] η1 and η2 are the weights set, with values ​​ranging from 0.2 to 0.8, for example, 0.5.

[0259] Using the Bert-NER model to analyze label C i The elements E that rank in the top k by overall word frequency i ={e1, e2, e3, ..., e k-1 e k Encode the tokens, where k is the set quantity, ranging from 1 to 10, for example, 5; then average all the encodings to obtain the word frequency token.

[0260] T(C i) = Avg(Bert-NER(E i )).

[0261] As a preferred option, step 3) addresses the significant hierarchical nature of professional domain tags by designing a hierarchical tagging system. This involves converting the tag data for document entities and relation_triggers into natural language, then further converting it into natural language annotations based on the annotation style, to obtain the tag token, which includes:

[0262] Step 3.1) Assess the feature label categories C = {C1, C2, C3, ..., C...} of the document data. m-1 C m}:

[0263] Given the significant hierarchical nature of professional domain labels, a hierarchical labeling system for professional domains is designed. For the parent label C... i It contains multiple sub-tags (C1, C2, C3, ..., C...). k-1 C k If we set its tag to Pos_tag+Sup_Type_tag+Type_tag, then we will set it to Pos_tag+Sup_Type_tag+Type_tag.

[0264] The Sup_Type_tag can be zero or multiple layers (1-20 layers, preferably 1-10 layers, more preferably 2-5 layers).

[0265] After decomposing the element tag category, we get three parts: Pos_tag, Sup_Type_tag, and Type_tag.

[0266] First, natural language conversion is performed on Pos_tag, Sup_Type_tag, and Type_tag respectively to obtain Pos_pre_word, Sup_Type_pre_word, and Type_pre_word.

[0267] Then, based on the annotation style, it is converted into natural language annotation form to obtain Pos_word, Sup_Type_word, and Type_word.

[0268] Input Pos_word, Sup_Type_word, and Type_word into the Bert auxiliary model to obtain a self-labeled token:

[0269] G(C)=(Bert[Pos_word]+Bert[Sup_Type_word]+Bert[Type_word]) / n;

[0270] Where n is the total number of Pos_tag, Sup_Type_tag, and Type_tag.

[0271] Preferably, step 4) involves adding tags to the Bert-NER model vocabulary, then fusing the tag tokens and term frequency tokens, and using them as tag tokens. The corresponding tokens of the tags added to the Bert-NER vocabulary are then fed into the model, including:

[0272] According to step 4.1), for the label category C = {C1, C2, C3, ..., C...} m-1 C m}, its word frequency token is T(C i ), with the tag token being G(C i ), and perform weighted fusion on it:

[0273] Final(C i )=α1*T(C i )+α2*G(C i ); where α1 and α2 are the weights set, with values ​​ranging from 0.2 to 0.8, for example, 0.5.

[0274] Get the dictionary mapping tags to tag tokens: Dic = {C i Final(C) i )};

[0275] Import the dictionary into the Bert-NER model.

[0276] As a preferred approach, step 5) redefines the NER task as a feature-oriented language model classification task for document-level applications in professional domains. This significantly accelerates decoding speed and effectively absorbs cross-sentence entity relation semantics. The Bert-NER model is used to perform NER extraction on the document, obtaining the Entity set and Relation_trigger set, including:

[0277] Step 5.1) For a dataset with n documents, extract each document sequentially. The longest document length is MAX_L, the shortest document length is MIN_L, and the extracted document length is L∈[MIN_L, MAX_L]. Each document contains C types of features, where C∈[MIN_C, MAX_C]. MIN_C represents the minimum number of document feature types, and MAX_C represents the maximum number of document feature types.

[0278] For a document L = {x1, x2, x3, ..., x...} m-1 x m}, where x represents a character element in the document.

[0279] First, the tokenizer of the Bert-NER model is used to segment L, and then vector embedding is performed, including token embedding, segment embedding, and position embedding.

[0280] Token embedding refers to inserting a [CLS] token at the beginning of each sentence and a [SEP] token at the end of each sentence. The [CLS] token represents the vector of the current sentence, and the [SEP] token represents the clause used to segment sentences in the text.

[0281] Segment embedding is used to distinguish between two sentences, with A and B tags preceding the different sentences respectively. Therefore, the input sentence is represented as (E A E B E A E B ,……).

[0282] Position embedding adds positional information based on the relative position of the sentence's index, such as [0,1,2,3…50,…100,…200,…].

[0283] After embedding, it passes through 12 cascaded transformer layers, and the hidden layer output vector is calculated using a multi-head attention mechanism: h = {h1, h2, h3, ..., h...} m-1 h m};

[0284] The entity tag set is: EL = {l1, l2, l3, ..., l w-1 , l w};

[0285] The set of Relation_trigger tags is: RL = {l1, l2, l3, ..., l...} q-1 , l q};

[0286] Its set of all labels is: l = EL + RL = {l1, l2, l3, ..., l m-1 , l m};

[0287] Calculate its loss:

[0288] Where P(X) i =l i |X)=softmax(h*w), where h is the output vector of the hidden layer and w is the weight of the LM linear layer (e.g., 0.2-0.8, such as 0.5).

[0289] The Bert-NER model is obtained by iteratively updating the model parameters using backpropagation of the loss function.

[0290] The trained Bert-NER model is used to perform NER extraction on the documents, obtaining the entity set EG = {e1, e2, e3, ..., e...}. p-1 e p} and the Relation_trigger set RG = {r1, r2, r3, ..., r o-1 r o}

[0291] As a preferred option, step 6) designs a prompt template for RE-MRC tasks, specifically for application scenarios with low resources in specialized fields. It uses the Entity set and Relation_trigger set to construct the template based on the prompt to obtain the RE-MRC task training corpus. The use of the prompt template to significantly expand the data to overcome the problem of low resources in specialized fields includes:

[0292] Based on step 6.1), the prompt template for the RE-MRC task is designed as follows:

[0293] p1: "What is the relationship between [Entity_1] and [Entity_2]?"

[0294] p2: "What relationship does [Relation_trigger_1] represent?"

[0295] p3: "Do [Entity_1] and [Entity_2] have a relationship?"

[0296] p4: "Are [Entity_1] and [Relation_trigger_1] related?"

[0297] p5: "Are [Entity_2] and [Relation_trigger_1] related?";

[0298] The prompt template P = {p1, p2, p3, p4, p5} is obtained;

[0299] 6.2) Construction of the prompt answer space mapping:

[0300] Let the Entity set EG = {e1, e2, e3, ..., e p-1 e p} and the Relation_trigger set RG = {r1, r2, r3, ..., r o-1 r oThe values ​​are filled into the prompt template P respectively to obtain the training corpus; [Z] is manually filled into the training corpus in the prompt template P as the answer space mapping to obtain the answer set [Z] = {z1, z2, z3, z4, z5}, thus obtaining the complete training corpus for the RE-MRC task;

[0301] Where: [z1], [z2], [z3], [z4], and [z5] are the answers manually judged from the training corpora p1, p2, p3, p4, and p5, respectively;

[0302] 6.3) Construction of the Bert-RE model:

[0303] For the Relation_trigger set RG = {r1, r2, r3, ..., r...} o-1 r o} and the set of entities EG = {e1, e2, e3, ..., e p-1 e p}, taking r and e as text and putting them into the prompt template as input, and using a pre-trained language model to predict the input text;

[0304] The predicted result is used as the label and compared with the answer set [Z]. The loss is calculated iteratively and backpropagated until the predicted result label matches the answer set [Z], thus obtaining the Bert-RE model.

[0305] As a preferred embodiment, step 6.3) of constructing the Bert-RE model is as follows:

[0306] For a dataset with n documents, each document is extracted sequentially. The longest document has a length of MAX_L, and the shortest document has a length of MIN_L. The length of the extracted documents is L∈[MIN_L, MAX_L]. Each document contains C types of features, where C∈[MIN_C, MAX_C]. MIN_C represents the minimum number of document feature types, and MAX_C represents the maximum number of document feature types.

[0307] For a document L = {x1, x2, x3, ..., x...} m-1 x m}, where x represents a character element in the document.

[0308] For the question Q = {q1, q2, q3, ..., q...} z-1 q z}

[0309] Take out q from Q in sequence, and each q and L together form the input corpus.

[0310] First, the tokenizer of the Bert-RE model is used to segment L and q, and then vector embedding is performed, including token embedding, segment embedding, and position embedding.

[0311] Token embedding refers to inserting a [CLS] token at the beginning of each sentence and a [SEP] token at the end of each sentence. The [CLS] token represents the vector of the current sentence, and the [SEP] token represents the clause used to segment sentences in the text.

[0312] Segment embedding is used to distinguish between two sentences, with A and B tags preceding the different sentences respectively. Therefore, the input sentence is represented as (E A E B E A E B ,……).

[0313] Position embedding adds positional information based on the relative position of the sentence's index, such as [0,1,2,3…50,…100,…200,…].

[0314] After embedding, it passes through 12 cascaded transformer layers, and the hidden layer output vector is calculated using a multi-head attention mechanism: h = {h1, h2, h3, ..., h...} n-1 h n};

[0315] Its set of all labels is: l = {l1, l2, l3, ..., l n-1 , l n};

[0316] Calculate its loss:

[0317] Where P(X) i =l i |X)=softmax(h*w), where h is the output vector of the hidden layer and w is the weight of the LM linear layer (e.g., 0.2-0.8, such as 0.5).

[0318] The Bert-RE model is obtained by iteratively updating the model parameters using backpropagation of the loss.

[0319] The method proposed in this invention is applicable to the judicial, tourism, e-commerce, video game, social networking, and financial fields. The following description uses the judicial field as an example; however, the method is equally applicable to other fields.

[0320] Example 1

[0321] like Figure 1As shown, a method for feature extraction in low-resource scenarios at the document level in the judicial field based on self-labeling and prompts is presented. This method includes the following steps:

[0322] 1) Use web crawlers to acquire data in the judicial field, and construct a basic dictionary for the judicial field data by statistically analyzing the TF-IDF of the data. Query entities in the judicial field data dictionary through the judicial Galaxybase knowledge graph to retrieve their neighboring edges (relationships) and nodes (entities). Add entities and relationships not in the dictionary to obtain the final dictionary for the judicial field data. Use the final dictionary for the judicial field data and large-scale unsupervised pre-training to obtain the Bert-NER teacher model. Then take a Bert model as the Bert-NER student model. Use this Bert-NER teacher model and the Bert-NER student model to self-train on large-scale unlabeled data to overcome the problem of low resources in the judicial field, and iterate to obtain the final Bert-NER model.

[0323] 2) Using Galaxybase knowledge graph remote supervision as annotation for unlabeled data, the word frequencies of entity and relation_trigger generated by the document and BERT auxiliary model are counted respectively, and they are fused to obtain the comprehensive word frequency. The top-k elements of the comprehensive word frequency of each tag are encoded and fused to obtain the word frequency token of that tag.

[0324] 3) In view of the significant hierarchical nature of labels in the judicial field, a hierarchical labeling system for the judicial field is designed. The label data of document entity and relation_trigger are converted into natural language form, and then converted into natural language annotation form according to the annotation style to obtain the self-label token.

[0325] 4) Add the tags to the Bert-NER model vocabulary, then merge the tag tokens and term frequency tokens, and use them as tag tokens. The corresponding tokens of the tags added to the Bert-NER vocabulary are then put into the model.

[0326] 5) For document-level application scenarios in the judicial field, the NER task is redefined as an element-oriented language model classification task, which significantly speeds up the decoding speed and effectively absorbs cross-sentence entity relation semantics. The Bert-NER model is used to extract NER from the document to obtain the Entity set and the Relation_trigger set.

[0327] 6) To address the low-resource application scenario in the judicial field, a prompt template for the RE-MRC task is designed. The RE-MRC task training corpus is obtained by constructing the template based on the prompt using the Entity set and the Relation_trigger set. The data is then expanded on a large scale using the prompt template to overcome the problem of low resources in the judicial field. The Bert-RE model is obtained using the RE-MRC task training corpus.

[0328] 7) Use the Bert-RE model obtained in step 6) to perform RE extraction on the document to obtain the relation set, thus completing the feature extraction task for document-level low-resource scenarios.

[0329] Example 2

[0330] Repeat Example 1, except that step 1) uses a web crawler to acquire data in the judicial field, performs TF-IDF statistical analysis on the data to construct a basic dictionary for the judicial field data, queries entities in the judicial field data dictionary using the Galaxybase knowledge graph to retrieve their neighboring edges (relationships) and nodes (entities), and adds entities and relationships not in the dictionary to obtain the final dictionary for the judicial field data. Using the final dictionary for the judicial field data and large-scale unsupervised pre-training on the data, a Bert-NER teacher model is obtained. Then, a Bert model is selected as the Bert-NER student model. This Bert-NER teacher model and the Bert-NER student model are used to self-train on large-scale unlabeled data to overcome the problem of low resources in the judicial field. The final Bert-NER model is obtained iteratively, including, as follows: Figure 1 As shown:

[0331] Step 2.1) The professional field of the documents to be extracted is to crawl available network resources using a big data framework based on Hadoop, while complying with laws and crawling protocols.

[0332] The unlabeled data is segmented using the BERT-assisted model, and the TF-IDF of the characters in the data is calculated:

[0333] TF i，j =(n i，j / ∑ k n k，j );

[0334] Indicates the term t i In document d j The frequency of occurrence.

[0335] IDF i =log(|D| / |j:t) i ∈d j |);

[0336] The total document contains the term t i The number of documents.

[0337] TF-IDF i =TF i，j ×IDF i ;

[0338] Among them: TF i，j Indicates the term t i In document d j Frequency of occurrence in; n i，j Document d represents document d j The number of terms i in n; k，j Document d represents document d j Number of terms k in the middle; IDF i Indicates the term t i The inverse document frequency; D represents all documents; j represents the index of the j-th document; t i d represents the i-th term; j This represents the j-th document; TF-IDF i This represents the document-inverse document frequency of the i-th term;

[0339] The TF-IDF calculation results were sorted, and 1,500 words were selected to form a basic dictionary of data in the judicial field.

[0340] For each word in the basic dictionary of judicial data, a remote query is performed through the Galaxybase knowledge graph. The related relationships and nearby entity nodes of the word in the Galaxybase knowledge graph are queried. If it is not in the basic dictionary of judicial data, it is added to the basic dictionary of judicial data to form the final dictionary of judicial data.

[0341] The processed data is fed into two Bert models. The final dictionary of judicial domain data is pre-trained using MLM and NSP tasks to obtain a Bert-NER teacher model and a Bert-NER student model.

[0342] The MLM task involves randomly selecting 15% of the tokens in the sentences of the input corpus, replacing these tokens with [MASK], and then using the BERT model to predict what word [MASK] is.

[0343] The NSP task involves labeling sentences in the input corpus. For each sample, two sentences A and B are selected, and the task is to predict whether B is the next sentence after A.

[0344] The Bert-NER teacher model is used to predict unlabeled data, and the prediction results are used as pseudo-labels.

[0345] Then, the labeled and unlabeled data are combined with labeled and pseudo-labeled data to train the Bert-NER student model.

[0346] After that, the Bert-NER student model was used as the new Bert-NER teacher model, and the Bert-NER teacher model was used as the Bert-NER student model for the next stage.

[0347] The process was repeated 25 times to obtain the final Bert-NER model.

[0348] Example 3

[0349] Repeat Example 2, except that in step 2), the word frequencies of the document and the entity and relation_trigger generated by the BERT-assisted model are counted separately, and then fused to obtain the comprehensive word frequency. The top-k elements of the comprehensive word frequency for each tag are encoded and fused to obtain the word frequency token for that tag, such as... Figure 1 As shown:

[0350] Step 2.1) For document data with n documents, containing m feature label categories, the set of feature label categories is: {C1, C2, C3, ..., C...} m-1 C m};

[0351] Perform word frequency statistics on the document data to obtain the word frequency mapping of element label categories:

[0352] F1(C i ) = argmaxα(x = w, y = C i );

[0353] Where x represents a character in the document, w represents a specific character content, y represents the element label category, and C... i This represents the label category of the i-th element.

[0354] The Bert auxiliary model directly uses the open-source Bert-base model, which consists of 12 layers of transformer-encoders.

[0355] Calculate the word frequency of all feature label prediction results for n documents to obtain the generated word frequency mapping of feature label categories:

[0356] F2(C i ) = argmaxβ(x = w, y = C i );

[0357] Where x represents a character in the document, w represents a specific character content, and y represents the element label category.

[0358] The comprehensive word frequency is obtained by using data word frequency mapping and generated word frequency mapping:

[0359] F(C i )=η 1* F1(C i )+η 2* F2(C i );

[0360] η1 and η2 are the weights set, with a value of 0.5.

[0361] Using the Bert-NER model to analyze label C i The elements E that rank in the top k by overall word frequency i ={e1, e2, e3, ..., e k-1 e k The encoding is performed, where k is the set quantity, with a value of 5. Then, the average of all encodings is taken to obtain the word frequency token.

[0362] T(C i ) = Avg(Bert-NER(E i )).

[0363] Example 4

[0364] Repeat Example 3, as follows Figure 1 As shown in step 3), a hierarchical tagging system for the judicial field is designed to address the significant hierarchical nature of tags. This involves converting the tag data of document entities and relation_triggers into natural language, then further converting it into natural language annotations based on the annotation style, thus obtaining the tag token, such as... Figure 3 As shown:

[0365] Step 3.1) Assess the tag categories C = {C1, C2, C3, ..., C...} of the document data. m-1 C m}:

[0366] Given the significant hierarchical nature of labels in the judicial field, a hierarchical labeling system for the judicial field is designed, where the parent label C... i It contains multiple sub-tags (C1, C2, C3, ..., C...). k-1 C k If we set its tag to Pos_tag+Sup_Type_tag+Type_tag, then we will set it to Pos_tag+Sup_Type_tag+Type_tag.

[0367] (Sup_Type_tag has 3 layers).

[0368] After decomposing the element tag category, we get three parts: Pos_tag, Sup_Type_tag, and Type_tag.

[0369] First, natural language conversion is performed on Pos_tag, Sup_Type_tag, and Type_tag respectively to obtain Pos_pre_word, Sup_Type_pre_word, and Type_pre_word.

[0370] Then, based on the annotation style, it is converted into natural language annotation form to obtain Pos_word, Sup_Type_word, and Type_word.

[0371] Input Pos_word, Sup_Type_word, and Type_word into the Bert auxiliary model to obtain a self-labeled token:

[0372] G(C)=(Bert[Pos_word]+Bert[Sup_Type_word]+Bert[Type_word]) / n

[0373] Where n is the total number of Pos_tag, Sup_Type_tag, and Type_tag.

[0374] Example 5

[0375] Repeat Example 4, as follows Figure 1 As shown, step 4) involves adding tags to the Bert-NER model vocabulary, then fusing the tag tokens and term frequency tokens, and using these as tag tokens to feed the corresponding tokens of the tags added to the Bert-NER vocabulary into the model.

[0376] According to step 4.1), for the label category C = {C1, C2, C3, ..., C...} m-1 C m}, its word frequency token is T(C i ), with the tag token being G(C i ), and perform weighted fusion: Final(C i )=α1*T(C i )+α2*G(C i ); where α1 and α2 are the set weights, with a value of 0.5.

[0377] Get the dictionary mapping tags to tag tokens: Dic = {C i Final(C) i )};

[0378] Import the dictionary into the Bert-NER model.

[0379] Example 6

[0380] Repeat Example 5, as follows Figure 2 As shown, step 5) addresses document-level applications in the judicial field by redefining the NER task as an element-oriented language model classification task. This significantly accelerates decoding speed and effectively absorbs cross-sentence entity relation semantics. The Bert-NER model is used to perform NER extraction on the document, obtaining the Entity set and Relation_trigger set.

[0381] Step 5.1) For a dataset with n documents, extract each document sequentially. The longest document length is MAX_L, the shortest document length is MIN_L, and the extracted document length is L∈[MIN_L, MAX_L]. Each document contains C types of features, where C∈[MIN_C, MAX_C]. MIN_C represents the minimum number of document feature types, and MAX_C represents the maximum number of document feature types.

[0382] For a document L = {x1, x2, x3, ..., x...} m-1 x m}, where x represents a character element in the document.

[0383] First, the tokenizer of the Bert-NER model is used to segment L, and then vector embedding is performed, including token embedding, segment embedding, and position embedding.

[0384] Token embedding refers to inserting a [CLS] token at the beginning of each sentence and a [SEP] token at the end of each sentence. The [CLS] token represents the vector of the current sentence, and the [SEP] token represents the clause used to segment sentences in the text.

[0385] Segment embedding is used to distinguish between two sentences, with A and B tags preceding the different sentences respectively. Therefore, the input sentence is represented as (E A E B E A E B ,……).

[0386] Position embedding adds positional information based on the relative position of the sentence's index, such as [0,1,2,3…50,…100,…200,…].

[0387] After embedding, it passes through 12 cascaded transformer layers, and the hidden layer output vector is calculated using a multi-head attention mechanism: h = {h1, h2, h3, ..., h...} m-1 h m};

[0388] The entity tag set is: EL = {l1, l2, l3, ..., lw-1 , l w};

[0389] The set of Relation_trigger tags is: RL = {l1, l2, l3, ..., l...} q-1 , l q};

[0390] Its set of all labels is: l = EL + RL = {l1, l2, l3, ..., l m-1 , l m};

[0391] Calculate its loss:

[0392] Where P(X) i =l i |X)=softmax(h*w), where h is the output vector of the hidden layer and w is the weight of the LM linear layer (e.g., 0.2-0.8, such as 0.5).

[0393] The Bert-NER model is obtained by iteratively updating the model parameters using backpropagation of the loss function.

[0394] The trained Bert-NER model is used to perform NER extraction on the documents, obtaining the entity set EG = {e1, e2, e3, ..., e...}. p-1 e p} and the Relation_trigger set RG = {r1, r2, r3, ..., r o-1 r o}

[0395] Example 7

[0396] Repeat Example 6, as follows Figure 3 As shown, step 6) addresses the low-resource application scenario in the judicial field by designing a prompt template for the RE-MRC task. The RE-MRC task training corpus is obtained by constructing the template based on the Entity set and Relation_trigger set using the prompt. The prompt template is then used to significantly expand the data to overcome the problem of low resources in the judicial field.

[0397] The prompt template designed for RE-MRC tasks is as follows:

[0398] p1: "What is the relationship between [Entity_1] and [Entity_2]?"

[0399] p2: "What relationship does [Relation_trigger_1] represent?"

[0400] p3: "Do [Entity_1] and [Entity_2] have a relationship?"

[0401] p4: "Are [Entity_1] and [Relation_trigger_1] related?"

[0402] p5: "Are [Entity_2] and [Relation_trigger_1] related?";

[0403] The prompt template P = {p1, p2, p3, p4, p5} is obtained;

[0404] 6.2) Construction of the prompt answer space mapping:

[0405] Let the Entity set EG = {e1, e2, e3, ..., e p-1 e p} and the Relation_trigger set RG = {r1, r2, r3, ..., r o-1 r o The values ​​are filled into the prompt template P respectively to obtain the training corpus; [Z] is manually filled into the training corpus in the prompt template P as the answer space mapping to obtain the answer set [Z] = {z1, z2, z3, z4, z5}, thus obtaining the complete training corpus for the RE-MRC task;

[0406] Where: [z1], [z2], [z3], [z4], and [z5] are the answers manually judged from the training corpora p1, p2, p3, p4, and p5, respectively;

[0407] 6.3) Construction of the Bert-RE model:

[0408] For the Relation_trigger set RG = {r1, r2, r3, ..., r...} o-1 r o} and the set of entities EG = {e1, e2, e3, ..., e p-1 e p}, taking r and e as text and putting them into the prompt template as input, and using a pre-trained language model to predict the input text;

[0409] The predicted result is used as the label and compared with the answer set [Z]. The loss is calculated iteratively and backpropagated until the predicted result label matches the answer set [Z], thus obtaining the Bert-RE model.

[0410] Example 8

[0411] Repeat Example 7, as follows Figure 3 As shown, step 6.3) specifically refers to:

[0412] For a dataset with n documents, each document is extracted sequentially. The longest document has a length of MAX_L, and the shortest document has a length of MIN_L. The length of the extracted documents is L∈[MIN_L, MAX_L]. Each document contains C types of features, where C∈[MIN_C, MAX_C]. MIN_C represents the minimum number of document feature types, and MAX_C represents the maximum number of document feature types.

[0413] For a document L = {x1, x2, x3, ..., x...} m-1 x m}, where x represents a character element in the document.

[0414] For the question Q = {q1, q2, q3, ..., q...} z-1 q z}

[0415] Take out q from Q in sequence, and each q and L together form the input corpus.

[0416] First, the tokenizer of the Bert-RE model is used to segment L and q, and then vector embedding is performed, including token embedding, segment embedding, and position embedding.

[0417] Token embedding refers to inserting a [CLS] token at the beginning of each sentence and a [SEP] token at the end of each sentence. The [CLS] token represents the vector of the current sentence, and the [SEP] token represents the clause used to segment sentences in the text.

[0418] Segment embedding is used to distinguish between two sentences, with A and B tags preceding the different sentences respectively. Therefore, the input sentence is represented as (E A E B E A E B ,……).

[0419] Position embedding adds positional information based on the relative position of the sentence's index, which is [0,1,2,3…50,…100,…200,…].

[0420] After embedding, it passes through 12 cascaded transformer layers, and the hidden layer output vector is calculated using a multi-head attention mechanism: h = {h1, h2, h3, ..., h...} n-1 h n};

[0421] Its set of all labels is: l = {l1, l2, l3, ..., l n-1 , l n};

[0422] Calculate its loss:

[0423] Where P(X) i =l i |X)=softmax(h*w); h is the hidden layer output vector; w is the weight of the LM linear layer, with a value of 0.5.

[0424] The Bert-RE model is obtained by iteratively updating the model parameters using backpropagation of the loss.

[0425] Example 9

[0426] A method for feature extraction in low-resource document-level scenarios in the tourism industry based on self-labeling and prompts, comprising the following steps:

[0427] 1) Use web crawlers to acquire tourism data, and construct a basic dictionary for tourism data by statistically analyzing the data using TF-IDF. Query entities in the tourism data dictionary using the knowledge graph of Chinese tourist attractions to retrieve their neighboring edges (relationships) and nodes (entities). Add entities and relationships not in the dictionary to obtain the final dictionary for tourism data. Use the final dictionary for tourism data and large-scale unsupervised pre-training to obtain a Bert-NER teacher model. Then take a Bert model as a Bert-NER student model. Use this Bert-NER teacher model and the Bert-NER student model to self-train on large-scale unlabeled data to overcome the problem of low resources in the tourism field, and iterate to obtain the final Bert-NER model.

[0428] 2) Using the knowledge graph of Chinese tourist attractions as a remote supervision tool to annotate unlabeled data, the word frequencies of entity and relation_trigger generated by the document and the BERT-assisted model are counted separately, and they are fused to obtain the comprehensive word frequency. The top-k elements of the comprehensive word frequency of each tag are encoded and fused to obtain the word frequency token of that tag.

[0429] 3) In view of the significant hierarchical nature of labels in the tourism field, a hierarchical labeling system for the tourism field is designed. The label data of document entity and relation_trigger are converted into natural language form, and then converted into natural language annotation form according to the annotation style to obtain the self-label token.

[0430] 4) Add the tags to the Bert-NER model vocabulary, then merge the tag tokens and term frequency tokens, and use them as tag tokens. The corresponding tokens of the tags added to the Bert-NER vocabulary are then put into the model.

[0431] 5) For document-level application scenarios in the tourism field, the NER task is redefined as an element-oriented language model classification task, which significantly speeds up the decoding speed and effectively absorbs cross-sentence entity relation semantics. The Bert-NER model is used to extract NER from the documents to obtain the Entity set and the Relation_trigger set.

[0432] 6) For the application scenario of low resources in the tourism field, a prompt template for RE-MRC task is designed. The RE-MRC task training corpus is obtained by constructing the template based on the prompt using the Entity set and Relation_trigger set. The data is greatly expanded by constructing the prompt template to overcome the problem of low resources in the tourism field. The Bert-RE model is obtained by using the RE-MRC task training corpus.

[0433] 7) Use the Bert-RE model obtained in step 6) to perform RE extraction on the document to obtain the relation set, thus completing the feature extraction task for document-level low-resource scenarios.

[0434] Example 10

[0435] Repeat Example 9, except that step 1) uses a web crawler to acquire tourism data, performs TF-IDF analysis on the data to construct a basic dictionary for the tourism data, queries entities in the tourism data dictionary using the Chinese tourist attraction knowledge graph, retrieves their neighboring edges (relationships) and nodes (entities), and adds entities and relationships not in the dictionary to obtain the final tourism data dictionary. Using the final tourism data dictionary and large-scale unsupervised pre-training on the data, a Bert-NER teacher model is obtained. Another Bert model is then used as the Bert-NER student model. This Bert-NER teacher model and the Bert-NER student model are used to self-train on large-scale unlabeled data to overcome the problem of low resources in the tourism field. The final Bert-NER model is obtained iteratively, including, for example... Figure 1 As shown:

[0436] Step 2.1) The professional field of the documents to be extracted is to crawl available network resources using a big data framework based on Hadoop, while complying with laws and crawling protocols.

[0437] The unlabeled data is segmented using the BERT-assisted model, and the TF-IDF of the characters in the data is calculated:

[0438] TF i，j =(n i，j / Σ k n k，j );

[0439] Indicates the term t i In document d j The frequency of occurrence.

[0440] IDF i =log(|D| / |j:t) i ∈d j |);

[0441] The total document contains the term t i The number of documents.

[0442] TF-IDF i =TF i，j ×IDF i ;

[0443] Among them: TF i，j Indicates the term t i In document d j Frequency of occurrence in; n i，j Document d represents document d j The number of terms i in n; k，j Document d represents document d j Number of terms k in the middle; IDF i Indicates the term t i The inverse document frequency; D represents all documents; j represents the index of the j-th document; t i d represents the i-th term; j This represents the j-th document; TF-IDF i This represents the document-inverse document frequency of the i-th term;

[0444] The TF-IDF calculation results were sorted, and 1000 words were selected to form a basic data dictionary for the tourism field.

[0445] For each word in the basic dictionary of tourism data, a remote query is performed using the knowledge graph of Chinese tourist attractions. The related relationships and nearby entity nodes of the corresponding words in the knowledge graph of Chinese tourist attractions are queried. If it is not in the basic dictionary of tourism data, it is added to the basic dictionary of tourism data to form the final dictionary of tourism data.

[0446] The processed data was fed into two Bert models. The final dictionary, based on tourism data, was pre-trained using the MLM and NSP tasks to obtain a Bert-NER teacher model and a Bert-NER student model.

[0447] The MLM task involves randomly selecting 15% of the tokens in the sentences of the input corpus, replacing these tokens with [MASK], and then using the BERT model to predict what word [MASK] is.

[0448] The NSP task involves labeling sentences in the input corpus. For each sample, two sentences A and B are selected, and the task is to predict whether B is the next sentence after A.

[0449] The Bert-NER teacher model is used to predict unlabeled data, and the prediction results are used as pseudo-labels.

[0450] Then, the labeled and unlabeled data are combined with labeled and pseudo-labeled data to train the Bert-NER student model.

[0451] After that, the Bert-NER student model was used as the new Bert-NER teacher model, and the Bert-NER teacher model was used as the Bert-NER student model for the next stage.

[0452] The process was repeated 20 times to obtain the final Bert-NER model.

[0453] Example 11

[0454] Repeat Example 10, except that in step 2), the word frequencies of the document and the entity and relation_trigger generated by the BERT-assisted model are counted separately, and then fused to obtain the comprehensive word frequency. The top-k elements of the comprehensive word frequency for each tag are encoded and fused to obtain the word frequency token for that tag, such as... Figure 1 As shown:

[0455] Step 2.1) For document data with n documents, containing m feature label categories, the set of feature label categories is: {C1, C2, C3, ..., C...} m-1 C m};

[0456] Perform word frequency statistics on the document data to obtain the word frequency mapping of element label categories:

[0457] F1(C i ) = argmaxα(x = w, y = C i );

[0458] Where x represents a character in the document, w represents a specific character content, y represents the element label category, and C... i This represents the label category of the i-th element.

[0459] The Bert auxiliary model directly uses the open-source Bert-base model, which consists of 12 layers of transformer-encoders.

[0460] Calculate the word frequency of all feature label prediction results for n documents to obtain the generated word frequency mapping of feature label categories:

[0461] F2(C i ) = argmaxβ(x = w, y = C i );

[0462] Where x represents a character in the document, w represents a specific character content, and y represents the element label category.

[0463] The comprehensive word frequency is obtained by using data word frequency mapping and generated word frequency mapping:

[0464] F(C i )=η1*F1(C i )+η2*F2(C i );

[0465] η1 and η2 are the weights set, with a value of 0.5.

[0466] Using the Bert-NER model to analyze label C i The elements E that rank in the top k by overall word frequency i ={e1, e2, e3, ..., e k-1 e k The encoding is performed, where k is the set quantity, with a value of 5. Then, the average of all encodings is taken to obtain the word frequency token.

[0467] T(C i ) = Avg(Bert-NER(E i )).

[0468] Example 12

[0469] Repeat Example 11, except for step 3), which addresses the significant hierarchical nature of tags in the tourism sector by designing a hierarchical tagging system for the tourism sector. This involves converting the tag data of document entities and relation_triggers into natural language form, and then further converting it into natural language annotations based on the annotation style to obtain the tag token.

[0470] Step 3.1) Assess the tag categories C = {C1, C2, C3, ..., C...} of the document data. m-1 C m}:

[0471] Given the significant hierarchical nature of labels in the tourism sector, a hierarchical labeling system for the tourism sector is designed, with parent label C...i It contains multiple sub-tags (C1, C2, C3, ..., C...). k-1 C k If we set its tag to Pos_tag+Sup_Type_tag+Type_tag, then we will set it to Pos_tag+Sup_Type_tag+Type_tag.

[0472] (Sup_Type_tag has 3 layers).

[0473] After decomposing the element tag category, we get three parts: Pos_tag, Sup_Type_tag, and Type_tag.

[0474] First, natural language conversion is performed on Pos_tag, Sup_Type_tag, and Type_tag respectively to obtain Pos_pre_word, Sup_Type_pre_word, and Type_pre_word.

[0475] Then, based on the annotation style, it is converted into natural language annotation form to obtain Pos_word, Sup_Type_word, and Type_word.

[0476] Input Pos_word, Sup_Type_word, and Type_word into the Bert auxiliary model to obtain a self-labeled token:

[0477] G(C)=(Bert[Pos_word]+Bert[Sup_Type_word]+Bert[Type_word]) / n

[0478] Where n is the total number of Pos_tag, Sup_Type_tag, and Type_tag.

[0479] Example 13

[0480] Repeat Example 12, except that in step 4), the tags are added to the Bert-NER model vocabulary, and then the tag tokens and term frequency tokens are merged and used as tag tokens. The corresponding tokens of the tags added to the Bert-NER vocabulary are then put into the model.

[0481] According to step 4.1), for the label category C = {C1, C2, C3, ..., C...} m-1 C m}, its word frequency token is T(C i ), with the tag token being G(C i ), and perform weighted fusion: Final(C i )=α1*T(C i )+α2*G(Ci ); where α1 and α2 are the set weights, with a value of 0.5.

[0482] Obtain the dictionary mapping tags to tag tokens: Dic = {c i Final(C) i )};

[0483] Import the dictionary into the Bert-NER model.

[0484] Example 14

[0485] Repeat Example 13, except that step 5) is used for document-level applications in the tourism sector. The NER task is redefined as an element-oriented language model classification task, which significantly speeds up decoding and effectively absorbs cross-sentence entity relation semantics. The Bert-NER model is used to extract NER from the document to obtain the Entity set and the Relation_trigger set.

[0486] Step 5.1) For a dataset with n documents, extract each document sequentially. The longest document length is MAX_L, the shortest document length is MIN_L, and the extracted document length is L∈[MIN_L, MAX_L]. Each document contains C types of features, where C∈[MIN_C, MAX_C]. MIN_C represents the minimum number of document feature types, and MAX_C represents the maximum number of document feature types.

[0487] For a document L = {x1, x2, x3, ..., x...} m-1 x m}, where x represents a character element in the document.

[0488] First, the tokenizer of the Bert-NER model is used to segment L, and then vector embedding is performed, including token embedding, segment embedding, and position embedding.

[0489] Token embedding refers to inserting a [CLS] token at the beginning of each sentence and a [SEP] token at the end of each sentence. The [CLS] token represents the vector of the current sentence, and the [SEP] token represents the clause used to segment sentences in the text.

[0490] Segment embedding is used to distinguish between two sentences, with A and B tags preceding the different sentences respectively. Therefore, the input sentence is represented as (E A E B E A E B ,……).

[0491] Position embedding adds positional information based on the relative position of the sentence's index, such as [0,1,2,3…50,…100,…200,…].

[0492] After embedding, it passes through 12 cascaded transformer layers, and the hidden layer output vector is calculated using a multi-head attention mechanism: h = {h1, h2, h3, ..., h...} m-1 h m};

[0493] The entity tag set is: EL = {l1, l2, l3, ..., l w-1 , l w};

[0494] The set of Relation_trigger tags is: RL = {l1, l2, l3, ..., l...} q-1 , l q};

[0495] Its set of all labels is: l = EL + RL = {l1, l2, l3, ..., l m-1 , l m};

[0496] Calculate its loss:

[0497] Where P(X) i =l i |X)=softmax(h*w), where h is the output vector of the hidden layer and w is the weight of the LM linear layer (e.g., 0.2-0.8, such as 0.5).

[0498] The Bert-NER model is obtained by iteratively updating the model parameters using backpropagation of the loss function.

[0499] The trained Bert-NER model is used to perform NER extraction on the documents, obtaining the entity set EG = {e1, e2, e3, ..., e...}. p-1 e p} and the Relation_trigger set RG = {r1, r2, r3, ..., r o-1 r o}

[0500] Example 15

[0501] Repeat Example 14, as follows Figure 3As shown, step 6) addresses the low-resource application scenario in the tourism sector by designing a prompt template for the RE-MRC task. The template is constructed using the Entity set and the Relation_trigger set to obtain the RE-MRC task training corpus. The prompt template is then used to significantly expand the data to overcome the low-resource problem in the tourism sector.

[0502] The prompt template designed for RE-MRC tasks is as follows:

[0503] p1: "What is the relationship between [Entity_1] and [Entity_2]?"

[0504] p2: "What relationship does [Relation_trigger_1] represent?"

[0505] p3: "Do [Entity_1] and [Entity_2] have a relationship?"

[0506] p4: "Are [Entity_1] and [Relation_trigger_1] related?"

[0507] p5: "Are [Entity_2] and [Relation_trigger_1] related?";

[0508] The prompt template P = {p1, p2, p3, p4, p5} is obtained;

[0509] 6.2) Construction of the prompt answer space mapping:

[0510] Let the Entity set EG = {e1, e2, e3, ..., e p-1 e p} and the Relation_trigger set RG = {r1, r2, r3, ..., r o-1 r o The values ​​are filled into the prompt template P respectively to obtain the training corpus; [Z] is manually filled into the training corpus in the prompt template P as the answer space mapping to obtain the answer set [Z] = {z1, z2, z3, z4, z5}, thus obtaining the complete training corpus for the RE-MRC task;

[0511] Where: [z1], [z2], [z3], [z4], and [z5] are the answers manually judged from the training corpora p1, p2, p3, p4, and p5, respectively;

[0512] 6.3) Construction of the Bert-RE model:

[0513] For the Relation_trigger set RG = {r1, r2, r3, ..., r...} o-1 r o} and the set of entities EG = {e1, e2, e3, ..., e p-1 e p}, taking r and e as text and putting them into the prompt template as input, and using a pre-trained language model to predict the input text;

[0514] The predicted result is used as the label and compared with the answer set [Z]. The loss is calculated iteratively and backpropagated until the predicted result label matches the answer set [Z], thus obtaining the Bert-RE model.

[0515] Example 16

[0516] Repeat Example 15, as follows Figure 3 As shown, step 6.3) specifically refers to:

[0517] For a dataset with n documents, each document is extracted sequentially. The longest document has a length of MAX_L, and the shortest document has a length of MIN_L. The length of the extracted documents is L∈[MIN_L, MAX_L]. Each document contains C types of features, where C∈[MIN_C, MAX_C]. MIN_C represents the minimum number of document feature types, and MAX_C represents the maximum number of document feature types.

[0518] For a document L = {x1, x2, x3, ..., x...} m-1 x m}, where x represents a character element in the document.

[0519] For the question Q = {q1, q2, q3, ..., q...} z-1 q z}

[0520] Take out q from Q in sequence, and each q and L together form the input corpus.

[0521] First, the tokenizer of the Bert-RE model is used to segment L and q, and then vector embedding is performed, including token embedding, segment embedding, and position embedding.

[0522] Token embedding refers to inserting a [CLS] token at the beginning of each sentence and a [SEP] token at the end of each sentence. The [CLS] token represents the vector of the current sentence, and the [SEP] token represents the clause used to segment sentences in the text.

[0523] Segment embedding is used to distinguish between two sentences, with A and B tags preceding the different sentences respectively. Therefore, the input sentence is represented as (E A E B E A E B ,……).

[0524] Position embedding adds positional information based on the relative position of the sentence's index, such as [0,1,2,3…50,…100,…200,…].

[0525] After embedding, it passes through 12 cascaded transformer layers, and the hidden layer output vector is calculated using a multi-head attention mechanism: h = {h1, h2, h3, ..., h...} n-1 h n};

[0526] Its set of all labels is: l = {l1, l2, l3, ..., l n-1 , l n};

[0527] Calculate its loss:

[0528] Where P(X) i =l i |X)=softmax(h*w); h is the hidden layer output vector; w is the weight of the LM linear layer, with a value of 0.5.

[0529] The Bert-RE model is obtained by iteratively updating the model parameters using backpropagation of the loss.

Claims

1. An element extraction method for document-level low-resource scenarios based on self-labeling and prompts, comprising the following steps: 1) Construction of a Bert-NER model: Use a crawler to obtain professional field data, construct a professional field data basic dictionary by statistically analyzing TF-IDF, query entities in the professional field data basic dictionary through a knowledge graph, retrieve their adjacent edges "relationships" and nodes "entities", add entities and relationships not in the dictionary to the dictionary, obtain a final professional field data dictionary, obtain a Bert-NER teacher model using the final professional field data dictionary and large-scale data unsupervised pre-training, take a Bert model as a Bert-NER student model, use the Bert-NER teacher model and the Bert-NER student model to self-train large-scale unlabeled data to overcome the problem of low resources in the professional field, and iteratively obtain a final Bert-NER model; 2) Obtaining word frequency token: Use remote knowledge base remote supervision as annotation of unlabeled data, respectively, statistically analyze the word frequency of document and bert auxiliary model generated results entity and relation_trigger, fuse to obtain comprehensive word frequency, encode and fuse the top-k elements of the comprehensive word frequency of each label to obtain the word frequency token of the label; 3) Obtaining self-label token: According to the characteristic of hierarchical professional field label, a professional field hierarchical label is designed, the label data of document entity and relation_trigger are converted into natural language form, and then according to the annotation style, the natural language annotation form is converted to obtain the self-label token; Specifically: 3.1) has an element label class C = {C1, C2, C3,..., Cm} for document data: m is the number of element label classes, m-1 , m , For the characteristics of professional field tag hierarchy, a professional field layered tag is designed, and the parent tag C i contains multiple child tags (C i1 , C i2 , C i3 ,..., C ik-1 , C ik ) , then we set the tag as Pos_tag+Sup_Type_tag+Type_tag; Wherein Sup_Type_tag is zero layer or one layer or multiple layers; 3.2) After the element label category is disassembled, three parts Pos_tag, Sup_Type_tag and Type_tag are obtained; 3.3) First, the Pos_tag, Sup_Type_tag and Type_tag are converted into natural language respectively to obtain Pos_pre_word, Sup_Type_pre_word and Type_pre_word; 3.4) Then, according to the annotation style, the natural language annotation form is converted to obtain Pos_word, Sup_Type_word and Type_word; 3.5) Input the Pos_word, Sup_Type_word and Type_word into the Bert auxiliary model to obtain the self-label token: G(C)=(Bert[Pos_word]+Bert[Sup_Type_word]+Bert[Type_word]) / n; Wherein n is the total number of Pos_tag, Sup_Type_tag and Type_tag; 4) The placement of the corresponding token of the label in the above model: the label is supplemented into the Bert-NER model vocabulary, and the self-label token and the word frequency token are fused to place the label token as the corresponding token of the label supplemented into the Bert-NER vocabulary into the model; 5) Obtaining the Entity set and the Relation_trigger set: for the application scenario of professional field document level, the NER task is redefined as an element-oriented language model classification task, which significantly speeds up the decoding speed and effectively absorbs the cross-sentence entity relationship semantics. The Bert-NER model is used for NER extraction of the document to obtain the Entity set and the Relation_trigger set; 6) Large-scale expansion of corpus data: for the application scenario of low resources in the professional field, a prompt template for the RE-MRC task is designed, and the Entity set and the Relation_trigger set are used to construct the template based on the prompt to obtain the RE-MRC task training corpus. The prompt template is used to expand the corpus data on a large scale to overcome the problem of low resources in the professional field. The Bert-RE model is obtained by using the RE-MRC task training corpus; 7) Extraction of elements: the Bert-RE model obtained in step 6) is used to perform RE extraction on the document to obtain the relation set, and the element extraction task in the document level low resource scenario is completed.

2. The element extraction method according to claim 1, characterized by, Step 1) more specifically includes: 1.1) The professional field to which the document to be extracted belongs is obtained by crawling available network resources based on the hadoop big data framework while complying with legal and crawler protocols; 1.2) The unmarked data is segmented using the Bert auxiliary model, and the TF-IDF of the characters in the data is calculated; TF i，j = (n i，j j / ∑ k n k，j ); IDF i = log(|D| / |j:t i ∈d j |) TF-IDF i = TF i，j x IDF i ; where: TF i,j represents the term t i in the document d j ; n i,j represents the number of occurrences of the term i j in the document d k,j ; n j represents the number of occurrences of the term k i in all documents D; j i represents the index of the jth document; t i represents the ith term; d j represents the jth document; TF-IDF i represents the document-inverse document frequency of the ith term 1.3) The TF-IDF calculation results are sorted, and 10-5000 words are selected to form a professional field data basic dictionary; 1.4) For each word in the professional field data basic dictionary, remote query is performed through the knowledge graph, and the related relationship and adjacent entity nodes of the word on the knowledge graph are queried. If it is not in the professional field data basic dictionary, it is added to the professional field data basic dictionary to form a professional field data final dictionary; 1.5) The processed data is respectively placed into two Bert models to perform pre-training using the MLM task and the NSP task using the professional field data final dictionary to obtain a Bert-NER teacher model and a Bert-NER student model; Wherein the MLM task is to randomly select 5-40% of the tokens in the input corpus sentence, replace these tokens with [MASK], and then use the bert model to predict what the [MASK] is. The NSP task is to mark the input corpus sentence, select two sentences A and B for each sample, and predict whether B is the next sentence of A. 1.6) Use the Bert-NER teacher model to predict unlabeled data, and the prediction results are used as pseudo-labels; 1.7) Then, the labeled data and the unlabeled data are combined with the labels and the pseudo-labels to train the Bert-NER student model; 1.8) After that, the Bert-NER student model is used as a new Bert-NER teacher model, and the Bert-NER teacher model is used as a Bert-NER student model in the next stage; 1.9) Repeat the iteration 3-300 times to obtain the final Bert-NER model.

3. The element extraction method according to claim 1, characterized by, Step 2) more specifically includes: 2.1) For document data with n documents containing m element label categories, the set of element label categories: {C1, C2, C3,..., C m-1 , C m}; Count the word frequency of the document data to obtain the data word frequency mapping of the element label categories: F1(C i ) = argmaxa(x = w, y = C i ) ; wherein x is a character in the document, w represents a specific character content, y represents an element label category, C i represents the i-th element label category; wherein the Bert auxiliary model directly uses an open source Bert-base model, which is composed of 12 layers of transformer-encoder; 2.2) Count the word frequency of all element label prediction results of the n documents to obtain the generated word frequency mapping of the element label categories: F2(C i ) = argmaxβ(x = w, y = C i ) ; Where x is a character in the document, w represents a specific character content, and y represents an element label category; 2.3) Use the data word frequency mapping and the generated word frequency mapping to obtain the comprehensive word frequency: F(C i ) = η1*F1(C i ) + η2*F2(C i ) ; Where η1 and η2 are set weights, and the values are 0.2-0.8; 2.4) Using the Bert-NER model on label C i The elements E whose word frequencies are in the top k i = {e1, e2, e3,..., e k-1 , e k} are encoded, where k is the number set, taking the value of 1-10; then all the encoding is averaged to get the word frequency token: T(C i ) = Avg(Bert-NER(E i )). T(C i ) = Avg(Bert-NER(E i )).

4. The element extraction method according to claim 1, characterized by, Step 4) more specifically includes: 4.1) For label categories C = {C1, C2, C3, ..., C...} m-1 C m }, its word frequency token is T(C i ), with the tag token being G(C i ), and perform weighted fusion on it: Final(C i ) = a1 * T(C i ) + a2 * G(C i ); Where: α1, α2 are set weights, and the values are 0.2-0.8; obtain the mapping dictionary of the label category and the self-label token: Dic = {C i : Final(C i )}; 4.2) Import the dictionary into the Bert-NER model.

5. The element extraction method according to claim 1, characterized by, Step 6) more specifically includes: 6.1) The prompt template designed for the RE-MRC task is: p1: "[Entity_1] and [Entity_2] what is the relationship?"、 p2: "[Relation_trigger_1] represents what relationship?"、 p3: "[Entity_1] and [Entity_2] do they have a relationship?"、 p4: "[Entity_1] and [Relation_trigger_1] are they related?"、 p5: "[Entity_2] and [Relation_trigger_1] are they related?"; Obtain the prompt template P={p1,p2,p3,P4,p5}; 6.2) Construction of prompt answer space mapping: An Entity set EG = {e1, e2, e3,..., en} and a Relation_trigger set RG = {r1, r2, r3,..., rn} are filled into the prompt template P to obtain a training corpus; p-1 p o-1 o A training corpus of a complete RE-MRC task is obtained by filling [Z] as an answer space mapping into the training corpus in the prompt template P to obtain an answer set [Z] = {z1, z2, z3, z4, z5}.​​​ Where: [z1], [z2], [z3], [z4], [z5] are the answers obtained by artificial judgment of the training corpus p1, p2, p3, p4, p5 respectively; 6.3) Construction of Bert-RE model: For the Relation_trigger set RG = {r1, r2, r3, ..., r...} o-1 r o } and the set of entities EG = {e1, e2, e3, ..., e p-1 e p }, taking r and e as text and putting them into the prompt template as input, and using a pre-trained language model to predict the input text; Compare the predicted results with the answer set [Z], and iteratively calculate the loss, perform loss backpropagation, and update the Bert-NER model until the predicted results label and the answer set [Z] are consistent, and obtain the Bert-RE model.

6. According to the element extraction method of claim 2, characterized in that, 1.3) Sort the TF-IDF calculation results, and select 15-4500 words to form a professional field data basic dictionary; and / or In step 1.5), randomly select 10-30% of the tokens, replace these tokens with [MASK], and then use the bert model to predict what word [MASK] is; and / or 1.9) Repeat 4-100 times to obtain the final Bert-NER model.

7. The element extraction method according to claim 6, characterized in that, 1.3) Sort the TF-IDF calculation results and select 20-4000 words to form a professional field data base dictionary; and / or In step 1.5), randomly select 15-20% of the tokens, replace these tokens with [MASK], and then use the bert model to predict what word [MASK] is; and / or 1.9) Repeat 5-10 times to obtain the final Bert-NER model.

Images