An entity relation joint extraction method and system for Chinese text in the field of carbon neutralization

Abstract

A kind of entity relation joint extraction method of carbon neutral field Chinese text, for carbon neutral Chinese text, the feature representation vector of text sentence is obtained using Atom-7B large model, then the entity relation triplets in text are obtained in the form of joint extraction by PFN.A kind of entity relation joint extraction system of carbon neutral field Chinese text, for the training data of relation type annotation imbalance, the text sentence with less relation type sample amount is data enhanced using ChatGLM3 interface;Using Atom-7B large model, by adding noise in Embedding layer and adopting the strategy of hierarchical unmasking in decoder layer, the feature representation vector of Chinese text sentence is extracted;According to the difficulty of named entity recognition task and relation extraction task, dynamic loss function is added, so that PFN can pay attention to more difficult subtasks.The performance of extracting entity relation triplets in Chinese text is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of natural language processing, and in particular to a method and system for joint extraction of entity relationships in carbon neutral field Chinese text. BACKGROUND

[0002] To build a knowledge graph in the field of carbon neutrality, the first step is to obtain carbon neutral field related text, and the key is to extract relationship triples from the text. This step of extracting relationship triples is also known as relationship extraction. There are pipeline methods and joint extraction methods for relationship extraction. Joint extraction methods use entity recognition and relationship classification joint models to directly obtain entity triples with relationships. Current joint extraction methods have achieved higher performance and have received more attention. PFN (APartition Filter Network for Joint Entity and Relation Extraction, 2021) proposes a partition filter network to correctly model the bidirectional interaction between tasks, uses joint extraction mode to deeply interact between the two subtasks of named entity recognition (NER) and relationship extraction (RE), so that the two tasks promote each other and achieve excellent extraction performance.

[0003] Recently, some studies have attempted to use large language models such as ChatGPT and ChatGLM in relationship extraction tasks. They treat the relationship extraction problem as a sequence-to-sequence task and linearize the relationship between entities as a target string generated according to the input. Some studies first use ChatGPT to generate thought chains, and then fine-tune the Flan-T5Large model using the generated thought chains and data through instruction methods, achieving good results on the data set. However, this method uses manual evaluation strategies and ignores the possibility that instruction fine-tuning methods may produce small differences between predicted results and labels.

[0004] Existing work has found that instruction fine-tuning methods may not perform well in sequence labeling tasks. Unlike text generation tasks, the label space of the above tasks is limited. LS-unLLaMA (LABEL SUPERVISED LLAMA FINETUNING, 2023) uses the LLaMA large model for sequence labeling tasks. It removes the causal mask from the decoder and restores the one-way attention of LLaMA to bidirectional attention, achieving state-of-the-art performance in named entity recognition (NER). LLaMA, LS-unLLaMA, and other large models are suitable for English text, while the Atom-7B large model is an open-source large model pre-trained on large-scale Chinese data based on LLaMA2-7B, and has better language understanding and generation capabilities for Chinese text. SUMMARY

[0005] In order to overcome the deficiencies of the prior art, the present application provides a kind of carbon neutral field Chinese text entity relationship joint extraction method and system, first, for the sample of uneven distribution, using ChatGLM3 interface to the text sentence of the relationship type of sample amount less is carried out data enhancement, generate and original sample have the same relationship triplets Sample set;Then add noise to the Embedding layer of Atom-7B large model, alleviate the overfitting phenomenon of Atom-7B large model, and improve the feature extraction capability of Chinese text sentence by hierarchical unmasking;Finally, according to the difficulty of named entity recognition task and relationship extraction task is different, join a kind of dynamic loss function, make PFN module pay more attention to the task of greater difficulty in training process.

[0006] The technical scheme adopted by the present application to solve its technical problems is:

[0007] A kind of carbon neutral field Chinese text entity relationship joint extraction method, comprising the following steps:

[0008] Step one, carbon neutral Chinese text data collection: obtain carbon emission data of China and countries in the world from China carbon accounting database, global real-time carbon data, world bank database website, collect carbon emission evaluation requirements guide of each province and city, in carbon neutral information network, use the method of web crawler to crawl carbon neutral policy standards, notices and related information, get the unstructured carbon neutral Chinese text to be extracted entity relationship from specified paragraph, and given ontology constraint set, the ontology constraint set includes relationship name, head entity type and tail entity type, according to the preset proportion, divide the carbon neutral Chinese text data that has been labeled into training set D1 and verification set D2, the carbon neutral Chinese text data includes subject, object, relationship and category label contained in each sample;

[0009] Step two, data enhancement is carried out to training set D1 using ChatGLM3: count the number of times of various relationships in all triplets in training set D1, judge whether the relationship type exists long tail distribution, for the training data of uneven distribution of relationship type, use ChatGLM3 interface to carry out data enhancement to the relationship type of sample amount less, generate the sample set of original text with the same relationship triplets and join D1;

[0010] Step three, training the entity relation joint extraction model with the expanded D1: input the Chinese text sentence of D1 into the Embedding layer, add noise after obtaining the embedding vector, input the Atom-7B large model and the PFN model in turn, output the entity and relation prediction value corresponding to the text sentence, and calculate the dynamic loss function by automatically adjusting the loss weight of named entity recognition and relation extraction, and optimize the parameters of the entity relation joint extraction model according to the loss gradient, and save the model parameters with the best extraction performance on the validation set D2;

[0011] Step four: input the unlabeled carbon neutral Chinese text sentence into the trained Atom-7B large model to obtain the feature representation vector of the text sentence, and then input the feature representation vector into the PFN module to output the entity relation triple.

[0012] Further, in step two, for the relationship type with less sample quantity in D1, the sample s0 containing the relationship type is selected, the ChatGLM3 large model interface is called, and the sample set with the same relationship triple as the original sample is generated in the form of dialogue, and is added to D1, the process is as follows:

[0013] 2.1 input "Please translate from Chinese to English, keeping the word # unchanged:%" into the ChatGLM3 large model interface, wherein # represents the relationship triple triple_list in the text, and % represents the text sentence s0 to be expanded;

[0014] 2.2 the ChatGLM3 large model interface outputs the sentence s′0 translated into English;

[0015] 2.3 input "Please translate the sentence into Chinese: $" into the ChatGLM3 large model interface, wherein $ represents s′0;

[0016] 2.4 the ChatGLM3 large model interface outputs the sentence translated into Chinese, that is, the expanded text sentence s1;

[0017] 2.5 repeat 2.2-2.4, add the expanded text sentence set {s1,…,s m} and its relationship triple label as a sample set to D1.

[0018] The process of step three is as follows:

[0019] 3.1 add noise obeying uniform distribution to the Embedding layer of the Atom-7B large model: input the text sentence s j , pass through the Tokenizer AtomTransformed into token sequence vector t, a noise vector ∈ is randomly generated and scaled to the set strength, added to the embedding vector t to obtain the noise-added embedding vector x as

[0020] x = Embedding Atom (t) + a∈

[0021] t = Tokenizer Atom (s j )

[0022] ∈ ~ Uniform (-1, 1)

[0023] Where s j is the text sentence corresponding to the jth sample in D1, Embedding Atom is the Embedding layer of Atom-7B large model, ∈ is noise obeying uniform distribution Uniform, and a is the scaling factor of noise;

[0024] 3.2 Input x into the Atom-7B large model decoder layer Decoder Layers based on hierarchical unmasking, and output the text sentence s i All token representation vectors h: Perform hierarchical operations on the decoder layers of the Atom-7B large model, and the decoder layers are denoted as W0 is the frozen pre-trained decoder layer parameter, ΔW is the decoder layer parameter fine-tuned by LoRA technology, the first to l layers, i.e. Atom 1:l No unmasking, the l+1 to n layers, i.e. BiAtom l+1:n Remove the mask to obtain the feature representation vector h of all tokens as

[0025]

[0026]

[0027]

[0028]

[0029]

[0030] Q = xW q

[0031] K = xW k

[0032] V = xW v

[0033] ΔW = W down W up

[0034] wherein is the i-th head of multi-head self-attention in the 1st to lth layer, is the i-th head of multi-head self-attention in the (l+1)th to nth layer, M is the mask causal mask of Atom-7B large model, -∞ represents that all tokens after the current token pair are invisible, that is, only one-way attention is calculated, M' is the mask after removing -∞, that is, bidirectional self-attention is calculated, Q, K and V are Query, Key and Value in the multi-head self-attention mechanism, SoftMax is an activation function, d k is the dimension size of K, W q , W k , W v are three trainable parameter matrices, W down is the dimension reduction matrix of LoRA fine-tuning, W up is the dimension increase matrix of LoRA fine-tuning, the parameters fine-tuned by LoRA technology include the three modules of q_proj, k_proj and v_proj of the decoder layer, the parameter update of the q_proj, k_proj and v_proj modules will affect W q , W k , W v these three parameter matrices;

[0035] 3.3 Input the representation vector h of all tokens into the PFN model to obtain the prediction values of entities and relations respectively and wherein represents the probability that the token pair starting with the i-th token and ending with the j-th token belongs to an entity of type k, represents the probability that the i-th token and the j-th token belong to the subject entity and the object entity starting word of the relationship type l, and a dynamic loss function is calculated, that is, at the beginning of training, the loss of named entity recognition and relationship extraction is added as the total loss, and the loss weight of named entity recognition and relationship extraction is automatically adjusted during the training process, giving a larger weight to the sub-task with a larger loss, so that the model pays more attention to the more difficult task during the training process:

[0036]

[0037]

[0038]

[0039]

[0040] wherein, θ is the parameter of the PFN model, and are the labels of entity and relation respectively, S and T are the sets of all entity labels and relation labels respectively, L ner and L re are the losses of the named entity recognition subtask and the relation extraction subtask respectively, BCELoss is the binary cross-entropy loss, σ represents the sigmoid activation function, exp represents the exponential operation, and λ is the threshold value;

[0041] 3.4 Jointly optimizing the parameters of the entity-relation joint extraction model: according to the gradient value of the loss function, fine-tuning part of the parameters ΔW of the Atom-7B large language model using the LoRA technology, and fully fine-tuning the parameters θ of the PFN model;

[0042] 3.5 Repeat steps 3.1-3.4 until the maximum number of iterations is reached, while saving the parameters of the entity-relation joint extraction model with the highest F1 score on the validation set D2.

[0043] An entity-relation joint extraction system for carbon neutral field Chinese text, the system comprises:

[0044] A carbon neutral data collection module is configured to obtain carbon emission and carbon neutral related data, obtain unstructured carbon neutral Chinese text to be extracted entity-relation from a specified paragraph, and obtain a given ontology constraint set, the ontology constraint set includes a relation name, a head entity type and a tail entity type, divide the labeled carbon neutral Chinese text data into a training set D1 and a validation set D2 according to a preset proportion, and the Chinese text data includes a subject, an object, a relation and a category label contained in the current sample;

[0045] A training set data enhancement module is configured to preprocess the training set D1, count the number of occurrences of various relations in all relation triples in the training set D1, determine whether the relation type has a long-tail distribution, and perform data enhancement on the relation type with a small sample quantity using the ChatGLM3 interface for the imbalanced relation type, and the generated text and the original text have the same relation triple;

[0046] A joint extraction model training module is configured to obtain feature representation vectors of text sentences in the training set D1 by sequentially passing the text sentences through an Embedding layer containing added noise and a hierarchical de-masked decoder layer in the Atom-7B large model, obtain corresponding entity and relation prediction probabilities after inputting the feature representation vectors of the text sentences into the PFN, automatically adjust the loss weights of the named entity recognition and the relation extraction through a dynamic loss function, fine-tune part of the parameters of the Atom-7B large language model using the LoRA technology in the joint optimization, fully fine-tune the PFN model, and save the model parameters with the best extraction performance on the validation set D2;

[0047] An entity relationship triple output module is configured to input the unlabeled carbon neutral Chinese text into the trained entity relationship joint extraction model, obtain a feature representation vector of a text sentence through the Atom-7B large model, and then output an entity relationship triple through the PFN.

[0048] The technical concept of the present application is that in the training set data preprocessing step, the number of occurrences of various relationship types is counted, and for the relationship type unbalanced samples in the carbon neutral text data, the ChatGLM3 large model interface is used to expand the samples; in the entity relationship extraction model training stage, for the Atom-7B large model suitable for Chinese, noise is added in the Embedding layer and a hierarchical unmasking decoder layer is used to better stimulate the feature extraction capability of the Atom-7B large model for Chinese text sentences, and a dynamic loss function is added to the PFN module to automatically adjust the loss weight of the named entity recognition subtask and the relationship extraction subtask during the training process, so that the PFN model can focus on the more difficult subtask.

[0049] The beneficial effects of the present application are that data augmentation is performed using the ChatGLM3 interface to alleviate the imbalance problem of the training set data of the carbon neutral field Chinese text. The Atom-7B open source large model suitable for Chinese is used to obtain the feature representation vector of the sentence, noise is added in the Embedding layer, and a hierarchical unmasking strategy is used in the decoder layer to better stimulate the Atom-7B large model to extract the feature representation capability of the Chinese text sentence. For the different difficulty levels of the named entity recognition task and the relationship extraction task, a dynamic loss function is added to make the PFN model focus on the more difficult subtask. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 The method block diagram of the present application.

[0051] Figure 2 The data augmentation block diagram of the present application. DETAILED DESCRIPTION

[0052] The present application will be further described below with reference to the accompanying drawings.

[0053] REFERENCE Figure 1 and Figure 2 A carbon neutral field Chinese text entity relationship joint extraction method, comprising the following steps:

[0054] Step one, collection of carbon neutral field Chinese text data: obtain carbon emission data of China and countries around the world from China carbon accounting database, global real-time carbon data, world bank database website, collect carbon emission evaluation requirements of each province and city, use web crawler method to crawl carbon neutral policy standards, notices and relevant information from carbon neutral information network, get unstructured carbon neutral Chinese text to be extracted entity relationship from specified paragraphs, and given ontology constraint set, the ontology constraint set includes relationship name, head entity type and tail entity type, divide the labeled carbon neutral Chinese text data into training set D1 and validation set D2 according to the preset proportion, the carbon neutral Chinese text data includes subject, object, relationship and category label contained in each sample;

[0055] Step two, data enhancement of training set D1 by ChatGLM3: count the number of times of various relationships in all triples in training set D1, judge whether the relationship type has long tail distribution, for the training data with unbalanced relationship type, use ChatGLM3 interface to enhance the data of relationship type with small sample quantity, generate sample set with the same relationship triple of original text and add it to D1;

[0056] In the step two, for the relationship type with small sample quantity in D1, select the sample s0 containing the relationship type, call ChatGLM3 large model interface, generate sample set with the same relationship triple as the original sample through dialogue, and add it to D1, the process is as follows:

[0057] 2.1 input "Please translate from Chinese to English, keeping the word # unchanged:%" to ChatGLM3 large model interface, where # represents relationship triple triple_list in text, and%represents text sentence s0 to be expanded;

[0058] 2.2 ChatGLM3 large model interface outputs the sentence s′0 translated into English;

[0059] 2.3 input "please translate the sentence into Chinese: $" to ChatGLM3 large model interface, where $ represents s′0;

[0060] 2.4 ChatGLM3 large model interface outputs the sentence translated into Chinese, that is, the expanded text sentence s1;

[0061] 2.5 repeat 2.2-2.4, add the expanded text sentence set {s1,…,s m} and its relationship triple label as sample set to D1.

[0062] Step three, training the entity relation joint extraction model with the expanded D1: input the Chinese text sentence of D1 into the Embedding layer, add noise after obtaining the embedding vector, input the Atom-7B large model and the PFN model in turn, output the entity and relation prediction value corresponding to the text sentence, and calculate the dynamic loss function by automatically adjusting the loss weight of named entity recognition and relation extraction, and optimize the parameters of the entity relation joint extraction model according to the loss gradient, and save the model parameters with the best extraction performance on the validation set D2;

[0063] The process of step three is as follows:

[0064] 3.1 Add noise conforming to uniform distribution to the Embedding layer of the Atom-7B large model: input the text sentence s j , convert it into a token sequence vector t through Tokenizer Atom , randomly generate a noise vector ∈, and scale it to the set intensity, add it to the embedding vector t to obtain the noise-added embedding vector x as

[0065] x=Embedding Atom (t)+α∈

[0066] t=Tokenizer Atom (s j )

[0067] ∈~Uniform(-1,1)

[0068] Where s j is the text sentence corresponding to the jth sample in D1, Embedding Atom is the Embedding layer of the Atom-7B large model, ∈ is the noise conforming to uniform distribution Uniform, and α is the scaling factor of the noise;

[0069] 3.2 Input x into the Atom-7B large model decoder layer Decoder Layers based on hierarchical unmasking, output the representation vector h of all tokens of the text sentence s i : perform hierarchical operation on the decoder layer of the Atom-7B large model, and the decoder layer is denoted as W0 is the frozen pre-trained decoder layer parameter, ΔW is the decoder layer parameter fine-tuned by LoRA technology, the first to l layers, i.e. Atom 1:l is not unmasked, and the l+1 to n layers, i.e. BiAtom l+1:n , are unmasked, the total number of layers n=32, and the feature representation vector h of all tokens is obtained as

[0070]

[0071]

[0072]

[0073]

[0074]

[0075] Q = xW q

[0076] K = xW k

[0077] V = xW v

[0078] AW = W down W up

[0079] wherein is the i-th head of multi-head self-attention in the 1st to l-th layers, is the i-th head of multi-head self-attention in the l+1th to n-th layers, M is the mask causal mask of Atom-7B large model, -∞ represents that all tokens after the current token pair are invisible, that is, only one-way attention is calculated, M' is the mask after removing -∞, that is, bidirectional self-attention is calculated, Q, K and V are Query, Key and Value in the multi-head self-attention mechanism, SoftMax is an activation function, d k is the dimension size of K, W q , W k , W v are three trainable parameter matrices, W down is the dimension reduction matrix of LoRA fine-tuning, W up is the dimension increase matrix of LoRA fine-tuning, the parameters fine-tuned by LoRA technology include the three modules q_proj, k_proj and v_proj of the decoder layer, the parameter update of the q_proj, k_proj and v_proj modules will affect W q , W k , W v three parameter matrices;

[0080] 3.3 Input the representation vector h of all tokens into the PFN model to obtain the prediction values of entities and relations respectively and wherein represents the probability that the token pair starting with the i-th token and ending with the j-th token belongs to an entity of type k, denotes the probability that the i-th token and the j-th token belong to the subject entity and the object entity starting word of the relationship type l, and a dynamic loss function is calculated, that is, at the beginning of training, the losses of named entity recognition and relationship extraction are added as the total loss, the loss weights of named entity recognition and relationship extraction are automatically adjusted during the training process, a larger weight is given to the subtask with a larger loss, and the model pays more attention to the more difficult task during the training process:

[0081]

[0082]

[0083]

[0084]

[0085] where θ is the parameter of the PFN model, and are the labels of entities and relationships, respectively, S and T are the sets of all entity labels and relationship labels, respectively, and L ner and L re are the losses of the named entity recognition subtask and the relationship extraction subtask, respectively, BCELoss is the binary cross-entropy loss, σ represents the sigmoid activation function, exp represents the exponential operation, and the threshold λ = 100;

[0086] 3.4 Jointly optimize the parameters of the entity relationship joint extraction model: according to the gradient value of the loss function, fine-tune part of the parameters ΔW of the Atom-7B large language model using the LoRA technology, and fine-tune the parameters θ of the PFN model.

[0087] 3.5 Repeat steps 3.1-3.4 until the maximum number of iterations is reached, while saving the parameters of the entity relationship joint extraction model with the highest F1 score on the validation set D2.

[0088] Step four: input the unannotated carbon neutral field Chinese text sentence into the trained Atom-7B large model, obtain the feature representation vector of the text sentence, and then input the feature representation vector into the PFN module to output the entity relationship triple.

[0089] The embodiment also provides an entity relationship joint extraction method and system for carbon neutral field Chinese text, which comprises a carbon neutral data collection module, a training set data enhancement module, a joint extraction model training module (including an Atom-7B unit, a PFN unit and a joint optimization unit), and an entity relationship triple output module. The above modules correspond to the contents of steps one to four of the method of the present application in sequence, that is:

[0090] The system comprises:

[0091] A carbon neutral data collection module is configured to obtain carbon emission and carbon neutral related data, obtain unstructured carbon neutral Chinese text to be extracted entity relationship from a specified paragraph, and obtain a given ontology constraint set, which comprises a relationship name, a head entity type and a tail entity type, divide annotated carbon neutral Chinese text data into a training set D1 and a validation set D2 according to a preset proportion, and the Chinese text data comprises a subject, an object, a relationship and a category label contained in a current sample.

[0092] A training set data enhancement module is configured to preprocess the training set D1, count the number of occurrences of various relationships in all relationship triples in the training set D1, determine whether the relationship type has a long-tail distribution, perform data enhancement on a relationship type with a small sample amount using a ChatGLM3 interface for an imbalanced relationship type, and generate a text and an original text with the same relationship triple.

[0093] A joint extraction model training module is configured to obtain a feature representation vector of a text sentence of the training set D1 by sequentially passing the text sentence through an Embedding layer containing added noise and a decoder layer with hierarchical unmasking in an Atom-7B large model, obtain a corresponding entity and relationship prediction probability after inputting the feature representation vector of the text sentence into a PFN, automatically adjust the loss weights of named entity recognition and relationship extraction through a dynamic loss function, fine-tune part of the parameters of the Atom-7B large language model using LoRA technology, fine-tune all parameters of the PFN model, and save the model parameters with the best extraction performance on the validation set D2.

[0094] An entity relationship triple output module is configured to input carbon neutral domain Chinese text without annotation into a trained entity relationship joint extraction model, obtain a feature representation vector of a text sentence of the model through the Atom-7B large model, and then output an entity relationship triple through the PFN.

[0095] As described above, the specific implementation steps of the patent implementation make the invention clearer. Any modifications and changes made to the invention within the spirit and protection scope of the claims fall within the protection scope of the invention.

Claims

1. A method for joint extraction of entity relationships in Chinese text in the field of carbon neutrality, characterized by, The method comprises the following steps: Step one, carbon neutralization Chinese text data collection: obtain carbon emission data of China and countries around the world from China carbon accounting database, global real-time carbon data, world bank database website, collect carbon emission evaluation requirement guidelines of each province and city, use web crawler method to crawl carbon neutralization policy standards, notices and relevant information from carbon neutralization information network, obtain unstructured carbon neutralization Chinese text to be extracted entity relationship from specified paragraphs, and given ontology constraint set, the ontology constraint set includes relationship name, head entity type and tail entity type, divide the labeled carbon neutralization Chinese text data into training set D1 and validation set D2 according to the preset proportion, the carbon neutralization Chinese text data includes subject, object, relationship and category label contained in each sample; Step two, data enhancement of training set D1 by ChatGLM3: count the number of various relationships appearing in all triples in training set D1, judge whether the relationship type has long tail distribution, for the unbalanced training data of relationship type, use ChatGLM3 interface to enhance the data of relationship type with small sample quantity, and generate sample set with the same relationship triple of original text to join D1; Step three, training entity relationship joint extraction model with expanded D1: input the Chinese text sentence of D1 into Embedding layer, add noise after obtaining the embedding vector, input Atom-7B large model and PFN model in turn, output the entity and relationship prediction value corresponding to the text sentence, calculate the dynamic loss function by automatically adjusting the loss weight of named entity recognition and relationship extraction, jointly optimize the parameters of the entity relationship joint extraction model according to the loss gradient, and save the model parameters with the best extraction performance on the validation set D2; Step four: input the Chinese text sentence without labeling into the trained Atom-7B large model, obtain the feature representation vector of the text sentence, and then input the feature representation vector into the PFN module to output the entity relationship triple.

2. The method of claim 1, wherein the method comprises: In step two, for the relationship type with small sample quantity in D1, the sample s0 containing the relationship type is selected, the ChatGLM3 large model interface is called, the sample set with the same relationship triple as the original sample is generated through the form of dialogue, and is added to D1, the process is as follows: 2.1 input "Please translate from Chinese to English, keeping the word unchanged: %" to ChatGLM3 large model interface, wherein # represents relationship triple triple_list in text, and % represents text sentence s0 to be expanded; 2.2 ChatGLM3 large model interface outputs the sentence s'0 translated into English; 2.3 input "Please translate the sentence into Chinese: $" to ChatGLM3 large model interface, wherein $ represents s'0; 2.4 ChatGLM3 large model interface outputs the sentence translated into Chinese, that is, the expanded text sentence s1; 2.5 Repeat 2.2-2.4, adding the augmented set of text sentences {s1,..., s m} and their relationship triple tags as a sample set to D1.

3. The method of claim 1 or 2, wherein the method comprises: The process of step three is as follows: 3.1 Add noise subject to uniform distribution in the Embedding layer of Atom-7B large model: input text sentence s j , convert to token sequence vector t by Tokenizer Atom , randomly generate a noise vector ∈, and scale to the set intensity, add to the embedding vector t to get the noise-added embedding vector x x = Embedding Atom (t) + a e ∈ ~ Uniform(-1, 1) wherein s j is the text sentence corresponding to the jth sample in D1, Embedding Atom is the Embedding layer of the Atom-7B large model, is noise obeying a uniform distribution, and is a scaling factor of the noise. 3.2 input x into the Atom-7B large model decoder layer Decoder Layers, output text sentence s i representation vector h of all tokens: the decoder layer of the Atom-7B large model is operated in a hierarchical manner, and the decoder layer is denoted as W0 is the frozen pre-trained decoder layer parameter, △W is the decoder layer parameter fine-tuned by LoRA technology, the first to the l-th layer, that is, Atom 1:l without unmasking, the l+1-th to n-th layer, that is, BiAtom l+1:n remove the mask, get the feature representation vector h of all tokens ΔW = W down W up wherein is the i-th head of multi-head self-attention in the 1st to l-th layers, is the i-th head of multi-head self-attention in the (l+1)th to n-th layers, M is the mask of Atom-7B large model, -∞ represents that all tokens after the current token pair are invisible, that is, only one-way attention is calculated, M' is the mask after removing -∞, that is, two-way self-attention is calculated, Q, K and V are Query, Key and Value in the multi-head self-attention mechanism, SoftMax is an activation function, d k is the dimension size of K, W q , W k , W v are three trainable parameter matrices, W down is the dimension reduction matrix of LoRA fine-tuning, W up is the dimension increase matrix of LoRA fine-tuning, the parameters fine-tuned by LoRA technology include the three modules of q_proj, k_proj and v_proj of the decoder layer, the parameter update of the q_proj, k_proj and v_proj modules will affect W q , W k , W v These three parameter matrices; 3.3 Input the representation vector h of all tokens into the PFN model to obtain the prediction value of entity and relation respectively and wherein represents the probability that the token pair starting with the i th token and ending with the j th token belongs to the entity of type k, represents the probability that the i th token and the j th token belong to the subject entity and the object entity starting word of the relation type l, and a dynamic loss function is calculated, that is, at the beginning of training, the loss of named entity recognition and relation extraction is added as the total loss, and the loss weight of named entity recognition and relation extraction is automatically adjusted during the training process, and the weight of the sub task with greater loss is greater: where θ is the parameter of the PFN model, and are the labels of entity and relation, respectively, S and T are the sets of all entity labels and relation labels, respectively, L ner and L re are the losses of the named entity recognition subtask and the relation extraction subtask, respectively, BCELoss is the binary cross-entropy loss, σ denotes the sigmoid activation function, exp denotes the exponential operation, and λ is the threshold value. 3.4 Joint optimization of entity relation joint extraction model parameters: according to the gradient value of the loss function, fine-tune part of the parameters △W of the Atom-7B large language model using LoRA technology, and fine-tune the parameters θ of the PFN model; 3.5 Repeat steps 3.1-3.4 until the maximum number of iterations is reached, while saving the parameters of the entity relation joint extraction model with the highest F1 score on the validation set D2.

4. A system for realizing the joint extraction of entity relationships in the carbon neutral field Chinese text as claimed in claim 1, characterized in that, The system comprises: A carbon neutral data collection module is configured to obtain carbon emission and carbon neutralization data, obtain unstructured carbon neutralization Chinese text to be extracted entity relation from a specified paragraph, and obtain a given ontology constraint set, wherein the ontology constraint set comprises a relation name, a head entity type, and a tail entity type, divide the labeled carbon neutralization Chinese text data into a training set D1 and a validation set D2 according to a preset proportion, and the Chinese text data comprises a subject, an object, a relation, and a category label contained in a current sample; A training set data enhancement module is configured to preprocess the training set D1, count the number of occurrences of various relations in all relation triples in the training set D1, determine whether the relation type has a long-tail distribution, perform data enhancement on the relation type with a small sample size using a ChatGLM3 interface for the relation type with an unbalanced annotation, and generate a text and an original text with the same relation triple; A joint extraction model training module is configured to obtain a feature representation vector of a text sentence of the training set D1 by sequentially passing the text sentence through an Embedding layer containing added noise and a decoder layer with hierarchical unmasking in an Atom-7B large model, obtain corresponding entity and relation prediction probabilities after inputting the feature representation vector of the text sentence into a PFN, automatically adjust the loss weights of named entity recognition and relation extraction through a dynamic loss function, fine-tune part of the parameters of the Atom-7B large language model using LoRA technology in joint optimization, fine-tune the PFN model with full parameters, and save the model parameters with the best extraction performance on the validation set D2; An entity relation triple output module is configured to input carbon neutralization Chinese text without annotation into the trained entity relation joint extraction model, obtain a feature representation vector of a text sentence through an Atom-7B large model, and then output an entity relation triple through a PFN.

Images