Method for identifying patent named entities in fields of carbon capture, utilization and storage based on ERNIE-BiLSTM-FNNAtention-CRF model

Abstract

The invention provides a patent named entity recognition method in the fields of carbon capture, utilization and storage based on an ERNIE-BiLSTM-FNNAtention-CRF model, and relates to the fields of natural language processing and patent information processing. A processed patent corpus CCUSdataset is pre-trained in an ERNIE layer, an input text is vectorized, context features are acquired in combination with BiLSTM, weighted average processing is performed on an operation vector of a BiLSTM layer by adopting FNNAttention, long text semantic information is enhanced, meanwhile, the problem that the same patent entity is inconsistent in annotation in different statements is avoided, and the semantic information of the long text is enhanced. And finally, sequence decoding labeling is carried out through a conditional random field CRF layer, and a prediction label with the highest score is output. According to the method, the problems of more special text content, unclear entity boundary, many professional vocabularies, polysemy and the like of patent texts are solved, word vectors with rich semantics can be obtained, semantic correlation between current information and context information is enhanced, potential semantic features between the texts are mined, a good recognition effect is achieved, and the method is suitable for popularization and application. And support is provided for patent value evaluation, patent analysis and retrieval and the like.

Description

Technical Field

[0001] The present invention relates to the fields of natural language processing and patent information processing, and in particular to a patent named entity recognition method in the field of carbon capture, utilization and storage based on an ERNIE-BiLSTM-FNNAttention-CRF model. Background Art

[0002] Carbon capture, utilization, and storage (CCUS) technology is an important means of addressing climate change internationally, and is of great significance to achieving carbon reduction targets and promoting sustainable development. Patents, as the knowledge carrier of scientific and technological innovation results, contain important scientific and technological details. With the influence of innovation-driven development, the number of patent applications in my country is also increasing year by year. How to mine important technical information and the relationship between technologies from these large and complex unstructured texts is a research hotspot in the current field of patent analysis. Traditional patent analysis is carried out manually, which has many disadvantages, such as long time, low efficiency, and high cost. With the continuous development of science and technology, the use of computers to store, process and analyze massive data has become the mainstream.

[0003] How to efficiently process the information contained in the text is a difficult but meaningful problem that has attracted extensive research in academia and industry. As one of the key technologies for text information processing, named entity recognition has seen an endless stream of related research methods in recent years, which can effectively solve the problem of identifying and extracting specific words in the text. This task is one of the important basic tasks in the field of natural language processing and a core technology in the field of artificial intelligence. It plays an important supporting role in many natural language processing downstream tasks such as syntactic analysis, text classification, automatic question answering, and knowledge graphs. Early machine learning technology laid a good foundation for named entity recognition technology. Now, recognition methods based on deep learning have improved the accuracy of entity recognition and have become mainstream.

[0004] However, in the process of feature extraction, named entity recognition does not deeply explore the hidden features of sentences, and the feature level of focus is too narrow, resulting in incomplete extracted features; improving the efficiency and accuracy of knowledge extraction from unstructured or semi-structured texts, promoting the efficient use of technical texts, and using knowledge graphs to extract and represent details and semantic information in patent texts, contributing to the development of patent analysis. It provides theoretical and technical support for the construction of high-quality and efficient patent knowledge graphs, provides new solutions for the efficient use of patent information resources in the CCUS field and the connection between unstructured patent text data, and helps the intelligent construction of patents in the CCUS field. Summary of the invention

[0005] The main technical problems solved by the present invention are: In view of the more special text content, language specifications, unclear entity boundaries, multiple professional vocabulary, and polysemy of patent texts, a patent named entity recognition method in the field of carbon capture, utilization, and storage based on the ERNIE-BiLSTM-FNNAttention-CRF model is proposed. The core information of the patent is extracted by natural language processing and deep learning technology, which improves the efficiency and accuracy of extracting knowledge from unstructured or semi-structured texts, promotes the efficient use of technical texts, and contributes to the development of patent analysis. It provides theoretical and technical support for the construction of high-quality and efficient patent knowledge graphs, provides new solutions for the efficient use of patent information resources in the CCUS field and the realization of the connection between unstructured patent text data, and provides knowledge support for patent value assessment, patent analysis and retrieval, etc.

[0006] In order to solve the above technical problems, a technical solution adopted by the present invention is: constructing a patent corpus CCUS_dataset, generating a vectorized representation of the input corpus based on a fine-tuned multi-layer bidirectional Transformer encoder ERNIE model, combining the bidirectional long short-term memory network BiLSTM to obtain context features, and using the forward neural network attention mechanism FNNAttention to perform weighted averaging on the BiLSTM layer operation vectors to strengthen the semantic information of long texts, while avoiding the problem of inconsistent annotations of the same patent entity in different sentences, and finally performing sequence decoding and annotation through the conditional random field CRF layer to output the predicted label with the highest score, thereby improving the model recognition accuracy. Provide knowledge support for patent achievement transformation, patent value assessment, patent analysis and retrieval, etc. It specifically includes the following steps:

[0007] Step S1: Using patents in the patent database as data sources, searching for the keyword "CCUS+carbon capture+carbon utilization+carbon storage", obtaining relevant patent texts, taking the abstracts in the patent texts as the main corpus, performing data cleaning, corpus classification, boundary clarification, and BIO annotation, and then constructing the patent corpus CCUS_dataset;

[0008] Step S2: Use the ERNIE model to represent the preprocessed patent text as a word vector. ERNIE can capture contextual information and generate semantically rich word embeddings through a pre-trained model. The ERNIE model determines whether the patent context has semantic cohesion, or predicts the masked words after applying a masking strategy to different granularities of words, characters or noun entities; each word vector contains semantic information and contextual relationships, which can capture the actual meaning of key terms in patent documents as input features for subsequent BiLSTM networks;

[0009] Step S3: For the word vector input from step S2, BiLSTM can learn context features more deeply, so as to obtain more comprehensive semantic information. LSTM is an improved solution for traditional RNN, which controls the forgetting, input and output of information by introducing memory units and threshold mechanisms.

[0010] Step S4: Use the feedforward neural network attention mechanism FNNAttention to calculate the semantic correlation between long-distance word vectors in the BiLSTM layer, automatically learn to adjust the semantic weights, and strengthen word-level relationships. FNNAttention can effectively utilize the semantic information between the contexts of long text paragraphs, obtain the global information vector, integrate it into the current feature vector for calculation, and mainly allocate attention to keywords;

[0011] Step S5: Use CRF as the output layer of the neural network, and conditionally constrain the output of all FNNAttention layers through the CRF layer. The CRF layer can use the relationship between adjacent annotation results to output the optimal tag sequence in the full text. Its powerful sequence modeling ability enables it to learn the constraints in the encoding process, thereby ensuring that the final prediction result has a high accuracy;

[0012] In step S1, “CCUS+carbon capture+carbon utilization+carbon storage” is used as the search keyword to retrieve patent texts related to the CCUS field, and the abstracts in the patent texts are used as the main corpus for related experiments. After data cleaning, corpus classification, boundary clarification, and BIO annotation, the patent corpus CCUS_dataset is constructed;

[0013] In step S2, the ERNIE model is used to represent the patent text preprocessed in step S1 with word vectors. ERNIE can capture contextual information and generate semantically rich word embeddings through a pre-trained model. The patent text information is input into the ERNIE model to generate contextual word embedding representations for each word as input features for the subsequent BiLSTM network.

[0014] In step S3, for the word vector input from step S2, the BiLSTM layer generates forward and backward hidden state sequences for the input sequence, and the forward and reverse LSTM layers generate forward and backward hidden state sequences respectively. The final output of BiLSTM is the connection of these two hidden state sequences.

[0015] In step S4, the feedforward neural network attention mechanism FNNAttention is used to calculate the semantic association between long-distance word vectors in the BiLSTM layer, automatically learn to adjust the semantic weights, and strengthen word-level relationships. FNNAttention can effectively utilize the semantic information between the contexts of long text paragraphs, obtain the global information vector, integrate it into the current feature vector for calculation, and mainly allocate attention to keywords;

[0016] In step S5, CRF replaces the traditional softmax as the output layer of the neural network. The output of step S4 is conditionally constrained by the CRF layer, and then the sequence with the highest score is selected from the CRF layer for final output, which can effectively reduce the probability of incorrectly predicting the sequence.

[0017] The beneficial effects of the present invention are as follows: first, at the pre-training level, the BERT model only learns information related to language text, and does not integrate knowledge information into it. The RRNIE model learns the semantic relationship of the real world by modeling prior semantic knowledge such as entity concepts in massive data. The ERNIE model masks semantic units such as words and entities, so that the model learns the semantic representation of complete concepts. In knowledge-driven tasks, the effect of the ERNIE pre-trained language model will be better than the BERT model, and it can simultaneously aggregate the context and knowledge entity information to build a knowledge-based language representation model.

[0018] Secondly, in neural networks, since RNN (recurrent neural network) is prone to gradient vanishing and gradient exploding problems when processing long sequences, the LSTM structure controls the flow of information by introducing a gating mechanism (input gate, forget gate, output gate), which can better capture long-term dependencies. The invention chooses to splice the forward and backward LSTM groups in series to obtain a BiLSTM model to obtain deep information of the context, which can capture the contextual relationship more comprehensively.

[0019] Finally, we added an FNNAttention layer based on the improved existing model. The biggest advantage of the FNNAttention layer is that it can effectively utilize the semantic information between the contexts of long text paragraphs, obtain the global information vector, integrate it into the current feature vector for calculation, and allocate attention mainly to keywords. While solving the problem of inconsistent annotations of the same patent entity in different sentences in long texts, it also strengthens the word-level relationship of the text.

[0020] In summary, the patent named entity recognition method in the field of carbon capture, utilization and storage based on the ERNIE-BiLSTM-FNNAttention-CRF model proposed in the present invention has more accurate precision, strong portability and generalization ability of text topics, and at the same time provides support for patent value assessment, patent analysis and retrieval, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of the process of a patent named entity recognition method in the field of carbon capture, utilization and storage based on the ERNIE-BiLSTM-FNNAttention-CRF model of the present invention;

[0022] Figure 2 This is a schematic diagram of a model in a patent named entity recognition method in the field of carbon capture, utilization and storage based on the ERNIE-BiLSTM-FNNAttention-CRF model of the present invention;

[0023] Figure 3 It is a schematic diagram of the FNNAttention structure in a patent named entity recognition method in the field of carbon capture, utilization and storage based on the ERNIE-BiLSTM-FNNAttention-CRF model of the present invention;

[0024] Figure 4 This is an example diagram of the edge feature solution process in a patent named entity recognition method in the field of carbon capture, utilization and storage based on the ERNIE-BiLSTM-FNNAttention-CRF model of the present invention;

[0025] Figure 5 It is a classification scoring framework diagram in a patent named entity recognition method in the field of carbon capture, utilization and storage based on the ERNIE-BiLSTM-FNNAttention-CRF model of the present invention. DETAILED DESCRIPTION

[0026] The specific implementation steps of the present invention are as follows:

[0027] The present invention is described below in conjunction with the accompanying drawings and specific embodiments. Figure 1 The process diagram of the patent named entity recognition method in the field of carbon capture, utilization and storage based on the ERNIE-BiLSTM-FNNAttention-CRF model is described; Figure 2 The model diagram of the patent named entity recognition method in the field of carbon capture, utilization and storage based on the ERNIE-BiLSTM-FNNAttention-CRF model is described; Figure 3Describes the FNNAttention structure diagram of the patent named entity recognition method in the field of carbon capture, utilization and storage based on the ERNIE-BiLSTM-FNNAttention-CRF model; Figure 4 Describes an example diagram of the edge feature solution process of the patent named entity recognition method in the field of carbon capture, utilization and storage based on the ERNIE-BiLSTM-FNNAttention-CRF model; Figure 5 The classification and scoring framework diagram of the patent named entity recognition method in the field of carbon capture, utilization and storage based on the ERNIE-BiLSTM-FNNAttention-CRF model is described.

[0028] like Figure 1 As shown, the present invention proposes a patent named entity recognition method in the field of carbon capture, utilization and storage based on the ERNIE-BiLSTM-FNNAttention-CRF model:

[0029] Step S1: In view of the lack of patent corpus in the CCUS field, patents in the patent database are used as the data source, and "CCUS+carbon capture+carbon utilization+carbon storage" is used as the search keyword to retrieve patent texts related to the CCUS field, including patent title, abstract, applicant, application date, legal status and other information. The abstract in the patent text is used as the main corpus. After data cleaning, corpus classification, boundary clarification, and BIO annotation, the patent corpus CCUS_dataset is constructed;

[0030] Step S2, using ERNIE as a pre-training model, outputting word vectors, ERNIE itself maintains modeling based on word feature input, so that the model does not need to rely on other information when applied, and has stronger versatility and scalability. The input samples of the ERNIE model include the representation of the input sentence pair, the sentence category of the sentence pair, the position code, and the representation of the word segmentation boundary; the sentence "The present invention discloses a carbon reduction system for LNG-powered ships and an EEDI calculation method" is input to the model, and each word of the sentence is vectorized respectively to obtain the corresponding word vector, text vector and position vector, and then input into the ERNIE model. After pre-training, the text vector is obtained and used as the input of the BiLSTM layer;

[0031] Since the composition of patent texts in the field of carbon capture, utilization and storage is quite different, there are often many professional words and entities with multiple meanings. Therefore, we use the ERNIE model to determine whether the context has semantic cohesion, or to predict the masked words after using masking strategies at different granularities of words, characters or noun entities, and then perform correct entity recognition based on the predicted results;

[0032] Specifically, the mask strategy is: randomly select 15% of the words in a sentence for prediction. For the words that are erased in the original sentence, a special symbol MASK is used to replace them in 80% of the cases, an arbitrary word is used to replace them in 10% of the cases, and the original words are kept unchanged in the remaining 10% of the cases. The main reason for this is that the [MASK] mark will not appear in the sentence in the subsequent fine-tuning task. Another benefit of this is that when predicting a word, the model does not know whether the word at the corresponding position of the input is the correct word (10% probability), which forces the model to rely more on contextual information to predict words and gives the model a certain error correction ability;

[0033] Step S3: For the word vector input from step S2, the BiLSTM layer performs T ), the forward LSTM layer generates a forward hidden state sequence (h1→,h2→,...,h T →), the reverse LSTM layer generates a reverse hidden state sequence (h1←,h2←,...,h T ←), the final output of BiLSTM is the concatenation of these two hidden state sequences (h t →,h t ←), learning semantic information from front to back and back to front.

[0034] BiLSTM can learn contextual features more deeply, thereby obtaining more comprehensive semantic information. LSTM is an improved solution to traditional RNN. By introducing memory units and threshold mechanisms to control the forgetting, input and output of information, it effectively solves the gradient vanishing and gradient explosion phenomena that are prone to occur in RNN networks while learning long-distance dependencies. The LSTM unit structure is optimized on the basis of traditional RNN and consists of three gate unit modules: input gate, forget gate and output gate;

[0035] When the LSTM structure processes data, it uses the input of the previous moment as the output of the current moment, and then uses the Sigmoid activation function to activate the input data. The formulas of the three parts are as follows:

[0036] i t =σ(W i ·[x t ,h t-1 ,C t-1 ]+b i )

[0037] f t =σ(W f ·[x t ,h t-1 ,C t-1 ]+b f)

[0038] o t =σ(W o ·[x t ,h t-1 ,C t-1 ]+b o )

[0039] Among them, i t 、f t , o t represents the three gate units of the LSTM unit structure, namely the input gate, forget gate and output gate, σ represents the Sigmoid activation function, and x t is the character vector representation of the input, h t-1 Indicates the hidden layer state of the LSTM structure at the last moment, C t-1 represents the cell state of the LSTM structure at the last moment, and b represents the bias vector. The formula for updating the cell state is as follows:

[0040] C t =f t ·C t-1 +i t tanh(W c [x i ,h i-1 ]+b c )

[0041] Where · is element-by-element multiplication, and the tanh function is the hyperbolic tangent activation function. The update process formula of the hidden layer state of the LSTM unit is as follows:

[0042] h t =o t tanh(C t )

[0043] The structure of LSTM enables it to obtain information from the previous context, but not from the subsequent context. However, it is particularly important to obtain bidirectional information from both the previous and the next context. The forward and backward LSTMs are connected in series to obtain a BiLSTM model to obtain deep information from both the previous and the next context.

[0044] Step S4, FNNAttention is essentially the weight distribution of word vectors. The semantic correlation between long-distance word vectors is calculated through a forward neural network, and the semantic weights are automatically learned and adjusted to strengthen the word-level relationship. FNNAttention can effectively utilize the semantic information between the contexts of long text paragraphs, obtain the global information vector, integrate it into the current feature vector for calculation, and mainly allocate attention to keywords. It can solve the problem of inconsistent annotations of the same patent entity in different sentences in a long text and strengthen the word-level relationship of the text;

[0045] The vector with global feature information is obtained by calculating the adaptive weighted average of the state sequence. The calculation formula is as follows:

[0046]

[0047] where μ(·) is the self-learning function of the feedforward neural network, which is obtained by only passing the state sequence h t Come and learn, h t It is the state sequence representation with bidirectional semantic information output by the BiLSTM layer, and k is the value obtained through self-learning of the neural network;

[0048] Pair vector c and state sequence h t The concatenation operation performed can obtain the joint p of the state and vector through the nonlinear activation function calculation. t The calculation formula is as follows:

[0049] g t =tanh[W p p t +b p ]

[0050] z t =tanh[W z p t +b z ]

[0051] Step S5, the conditional random field (CRF) layer is a popular method for calculating the named entity category with the highest probability score. CRF replaces the traditional softmax as the output layer of the neural network. Its powerful sequence modeling ability enables it to learn the constraints in the encoding process, thereby ensuring that the final prediction result has a high accuracy. The specific method is: X = {x1, x2, ..., x n}, Y = {y1, y2, ..., y n} are the sequences of BiLSTM layer input and corresponding output respectively. The formula for calculating the label sequence score is as follows:

[0052]

[0053] Where W is the transfer matrix, y i is x i The corresponding output, y i+1 is x i+1 The corresponding output is, It means the label is from y i Transfer to y i+1 The score, Indicates that the i+1th character in the input sequence corresponds to label y i+1Then calculate the probability of the label sequence y, and the calculation formula is as follows:

[0054]

[0055] where Y X is a set of all possible labels. Calculate the label set with the highest probability and use it as the final output sequence label;

[0056] In order to prove that the proposed ERNIE-BiLSTM-FNNAttentio-CRF model is more effective than other existing models, this section uses the control model variable method to compare the ERNIE and BERT experiments, as well as the ERNIE and FNNAttention layers. The comparison results are shown in the table below.

[0057] Table 1 Comparison of accuracy performance of different models on CCUS_Dataset

[0058]

[0059] The comparison of experiments 1 and 4, 2 and 5 shows that under the same network structure, with different Embbeding pre-training layers, the ERNIE model is proven to be significantly better than the commonly used BERT model in this experiment. Experiments 4 and 5, 1 and 2 can prove that adding FNNAttention to the existing pre-trained word segmentation model and BiLSTM can also significantly help improve the experimental results. In summary, the ERNIE-BiLSTM-FNNAttention-CRF model used in this paper has higher accuracy, recall and F1 value than other models in the patent corpus CCUS_Database.

[0060] The method of the present invention uses the ERNIE model for pre-training on the CCUS_dataset corpus. The ERNIE model learns the semantic relationship of the real world by modeling prior semantic knowledge such as entity concepts in massive data, and can simultaneously aggregate the information of context and knowledge entities to build a knowledge-based language representation model. At the same time, the present invention uses BiLSTM and FNNAttention networks to capture contextual relationships more comprehensively, and extracts semantic information with greater attention weight in patent texts through the FNNAttention layer.

[0061] The above description is only by way of illustration of certain exemplary embodiments of the present invention. It is undoubted that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. The present invention proposes a patent named entity recognition method in the field of carbon capture, utilization and storage based on the ERNIE-BiLSTM-FNNAttention-CRF model, which is characterized by: The following steps are involved: S1. In view of the lack of patent corpus in the field of carbon capture, utilization, and storage (CCUS), the patents in the patent database were used as the data source, and the keyword "CCUS+carbon capture+carbon utilization+carbon storage" was used to retrieve the relevant patent texts. The abstracts in the patent texts were used as the main corpus. After data cleaning, corpus classification, boundary clarification, and BIO annotation, the patent corpus CCUS_dataset was constructed. S2. The ERNIE model is used as a preprocessing model to convert patent text information into corresponding word vectors. The input samples of the ERNIE model include the representation of the input sentence pair, the category of the sentence pair, the position encoding, and the representation of the word segmentation boundary. The ERNIE model determines whether the patent context has semantic cohesion, or predicts the masked words after performing masking strategies on different granularities of words, characters or noun entities. Each word vector contains semantic information and contextual relationships, which can capture the actual meaning of key terms in patent documents and serve as the input features of the subsequent BiLSTM network. S3, for the word vector input from step S2, the BiLSTM layer is used for the input sequence X = (x1, x2, ..., x T ), the forward LSTM layer generates a forward hidden state sequence (h1→,h2→,...,h T →), the reverse LSTM layer generates a reverse hidden state sequence (h1←,h2←,...,h T ←), the final output of BiLSTM is the concatenation of these two hidden state sequences (h t →,h t ←), learn semantic information from front to back and from back to front; S4. Use the feedforward neural network attention mechanism FNNAttention to calculate the semantic correlation between long-distance word vectors in step S3, automatically learn to adjust the semantic weight, and strengthen the word-level relationship. FNNAttention can effectively utilize the semantic information between the contexts of long text paragraphs, obtain the global information vector, integrate it into the current feature vector for calculation, and mainly allocate attention to keywords. It can solve the problem of inconsistent annotations of the same patent entity in different sentences in the long text and strengthen the word-level relationship of the text. S5. Use CRF instead of the traditional softmax as the output layer of the neural network. CRF models the dependency between labels through the transfer probability matrix, and conditionally constrains the output of step S4. All possible label sequences are considered during decoding, and the sequence with the largest joint probability is selected as the output. CRF uses the negative log-likelihood loss function to measure the degree of matching. The larger the loss value, the better the sequence, thereby improving the recognition accuracy.

2. According to claim 1, a patent named entity recognition method in the field of carbon capture, utilization and storage based on the ERNIE-BiLSTM-FNNAttention-CRF model is characterized by: In step S1, the patent corpus CCUS_dataset is constructed, and the specific construction method is as follows: Step S11 uses patents in the patent database as data source, selects the advanced search module, searches for the keyword "CCUS+carbon capture+carbon utilization+carbon storage", and obtains relevant patent texts; Step S12: After screening and processing the patent text obtained in S11, data cleaning, corpus classification, and boundary clarification are performed, and the abstract in the patent text is used as the main corpus for conducting relevant experiments; After the entity type is defined in step S13, the BIO annotation method is adopted, where B represents the beginning part of the entity, I represents the middle and end parts of the entity, and O represents the non-entity part.

3. According to claim 1, a patent named entity recognition method in the field of carbon capture, utilization and storage based on the ERNIE-BiLSTM-FNNAttention-CRF model is characterized by: In step S2, the ERNIE model is constructed, and the specific construction method is as follows: Step S21: The input sample of the ERNIE model includes the following parts: the representation of the input sentence pair, the sentence category of the sentence pair, the position encoding, and the representation of the word boundary; Step S22: the ERNIE model predicts the masked words by judging whether there is semantic cohesion in the patent context or by applying masking strategies to different granularities of words, characters or noun entities; The mask strategy in step S23 is: randomly select 15% of the words in a sentence for prediction. For the words erased in the original sentence, replace them with a special symbol mask in 80% of the cases, replace them with an arbitrary word in 10% of the cases, and keep the original words unchanged in the remaining 10% of the cases.

4. According to claim 1, a patent named entity recognition method in the field of carbon capture, utilization and storage based on the ERNIE-BiLSTM-FNNAttention-CRF model is characterized by: In step S3, a BiLSTM-FNNAttention layer is constructed, and the specific construction method is as follows: Step S31 BiLSTM layer for the input sequence X = (x1, x2, ..., x T ), the forward LSTM layer generates a forward hidden state sequence (h1→,h2→,...,h T →); Step S32: The reverse LSTM layer generates a reverse hidden state sequence (h1←,h2←,...,h T ←), the final output of BiLSTM is the concatenation of these two hidden state sequences (h t →,h t ←).

5. According to claim 1, a patent named entity recognition method in the field of carbon capture, utilization and storage based on the ERNIE-BiLSTM-FNNAttention-CRF model is characterized by: In step S4, the FNNAttention layer is constructed, and the specific construction method is as follows: Step S41 uses the feedforward neural network attention mechanism FNNAttention to calculate the semantic association degree between the long-distance word vectors of the BiLSTM layer, automatically learns to adjust the semantic weight, and strengthens the word-level relationship; Step S42 effectively utilizes the semantic information between the contexts of long text paragraphs through FNNAttention, obtains the global information vector, integrates it into the current feature vector for calculation, and mainly allocates attention to keywords.

6. According to claim 1, a patent named entity recognition method in the field of carbon capture, utilization and storage based on the ERNIE-BiLSTM-FNNAttention-CRF model is characterized by: In step S5, a CRF output layer is constructed, and the specific construction method is as follows: Step S51 CRF models the dependencies between labels through the transition probability matrix, considers all possible label sequences during decoding, conditionally constrains all outputs of step S32, and selects the sequence with the largest joint probability as the output; Step S52 CRF loss function aims to minimize the difference between the model output and the true sequence, which is achieved by comparing the total probability of all paths PToual with the true path probability PRealPath. The larger the loss value, the better the sequence, thereby improving the recognition accuracy.

Images