Large-scale rare earth data information extraction method

Abstract

The invention discloses a large-scale rare earth data information extraction method, and relates to the technical field of rare earth research, the large-scale rare earth data information extraction method comprises the construction of a rare earth product production method patent named entity identification data set, and aims at the current situation of lack of a data set in the rare earth product production method patent field. The method comprises the following steps of: analyzing the content of the patent text of the production method of the rare earth product according to the related summary of the data set in the patent field of the production method of the rare earth product, dividing the patent data set of the production method of the rare earth product into entity categories, and introducing the condition of each category in detail; the introduction of labeling work is carried out on entities of a rare earth product production method patent data set, and comprises the steps of defining entity category labels, labeling entity categories by using a tool, exporting a json data file, and carrying out BIOES conversion on the json data file by using a script program.

Description

Technical Field

[0001] The invention relates to the technical field of rare earth research, and in particular to a large-scale rare earth data information extraction method. Background Art

[0002] Rare earth is a very important strategic resource. Because of its superior magnetic and photoelectric properties, it can greatly improve the performance of materials and significantly improve product quality and performance. Rare earth is not only widely used in traditional industries such as metallurgy, glass, ceramics, petrochemicals, and agriculture[], but also in high-end equipment, new materials, high-end electronic information products, energy conservation and environmental protection, and new energy vehicles. Strategic emerging industries are also widely used. Therefore, rare earth is called the "mother of new materials."

[0003] At present, there are relatively few studies on the rare earth field in my country, which leads to the inability to extract valuable information from a large number of rare earth patent texts, to innovate ideas for subsequent rare earth product production methods and to accelerate the digitalization and intelligent development of the rare earth field. It takes a lot of manpower and financial resources to extract valuable information from a large number of rare earth patent fields manually. At present, how to extract valuable information from large-scale rare earth field text data to better innovate production methods and promote the digitalization and intelligent development of the rare earth field is a problem that many researchers are thinking about. To this end, we provide a large-scale rare earth data information extraction method. Summary of the invention

[0004] The purpose of the present invention is to provide a large-scale rare earth data information extraction method to solve the problem raised in the above background technology that how to extract valuable information from large-scale rare earth field text data to better innovate production methods and promote the digitalization and intelligent development of the rare earth field is a problem that many researchers have been thinking about.

[0005] To achieve the above object, the present invention provides the following technical solution: a large-scale rare earth data information extraction method, comprising the following steps; S1. Construction of named entity recognition dataset for rare earth product production method patents, targeting the current lack of datasets in the field of rare earth product production method patents; (1) An overview of the dataset on rare earth product production method patents; (2) Analyze the content of the patent text of the rare earth product production method; (3) The rare earth product production method patent dataset was divided into entity categories, and the situation of each category was introduced in detail; (4) Introduction to the entity annotation work of the rare earth product production method patent dataset, including the definition of entity category labels, the use of tools to annotate entity categories, the export of json data files, and the use of scripts to convert json data files into BIOES; (5) Divide the data set and conduct quantitative analysis on each entity category. In addition, this chapter also introduces the evaluation indicators and experimental environment of this research experiment; S2. Patent entity recognition of rare earth product production method based on attention mechanism feature fusion. The single-channel recurrent neural network has limited ability to obtain text features and cannot fully obtain multi-dimensional encoding feature information. Therefore, an expanded convolutional network is introduced to obtain spatial sequence features. The dataset constructed in this paper has the problem of imbalanced entity categories. Therefore, the FlocalLoss loss function is introduced. For this reason, this chapter proposes a dual-channel RBIAC model for obtaining feature information. (1) The overall network structure of the model is introduced, and the various components of the model and the introduced FocalLoss loss function are introduced in detail; (2) The main parameter settings of the experiments in this chapter are explained; (3) A comparative experiment was conducted on three mainstream pre-training models and various algorithm models, verifying that the pre-training models and various algorithm models have different effects on the construction of the model; (4) Through ablation experiments, it is found that each part of the model affects the overall effect of the model, proving that the construction of each part of the model in this chapter is reasonable; S3. Patent entity recognition of rare earth product production method based on vocabulary enhancement and adversarial training. In view of the problem that the model in the previous chapter fails to make full use of the information of vocabulary granularity and the model is easily unstable due to noise, a vocabulary enhancement module is introduced to obtain more useful vocabulary information, and adversarial training is used to improve the stability of the model. A CWAFC model is proposed. (1) The overall structure of the CWAFC model is introduced, and the vocabulary enhancement module and three mainstream adversarial training methods in the NLP field are introduced in detail; (2) This chapter describes the data sets and main parameters of the experiments, and conducts four experiments and results analysis; (3) The first experiment is to compare the model with the mainstream important models in the Chinese classic dataset to prove the universality of the model in this chapter. The second experiment is to compare the impact of incorporating different numbers of words on the model in the RPNER dataset proposed in this paper. The third experiment is to compare the RPNER dataset proposed in this paper based on different adversarial training methods. The fourth experiment is to prove the effectiveness of the two parts of this chapter, namely the vocabulary enhancement module introduced in this chapter and the application of adversarial training, through ablation experiments in the RPNER dataset proposed in this paper.

[0006] Preferably, the construction of a named entity recognition dataset for rare earth product production method patents includes the following steps: S1-1. Content analysis: In general, the first part of the abstract text introduces the product or production method of the patent. The detailed patent abstract will also explain the field to which it belongs. The second part introduces the chemical composition and content of the product production method, or the quality or quantity of the material raw materials required to produce the product. The third part introduces the required process flow or the technology used, and the process flow is introduced in chronological order. Some process flows will also explain the specific temperature or temperature range required. The fourth part will introduce the characteristics of the product or production method, that is, its advantages in which aspects. S1-2. Classification of text entity types: For processing common named entity tasks, there are mainly three major types of entities (number type, time type, entity type) and seven minor types of entities (name type, time type, organization name type, currency type, percentage type, date type). For the entity types in the patent field, they are generally divided into two categories: general entities and professional entities. The main content of this article is the entity recognition of the rare earth product production method, so the entity type of this article will define professional entities. Through the content analysis of a large number of rare earth product production method patent abstracts, the entity types of rare earth product production method patent abstracts mainly include: field type entities, product type entities, applied process or technology type entities, raw materials used or product chemical composition type entities, product efficacy type entities, and temperature type entities required to produce the product; S1-3. Entity annotation of text: Currently, named entity recognition tasks have the same problem as many other tasks in the field of artificial intelligence: data-driven tasks. The quality of the data set will affect the effect of named entity recognition. Therefore, entity annotation of the abstract text of rare earth product production technology patents is a crucial task. At present, there are three main types of data annotation methods in the field of artificial intelligence: manual annotation, semi-automatic annotation, and automatic annotation. These three methods are suitable for different scenario requirements. Due to the current lack of rare earth product production technology patent abstract text data corpus and dictionary, this study uses manual annotation methods to annotate data; S1-4. Evaluation indicators and experimental environment: Software environment: Pycharm is used locally for coding and debugging, and the program is uploaded to the server for experimental training and testing. The image in the environment is PyTorch version 1.6.0, Python version 3.8 (ubuntu18.04), and Cuda version 10.1; Hardware environment: The server GPU parameters are RTX2080Ti (11GB)*1, the CPU parameters are 12vCPUIntel(R)Xeon(R)Platinum8255CCPU@2.50GHz, and the memory is 40G.

[0007] Preferably, the patent entity recognition of the rare earth product production method based on attention mechanism feature fusion includes the following steps: S2-1. Model framework: Overall model processing: First, the text is input into the embedding layer to obtain a vector representation at the word granularity level; second, it is simultaneously input into the BiGRU model and IDCNN model of the context encoding layer to obtain time series features and spatial sequence features respectively; then, the time series features and spatial sequence features are dynamically fused using the attention mechanism in the feature fusion layer; finally, the result is input into the label decoding layer. The detailed training process of the model from the input training set, validation set to the output loss function and evaluation index is as follows: (1) Process the input data set: for example, truncate the maximum text length and preprocess special characters; (2) Input the preprocessed text into the RoBRTa-wwm-ext pre-trained model to obtain a tensor whose shape contains three dimensions: batch size, text length, and feature dimension; (3) Input the tensor obtained in step (2) into the BiGRU network and the IDCNN network at the same time, and obtain tensors with the same shape and after conversion by the fully connected layer; (4) Input the tensor and input into the self-attention mechanism network to obtain dynamic weights and fuse them to obtain a vector; (5) Input the vector into the decoding layer CRF to obtain its corresponding predicted label result; (6) Input the true label and predicted label of the data set into the loss function to obtain the loss result; (7) Based on the loss results, use the gradient descent method to modify the weights and biases; Through iterative training, evaluation indicators are obtained to evaluate the model training status; S2-2, Embedding layer: Compared with BERT, RoBERTa-wwm-ext has three main optimizations: the first is to use a larger and richer training corpus to train the model, which effectively improves the effect on downstream tasks; the second is to abandon the NSP task, that is, to abandon the prediction of whether two sentences are consecutive sentences; the third is to use the dynamic mask method to replace the static mask method in BERT, where the static mask refers to completing the MASK operation when processing the data first, and then inputting it into the model for training, and the dynamic mask refers to directly performing the dynamic MASK operation during model training; S2-3. In order to better obtain the contextual time series feature information of sentences in rare earth patents, the gated recurrent unit (GRU) is introduced into the model structure of this chapter. GRU is improved to deal with the long-term memory of traditional RNN and the gradient disappearance problem in the back propagation process. Its effect of extracting feature information is similar to LSTM, but it has the characteristics of simpler structure composition and calculation process and fewer parameters. Therefore, its convergence speed has advantages, the required hardware and time overhead are relatively lower, and the generalization ability is stronger when the data set sample is small. Therefore, the GRU network is more suitable for the entity recognition task of the patent text in this article to extract the time series features of the sentence context; S2-4. Dilated convolution: Since the text of the patent for the production method of rare earth products has the characteristic of high entity density, and the size of the convolution kernel of the traditional CNN convolutional network is fixed, it can only obtain local features within the convolution window, and cannot capture multiple local features of texts of different lengths. In addition, the weights between its convolutional layers are shared, and there may be confusion when the network processes different position information. The CNN network may also have the problem of overfitting. The dilated convolutional network is also a convolutional network. It is different from the traditional CNN network in the following aspects in natural language processing: (1) In terms of parameter efficiency. Compared with the traditional CNN network structure, the dilated convolutional network introduces depth-separable convolutional layers and depth-convolutional layers to replace the traditional convolutional layers, which can reduce the number of parameters and the amount of calculation to a certain extent; (2) In terms of multi-scale local features. The dilated convolutional network can set the dilation parameter to capture spatial local features of different scales, while the convolution kernel of the traditional CNN network is fixed; (3) In terms of iteration. The dilated convolution adds an iterative dilated convolution network operation, which can expand the network's perception field of view, thereby improving the network's perception ability and local feature acquisition ability; (4) In terms of long-distance dependencies, the traditional CNN network cannot obtain long-distance text dependencies well, while the dilated convolutional network can obtain long-distance text dependencies to improve the performance of the network; S2-5. Experimental parameter settings: Pytorch framework is used in the encoding process, and RoBERTa-wwm-ext model is used in the embedding layer. This pre-trained model is trained by the Joint Laboratory of Harbin Institute of Technology. The transformer in this pre-trained model has 12 layers and the dimension of the hidden layer is 768. In the model training process of the dataset in this chapter, the training cycle parameter epoch is set to 50. Because the text entity density of the dataset in this article is relatively large, the maximum text input length parameter max_length is set to 64, and the batch size parameter batch_size is set to 8; in the feature acquisition layer, the GRU network is set to bidirectional, the number of network layers is 2, and the number of hidden layers is 128. In order to prevent the model from overfitting, the dropout value is set to 0.1, and the convolution step parameter stride in the 3-layer iterative convolution layer of the IDCNN network is 1. In order to keep the shape of the feature unchanged during the convolution process, the padding parameter padding is set to dilation*(kernel_size-1) / / 2, where dilation is the expansion coefficient and kernel_size is the size of the convolution kernel; S2-6. Experimental results and analysis: The time series features and spatial sequence features of the text are effective in improving the effect of the named entity recognition task; the F1 value of the BiGRU+CRF model is 0.14% higher than that of the BiLSTM+CRF model, indicating that the bidirectional gated convolutional recurrent unit network is more suitable for extracting time series features from the patent text of the rare earth product production method in this chapter than the bidirectional long short-term memory network; the F1 value of the IDCNN+CRF model is 0.24% higher than that of the CNN+CRF model, indicating that the dilated convolutional network is more suitable for obtaining spatial sequence feature information in this chapter than the traditional CNN; the F1 value of the BiGRU+IDCNN+ATT+CRF model is 0.53% and 0.93% higher than that of the BiGRU+CRF model and the IDCNN+CRF model, respectively, indicating that the effect of fusing the time series features and spatial sequence features of the text is better than that of the single time series features or the single spatial sequence features, which is conducive to the complementary advantages of the two features, thereby obtaining more comprehensive text feature information; S2-7. In order to verify the effectiveness and rationality of each component of the RBIAC model proposed in this chapter, an ablation experiment was carried out, in which -ATT means removing the multi-head attention mechanism and using direct vector concatenation instead of dynamic fusion of the multi-head attention mechanism; -BiGRU means removing the bidirectional gated convolutional recurrent unit network part and only using the dilated convolutional network for feature acquisition; -IDCNN means removing the dilated convolutional network part and only using the recurrent gated unit network for feature acquisition; -FocalLoss means removing the FocalLoss loss function and using only the cross entropy loss function when calculating the model loss.

[0008] Preferably, the rare earth product production method patent entity recognition based on vocabulary enhancement and adversarial training includes the following steps: S3-1. Model framework: The model processes the input text as follows: First, the input text is passed through the RoBRTa-wwm-ext embedding layer to obtain the character-level vector, and the input text is passed through the vocabulary enhancement module to obtain the vocabulary-level vector, and the character-level and vocabulary-level information are dynamically fused to obtain the embedding vector; secondly, the embedding vector is trained to generate the corresponding adversarial sample; then, the BiGRU model and IDCNN model are used to obtain the time series features and spatial series features in the two channels respectively, and the multi-head attention mechanism is used to dynamically fuse the feature encoding of the two channels; finally, it is input into the CRF model to obtain the predicted label corresponding to the input text; S3-2. Experimental data set and parameters: In the experiment of the MSRA data set model, the number of transformer layers in the pre-trained model is 12, and the hidden layer dimension is 768; in order to fully train the model, the epoch parameter representing the training cycle is set to 50; since the text sentences in the MSRA data set come from the text in the news field, it is somewhat challenging for the model to understand. In order to allow the model to fully utilize the feature encoding information of the sentence to identify entities, the maximum text input length parameter max_length is set to 128; the batch size parameter batch_size is set to 8; in the feature acquisition layer, the GRU network is set to bidirectional, the number of network layers is 1, and the number of hidden layers is 128; in order to prevent the model from overfitting during training, the dropout parameter is set The value is set to 0.5, and the convolution step parameter stride in the three-layer iterative convolution layer of the IDCNN network is 1. In order to keep the shape of the convolution feature unchanged, the padding parameter padding is set to dilation*(kernel_size-1) / / 2, where dilation is the expansion coefficient of 1, 1, and 2 respectively, and kernel_size is the size of the convolution kernel; In addition, the optimizer used in this model is Adam, which can dynamically adjust the learning rate of the model parameters to minimize the loss of the model. The MSRA dataset is one of the four classic datasets in the current Chinese NER task. It comes from the text in the news field and includes three types of entities: address entities (LOC), person entities (PER), and organization name entities (ORG); S3-3. Experimental results and analysis: Based on the classic dataset MSRA in the current Chinese NER task and the RPNER dataset constructed in this paper, several comparative experiments and ablation experiments were conducted. The comparative experiments include: comparing with the current mainstream Chinese NRE task model, exploring the integration of different k words for comparison, and comparing different adversarial training methods in the NLP field; and the ablation experiment is to verify the impact on the overall effect of the model by removing the newly introduced modules in this chapter. The specific experimental design is as follows: (1) Comparative experiment with current mainstream models: This chapter selects the mainstream models in Chinese NER tasks in recent years and compares them with the CWAFC model proposed in this chapter on the Chinese classic MSRA dataset to verify the universality and effectiveness of the model in this chapter; (2) Comparative experiment to explore the effect of different words on the model: This experiment explores the effect of different words on the model by incorporating them into the embedding layer; (3) Comparative experiments based on different adversarial training methods: This experiment mainly uses the three mainstream adversarial training methods in the field of natural language processing to test the effects on the model in this chapter; (4) Ablation experiment: This experiment removes the modules introduced in this paper to solve the problems existing in this model, so as to verify the impact of the introduced modules on the model in this chapter.

[0009] Compared with the prior art, the present invention has the following beneficial effects: 1. Construction of a rare earth product production method patent dataset. At present, there is a lack of research on named entity recognition in rare earth product production method patents, so there is a lack of datasets in this field. This paper analyzes and studies the specific content of rare earth product production method patent texts and the corresponding characteristics of the content. In the process, 820 rare earth product production method patent texts were obtained from relevant websites, 6 types of entities were summarized, and the sentences were annotated using annotation tools to construct a rare earth product production method patent dataset.

[0010] 2. Research on entity recognition of rare earth product production method patents based on attention mechanism feature fusion. For recurrent gated unit neural networks that only use a single channel, the feature encoding acquisition method is relatively simple and cannot obtain multi-dimensional feature encoding information, which makes it difficult to fully capture the features of sentences with high entity density in the text of rare earth product production method patents. Therefore, a rare earth product production method patent entity recognition model with attention mechanism feature fusion is proposed. The model structure uses the attention mechanism to fuse dual channels to obtain multi-dimensional feature encoding information of the time series and spatial sequence of sentences in the encoding layer to fully obtain the feature encoding in the sentence, and the model also introduces the FocalLoss loss function to solve the problem of unbalanced entity type distribution in the rare earth product production method patent data set.

[0011] 3. Research on entity recognition of rare earth product production method patents based on vocabulary enhancement and adversarial training. Compared with the traditional model, the effect of the rare earth product production method patent entity recognition model based on attention mechanism feature fusion has been improved to a certain extent, but it still does not make full use of vocabulary information, which is particularly important for identifying highly professional and complex entities in the rare earth product production method patent text, and the model is susceptible to interference and instability. In response to these two problems, a rare earth product production method patent entity recognition model based on vocabulary enhancement and adversarial training is proposed. A vocabulary enhancement module is designed in the model structure to fully obtain useful vocabulary information. The adversarial training method is introduced in the model structure to improve the model's anti-interference ability and generalization. The universality of the model is verified by experiments in the Chinese classic dataset, and the effectiveness of the model is verified by experiments in the rare earth product production method patent dataset. DETAILED DESCRIPTION

[0012] The technical solutions in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0013] An embodiment provided by the present invention is a method for extracting large-scale rare earth data information, the method comprising the following steps: S1. Construction of named entity recognition dataset for rare earth product production method patents, targeting the current lack of datasets in the field of rare earth product production method patents; (1) An overview of the dataset on rare earth product production method patents; (2) Analyze the content of the patent text of the rare earth product production method; (3) The rare earth product production method patent dataset was divided into entity categories, and the situation of each category was introduced in detail; (4) Introduction to the entity annotation work of the rare earth product production method patent dataset, including the definition of entity category labels, the use of tools to annotate entity categories, the export of json data files, and the use of scripts to convert json data files into BIOES; (5) Divide the data set and conduct quantitative analysis on each entity category. In addition, this chapter also introduces the evaluation indicators and experimental environment of this research experiment; S2. Patent entity recognition of rare earth product production method based on attention mechanism feature fusion. The single-channel recurrent neural network has limited ability to obtain text features and cannot fully obtain multi-dimensional encoding feature information. Therefore, an expanded convolutional network is introduced to obtain spatial sequence features. The dataset constructed in this paper has the problem of imbalanced entity categories. Therefore, the FlocalLoss loss function is introduced. For this reason, this chapter proposes a dual-channel RBIAC model for obtaining feature information. (1) The overall network structure of the model is introduced, and the various components of the model and the introduced FocalLoss loss function are introduced in detail; (2) The main parameter settings of the experiments in this chapter are explained; (3) A comparative experiment was conducted on three mainstream pre-training models and various algorithm models, verifying that the pre-training models and various algorithm models have different effects on the construction of the model; (4) Through ablation experiments, it is found that each part of the model affects the overall effect of the model, proving that the construction of each part of the model in this chapter is reasonable; S3. Patent entity recognition of rare earth product production method based on vocabulary enhancement and adversarial training. In view of the problem that the model in the previous chapter fails to make full use of the information of vocabulary granularity and the model is easily unstable due to noise, a vocabulary enhancement module is introduced to obtain more useful vocabulary information, and adversarial training is used to improve the stability of the model. A CWAFC model is proposed. (1) The overall structure of the CWAFC model is introduced, and the vocabulary enhancement module and three mainstream adversarial training methods in the NLP field are introduced in detail; (2) This chapter describes the data sets and main parameters of the experiments, and conducts four experiments and results analysis; (3) The first experiment is to compare the model with the mainstream important models in the Chinese classic dataset to prove the universality of the model in this chapter. The second experiment is to compare the impact of incorporating different numbers of words on the model in the RPNER dataset proposed in this paper. The third experiment is to compare the RPNER dataset proposed in this paper based on different adversarial training methods. The fourth experiment is to prove the effectiveness of the two parts of this chapter, namely the vocabulary enhancement module introduced in this chapter and the application of adversarial training, through ablation experiments in the RPNER dataset proposed in this paper.

[0014] The construction of the named entity recognition dataset of rare earth product production method patents includes the following steps: S1-1. Content analysis: In general, the first part of the abstract text introduces the product or production method of the patent. The detailed patent abstract will also explain the field to which it belongs. The second part introduces the chemical composition and content of the product production method, or the quality or quantity of the material raw materials required to produce the product. The third part introduces the required process flow or the technology used, and the process flow is introduced in chronological order. Some process flows will also explain the specific temperature or temperature range required. The fourth part will introduce the characteristics of the product or production method, that is, its advantages in which aspects. S1-2. Classification of text entity types: For processing common named entity tasks, there are mainly three major types of entities (number type, time type, entity type) and seven minor types of entities (name type, time type, organization name type, currency type, percentage type, date type). For the entity types in the patent field, they are generally divided into two categories: general entities and professional entities. The main content of this article is the entity recognition of the rare earth product production method, so the entity type of this article will define professional entities. Through the content analysis of a large number of rare earth product production method patent abstracts, the entity types of rare earth product production method patent abstracts mainly include: field type entities, product type entities, applied process or technology type entities, raw materials used or product chemical composition type entities, product efficacy type entities, and temperature type entities required to produce the product; S1-3. Entity annotation of text: Currently, named entity recognition tasks have the same problem as many other tasks in the field of artificial intelligence: data-driven tasks. The quality of the data set will affect the effect of named entity recognition. Therefore, entity annotation of the abstract text of rare earth product production technology patents is a crucial task. At present, there are three main types of data annotation methods in the field of artificial intelligence: manual annotation, semi-automatic annotation, and automatic annotation. These three methods are suitable for different scenario requirements. Due to the current lack of rare earth product production technology patent abstract text data corpus and dictionary, this study uses manual annotation methods to annotate data; S1-4. Evaluation indicators and experimental environment: Software environment: Pycharm is used locally for coding and debugging, and the program is uploaded to the server for experimental training and testing. The image in the environment is PyTorch version 1.6.0, Python version 3.8 (ubuntu18.04), and Cuda version 10.1; Hardware environment: The server GPU parameters are RTX2080Ti (11GB)*1, the CPU parameters are 12vCPUIntel(R)Xeon(R)Platinum8255CCPU@2.50GHz, and the memory is 40G.

[0015] Patent entity recognition of rare earth product production method based on attention mechanism feature fusion includes the following steps: S2-1. Model framework: Overall model processing: First, the text is input into the embedding layer to obtain a vector representation at the word granularity level; second, it is simultaneously input into the BiGRU model and IDCNN model of the context encoding layer to obtain time series features and spatial sequence features respectively; then, the time series features and spatial sequence features are dynamically fused using the attention mechanism in the feature fusion layer; finally, the result is input into the label decoding layer. The detailed training process of the model from the input training set, validation set to the output loss function and evaluation index is as follows: (1) Process the input data set: for example, truncate the maximum text length and preprocess special characters; (2) Input the preprocessed text into the RoBRTa-wwm-ext pre-trained model to obtain a tensor whose shape contains three dimensions: batch size, text length, and feature dimension; (3) Input the tensor obtained in step (2) into the BiGRU network and the IDCNN network at the same time, and obtain tensors with the same shape and after conversion by the fully connected layer; (4) Input the tensor and input into the self-attention mechanism network to obtain dynamic weights and fuse them to obtain a vector; (5) Input the vector into the decoding layer CRF to obtain its corresponding predicted label result; (6) Input the true label and predicted label of the data set into the loss function to obtain the loss result; (7) Based on the loss results, use the gradient descent method to modify the weights and biases; Through iterative training, evaluation indicators are obtained to evaluate the model training status; S2-2, Embedding layer: Compared with BERT, RoBERTa-wwm-ext has three main optimizations: the first is to use a larger and richer training corpus to train the model, which effectively improves the effect on downstream tasks; the second is to abandon the NSP task, that is, to abandon the prediction of whether two sentences are consecutive sentences; the third is to use the dynamic mask method to replace the static mask method in BERT, where the static mask refers to completing the MASK operation when processing the data first, and then inputting it into the model for training, and the dynamic mask refers to directly performing the dynamic MASK operation during model training; S2-3. In order to better obtain the contextual time series feature information of sentences in rare earth patents, the gated recurrent unit (GRU) is introduced into the model structure of this chapter. GRU is improved to deal with the long-term memory of traditional RNN and the gradient disappearance problem in the back propagation process. Its effect of extracting feature information is similar to LSTM, but it has the characteristics of simpler structure composition and calculation process and fewer parameters. Therefore, its convergence speed has advantages, the required hardware and time overhead are relatively lower, and the generalization ability is stronger when the data set sample is small. Therefore, the GRU network is more suitable for the entity recognition task of the patent text in this article to extract the time series features of the sentence context; S2-4. Dilated convolution: Since the text of the patent for the production method of rare earth products has the characteristic of high entity density, and the size of the convolution kernel of the traditional CNN convolutional network is fixed, it can only obtain local features within the convolution window, and cannot capture multiple local features of texts of different lengths. In addition, the weights between its convolutional layers are shared, and there may be confusion when the network processes different position information. The CNN network may also have overfitting problems. The dilated convolutional network is also a convolutional network. It is different from the traditional CNN network in the following aspects in natural language processing: (1) In terms of parameter efficiency. Compared with the traditional CNN network structure, the dilated convolutional network introduces depth-separable convolutional layers and depth-convolutional layers to replace the traditional convolutional layers, which can reduce the number of parameters and the amount of calculation to a certain extent; (2) In terms of multi-scale local features. The dilated convolutional network can set the dilation parameter to capture spatial local features of different scales, while the convolution kernel of the traditional CNN network is fixed; (3) In terms of iteration. The dilated convolution adds an iterative dilated convolution network operation, which can expand the network's perception field of view, thereby improving the network's perception ability and local feature acquisition ability; (4) In terms of long-distance dependencies, the traditional CNN network cannot obtain long-distance text dependencies well, while the dilated convolutional network can obtain long-distance text dependencies to improve the performance of the network; S2-5. Experimental parameter settings: Pytorch framework is used in the encoding process, and RoBERTa-wwm-ext model is used in the embedding layer. This pre-trained model is trained by the Joint Laboratory of Harbin Institute of Technology. The transformer in this pre-trained model has 12 layers and the dimension of the hidden layer is 768. In the model training process of the dataset in this chapter, the training cycle parameter epoch is set to 50. Because the text entity density of the dataset in this article is relatively large, the maximum text input length parameter max_length is set to 64, and the batch size parameter batch_size is set to 8; in the feature acquisition layer, the GRU network is set to bidirectional, the number of network layers is 2, and the number of hidden layers is 128. In order to prevent the model from overfitting, the dropout value is set to 0.1, and the convolution step parameter stride in the 3-layer iterative convolution layer of the IDCNN network is 1. In order to keep the shape of the feature unchanged during the convolution process, the padding parameter padding is set to dilation*(kernel_size-1) / / 2, where dilation is the expansion coefficient and kernel_size is the size of the convolution kernel; S2-6. Experimental results and analysis: The time series features and spatial sequence features of the text are effective in improving the effect of the named entity recognition task; the F1 value of the BiGRU+CRF model is 0.14% higher than that of the BiLSTM+CRF model, indicating that the bidirectional gated convolutional recurrent unit network is more suitable for extracting time series features from the patent text of the rare earth product production method in this chapter than the bidirectional long short-term memory network; the F1 value of the IDCNN+CRF model is 0.24% higher than that of the CNN+CRF model, indicating that the dilated convolutional network is more suitable for obtaining spatial sequence feature information in this chapter than the traditional CNN; the F1 value of the BiGRU+IDCNN+ATT+CRF model is 0.53% and 0.93% higher than that of the BiGRU+CRF model and the IDCNN+CRF model, respectively, indicating that the effect of fusing the time series features and spatial sequence features of the text is better than that of the single time series features or the single spatial sequence features, which is conducive to the complementary advantages of the two features, thereby obtaining more comprehensive text feature information; S2-7. In order to verify the effectiveness and rationality of each component of the RBIAC model proposed in this chapter, an ablation experiment was carried out, in which -ATT means removing the multi-head attention mechanism and using direct vector concatenation instead of dynamic fusion of the multi-head attention mechanism; -BiGRU means removing the bidirectional gated convolutional recurrent unit network part and only using the dilated convolutional network for feature acquisition; -IDCNN means removing the dilated convolutional network part and only using the recurrent gated unit network for feature acquisition; -FocalLoss means removing the FocalLoss loss function and using only the cross entropy loss function when calculating the model loss.

[0016] Patent entity recognition of rare earth product production method based on vocabulary enhancement and adversarial training includes the following steps: S3-1, model framework: the model processes the input text as follows: first, the input text is passed through the RoBRTa-wwm-ext embedding layer to obtain the character-level vector, and at the same time, the input text is passed through the vocabulary enhancement module to obtain the vocabulary-level vector, and the character-level and vocabulary-level information are dynamically fused to obtain the embedding vector; secondly, the embedding vector is used to generate the corresponding adversarial sample by adversarial training; then, the BiGRU model and IDCNN model are used to obtain the time series features and spatial series features in two channels respectively, and the multi-head attention mechanism is used to dynamically fuse the feature encoding of the two channels; finally, it is input into the CRF model to obtain the predicted label corresponding to the input text; S3-2. Experimental data set and parameters: In the experiment of the MSRA data set model, the number of transformer layers in the pre-trained model is 12, and the hidden layer dimension is 768; in order to fully train the model, the epoch parameter representing the training cycle is set to 50; since the text sentences in the MSRA data set come from the text in the news field, it is somewhat challenging for the model to understand. In order to allow the model to fully utilize the feature encoding information of the sentence to identify entities, the maximum text input length parameter max_length is set to 128; the batch size parameter batch_size is set to 8; in the feature acquisition layer, the GRU network is set to bidirectional, the number of network layers is 1, and the number of hidden layers is 128; in order to prevent the model from overfitting during training, the dropout parameter is set The value is set to 0.5, and the convolution step parameter stride in the three-layer iterative convolution layer of the IDCNN network is 1. In order to keep the shape of the convolution feature unchanged, the padding parameter padding is set to dilation*(kernel_size-1) / / 2, where dilation is the expansion coefficient of 1, 1, and 2 respectively, and kernel_size is the size of the convolution kernel; In addition, the optimizer used in this model is Adam, which can dynamically adjust the learning rate of the model parameters to minimize the loss of the model. The MSRA dataset is one of the four classic datasets in the current Chinese NER task. It comes from the text in the news field and includes three types of entities: address entities (LOC), person entities (PER), and organization name entities (ORG); S3-3. Experimental results and analysis: Based on the classic dataset MSRA in the current Chinese NER task and the RPNER dataset constructed in this paper, several comparative experiments and ablation experiments were conducted. The comparative experiments include: comparing with the current mainstream Chinese NRE task model, exploring the integration of different k words for comparison, and comparing different adversarial training methods in the NLP field; and the ablation experiment is to verify the impact on the overall effect of the model by removing the newly introduced modules in this chapter. The specific experimental design is as follows: (1) Comparative experiment with current mainstream models: This chapter selects the mainstream models in Chinese NER tasks in recent years and compares them with the CWAFC model proposed in this chapter on the Chinese classic MSRA dataset to verify the universality and effectiveness of the model in this chapter; (2) Comparative experiment to explore the effect of different words on the model: This experiment explores the effect of different words on the model by incorporating them into the embedding layer; (3) Comparative experiments based on different adversarial training methods: This experiment mainly uses the three mainstream adversarial training methods in the field of natural language processing to test the effects on the model in this chapter; (4) Ablation experiment: This experiment removes the modules introduced in this paper to solve the problems existing in this model, so as to verify the impact of the introduced modules on the model in this chapter.

Claims

1. A large-scale rare earth data information extraction method, the large-scale rare earth data information extraction method comprising the following steps; S1. Construction of named entity recognition dataset for rare earth product production method patents, targeting the current lack of datasets in the field of rare earth product production method patents; a. An overview of the rare earth product production method patent data set; b. Analyze the content of the patent text of the rare earth product production method; c. The rare earth product production method patent dataset was divided into entity categories, and the situation of each category was introduced in detail; d. Introduction to the entity annotation work of the rare earth product production method patent dataset, including the definition of entity category labels, the use of tools to annotate entity categories, the export of json data files, and the use of scripts to convert json data files into BIOES; e. Divide the data set and conduct quantitative analysis on each entity category. In addition, this chapter also introduces the evaluation indicators and experimental environment of this research experiment; S2. Patent entity recognition of rare earth product production method based on attention mechanism feature fusion. The single-channel recurrent neural network has limited ability to obtain text features and cannot fully obtain multi-dimensional encoding feature information. Therefore, an expanded convolutional network is introduced to obtain spatial sequence features. The dataset constructed in this paper has the problem of imbalanced entity categories. Therefore, the FlocalLoss loss function is introduced. For this reason, this chapter proposes a dual-channel RBIAC model for obtaining feature information. a. The overall network structure of the model is introduced, and the various components of the model and the introduced FocalLoss loss function are introduced in detail; b. Explain the main parameter settings of the experiments in this chapter; c. Comparative experiments were conducted on three mainstream pre-training models and various algorithm models, verifying that pre-training models and various algorithm models have different effects on model construction; d. Through ablation experiments, it is found that each part of the model affects the overall effect of the model, proving that the construction of each part of the model in this chapter is reasonable; S3. Patent entity recognition of rare earth product production method based on vocabulary enhancement and adversarial training. In view of the problem that the model in the previous chapter fails to make full use of the information of vocabulary granularity and the model is easily unstable due to noise, a vocabulary enhancement module is introduced to obtain more useful vocabulary information, and adversarial training is used to improve the stability of the model. A CWAFC model is proposed. a. The overall structure of the CWAFC model is introduced, and the vocabulary enhancement module and three mainstream adversarial training methods in the NLP field are introduced in detail; b. Explain the data sets and main parameters of the experiments in this chapter, and conduct four experiments and result analysis; c. The first experiment is to conduct a comparative experiment with the mainstream important models in the Chinese classic dataset to prove the universality of the model in this chapter. The second experiment is to conduct a comparative experiment on the RPNER dataset proposed in this article to explore the impact of incorporating different numbers of vocabulary on the model. The third experiment is to conduct a comparative experiment based on different adversarial training methods in the RPNER dataset proposed in this article. The fourth experiment uses the ablation experiment in the RPNER dataset proposed in this article to prove the effectiveness of the two parts of this chapter, which are the introduction of the vocabulary enhancement module and the application of adversarial training.

2. According to the large-scale rare earth data information extraction method described in claim 1, the construction of the rare earth product production method patent named entity recognition data set comprises the following steps: S1-1. Content analysis: In general, the first part of the abstract text introduces the product or production method of the patent. The detailed patent abstract will also explain the field to which it belongs. The second part introduces the chemical composition and content of the product production method, or the quality or quantity of the material raw materials required to produce the product. The third part introduces the required process flow or the technology used, and the process flow is introduced in chronological order. Some process flows will also explain the specific temperature or temperature range required. The fourth part will introduce the characteristics of the product or production method, that is, its advantages in which aspects. S1-2. Classification of text entity types: For processing common named entity tasks, there are mainly three major entity types and seven minor entity types. For the entity types in the patent field, they are generally divided into two categories: general entity types and professional entity types. The main content of this article is the entity recognition of the rare earth product production method. Therefore, the entity type of this article will define the professional entity type. Through the content analysis of a large number of rare earth product production method patent abstracts, the entity types of the rare earth product production method patent abstracts mainly include: the field entity type, product entity type, applied process or technology entity type, raw material or product chemical composition entity type, product efficacy entity type, and temperature entity type required to produce the product. S1-3. Entity annotation of text: Currently, named entity recognition tasks have the same problem as many other tasks in the field of artificial intelligence: data-driven tasks. The quality of the data set will affect the effect of named entity recognition. Therefore, entity annotation of the abstract text of rare earth product production technology patents is a crucial task. At present, there are three main types of data annotation methods in the field of artificial intelligence: manual annotation, semi-automatic annotation, and automatic annotation. These three methods are suitable for different scenario requirements. Due to the current lack of rare earth product production technology patent abstract text data corpus and dictionary, this study uses manual annotation methods to annotate data; S1-4. Evaluation indicators and experimental environment: Software environment: Pycharm is used locally for coding and debugging, and the program is uploaded to the server for experimental training and testing. The image in the environment is PyTorch version 1.6.0, Python version 3.8, and Cuda version 10.1; Hardware environment: The server GPU parameters are RTX2080Ti*1, the CPU parameters are 12vCPUIntelXeonPlatinum8255CCPU@2.50GHz, and the memory is 40G.

3. According to the large-scale rare earth data information extraction method described in claim 1, the rare earth product production method patent entity recognition based on attention mechanism feature fusion comprises the following steps: S2-1. Model framework: Overall model processing: First, the text is input into the embedding layer to obtain a vector representation at the word granularity level; second, it is simultaneously input into the BiGRU model and IDCNN model of the context encoding layer to obtain time series features and spatial series features respectively; Then, the attention mechanism is used in the feature fusion layer to dynamically fuse the time series features with the spatial series features; finally, the results are input to the label decoding layer. The detailed training process of the model from the input training set, validation set to the output loss function and evaluation index is as follows: a. Process the input data set: for example, truncate the maximum text length and pre-process special characters; b. Input the preprocessed text into the RoBRTa-wwm-ext pre-trained model to obtain the tensor , whose shape contains three dimensions: batch size, text length, and feature dimension; c. Input the tensor obtained in step b into the BiGRU network and IDCNN network at the same time, and obtain tensors with the same shape after conversion in the fully connected layer. and ; d. Tensor and Input into the self-attention mechanism network to obtain dynamic weights and fusion to obtain the vector ; e. The vector is input into the decoding layer CRF to obtain its corresponding predicted label result; f. Input the real label and predicted label of the data set into the loss function to obtain the loss result; g. Based on the loss results, use the gradient descent method to modify the weights and bias ; Through iterative training, evaluation indicators are obtained to evaluate the model training status; S2-2, Embedding layer: Compared with BERT, RoBERTa-wwm-ext has three main optimizations: the first is to use a larger and richer training corpus to train the model, which effectively improves the effect on downstream tasks; the second is to abandon the NSP task, that is, to abandon the prediction of whether two sentences are consecutive sentences; the third is to use the dynamic mask method to replace the static mask method in BERT, where the static mask refers to completing the MASK operation when processing the data first, and then inputting it into the model for training, and the dynamic mask refers to directly performing the dynamic MASK operation during model training; S2-3. In order to better obtain the contextual time series feature information of sentences in rare earth patents, the gated recurrent unit is introduced into the model structure of this chapter. GRU is improved to deal with the long-term memory of traditional RNN and the gradient disappearance problem in the back propagation process. Its effect of extracting feature information is similar to LSTM, but it has the characteristics of simpler structure composition and calculation process and fewer parameters. Therefore, its convergence speed has advantages, the required hardware and time overhead are relatively lower, and the generalization ability is stronger when the data set sample is small. Therefore, the GRU network is more suitable for the entity recognition task of the patent text in this article to extract the time series features of the sentence context; S2-4. Dilated convolution: Since the text of the patent for the production method of rare earth products has the characteristic of high entity density, and the size of the convolution kernel of the traditional CNN convolutional network is fixed, it can only obtain local features within the convolution window, and cannot capture multiple local features of texts of different lengths. In addition, the weights between the convolutional layers are shared, and there may be confusion when the network processes different position information. The CNN network may also have the problem of overfitting. The dilated convolutional network is also a convolutional network. It is different from the traditional CNN network in the following aspects in natural language processing: a. In terms of parameter efficiency, compared with the traditional CNN network structure, the dilated convolutional network introduces depth-separable convolutional layers and depth-convolutional layers to replace traditional convolutional layers, which can reduce the number of parameters and calculations to a certain extent; b. In terms of multi-scale local features, the dilated convolutional network can capture spatial local features of different scales by setting the dilation parameter, while the convolution kernel of the traditional CNN network is fixed; c. In terms of iteration, the dilated convolution adds an operation of iteratively dilating the convolutional network, which can expand the network's perception field of view, thereby improving the network's perception ability and local feature acquisition ability; d. In terms of long-distance dependencies, the traditional CNN network cannot obtain longer text dependencies well, while the dilated convolutional network can obtain longer text dependencies to improve the performance of the network; S2-5. Experimental parameter settings: Pytorch framework is used in the encoding process, and RoBERTa-wwm-ext model is used in the embedding layer. This pre-trained model is trained by the Joint Laboratory of Harbin Institute of Technology. The transformer in this pre-trained model has 12 layers and the dimension of the hidden layer is 768. In the model training process of the dataset in this chapter, the training cycle parameter epoch is set to 50. Because the text entity density of the dataset in this article is relatively large, the maximum text input length parameter max_length is set to 64, and the batch size parameter batch_size is set to 8; in the feature acquisition layer, the GRU network is set to bidirectional, the number of network layers is 2, and the number of hidden layers is 128. In order to prevent the model from overfitting, the dropout value is set to 0.

1. The convolution step parameter stride in the 3-layer iterative convolution layer of the IDCNN network is 1. In order to keep the shape of the feature unchanged during the convolution process, the padding parameter padding is set to dilation*kernel_size-1 / / 2, where dilation is the expansion coefficient and kernel_size is the size of the convolution kernel; S2-6. Experimental results and analysis: The time series features and spatial sequence features of the text are effective in improving the named entity recognition task. The F1 value of the BiGRU+CRF model is 0.14% higher than that of the BiLSTM+CRF model, indicating that the bidirectional gated convolutional recurrent unit network is more suitable for extracting time series features from the patent text of the rare earth product production method in this chapter than the bidirectional long short-term memory network. The F1 value of the IDCNN+CRF model is 0.24% higher than that of the CNN+CRF model, which shows that the dilated convolutional network is more suitable for obtaining spatial sequence feature information in this chapter than the traditional CNN; The F1 value of the BiGRU+IDCNN+ATT+CRF model is 0.53% and 0.93% higher than that of the BiGRU+CRF model and the IDCNN+CRF model, respectively. This shows that fusing the time series features and the space series features of the text is more effective than the time series features alone or the space series features alone, which is conducive to the complementary advantages of the two features, thereby obtaining more comprehensive text feature information; S2-7. In order to verify the effectiveness and rationality of each component of the RBIAC model proposed in this chapter, an ablation experiment was carried out, in which -ATT means removing the multi-head attention mechanism and using direct vector concatenation instead of dynamic fusion of the multi-head attention mechanism; -BiGRU means removing the bidirectional gated convolutional recurrent unit network part and only using the dilated convolutional network for feature acquisition; -IDCNN means removing the dilated convolutional network part and only using the recurrent gated unit network for feature acquisition; -FocalLoss means removing the FocalLoss loss function and using only the cross entropy loss function when calculating the model loss.

4. According to claim 1, a large-scale rare earth data information extraction method, based on vocabulary enhancement and adversarial training, patent entity recognition of rare earth product production method, comprises the following steps: S3-1. Model framework: The model processes the input text as follows: First, the input text is passed through the RoBRTa-wwm-ext embedding layer to obtain the character-level vector, and the input text is passed through the vocabulary enhancement module to obtain the vocabulary-level vector, and the character-level and vocabulary-level information are dynamically fused to obtain the embedding vector; secondly, the embedding vector is trained to generate the corresponding adversarial sample; then, the BiGRU model and IDCNN model are used to obtain the time series features and spatial series features in the two channels respectively, and the multi-head attention mechanism is used to dynamically fuse the feature encoding of the two channels; finally, it is input into the CRF model to obtain the predicted label corresponding to the input text; S3-2. Experimental data set and parameters: In the experiment of the MSRA data set model, the number of transformer layers in the pre-trained model is 12, and the hidden layer dimension is 768; in order to fully train the model, the epoch parameter representing the training cycle is set to 50; since the text sentences in the MSRA data set come from the text in the news field, it is somewhat challenging for the model to understand. In order to allow the model to fully utilize the feature encoding information of the sentence to identify entities, the maximum text input length parameter max_length is set to 128; the batch size parameter batch_size is set to 8; in the feature acquisition layer, the GRU network is set to bidirectional, the number of network layers is 1, and the number of hidden layers is 128; in order to prevent the model from overfitting during training, the parameter max_length is set to 128. The dropout value is set to 0.5, and the convolution step parameter stride in the three-layer iterative convolution layer of the IDCNN network is 1. In order to keep the shape of the convolution feature unchanged, the padding parameter padding is set to dilation*kernel_size-1 / / 2, where dilation is the expansion coefficient of 1, 1, and 2 respectively, and kernel_size is the size of the convolution kernel; In addition, the optimizer used in this model is Adam, which can dynamically adjust the learning rate of the model parameters to minimize the loss of the model. The MSRA dataset is one of the four classic datasets in the current Chinese NER task. It comes from the text in the news field and includes three types of entities: address entities, person entities, and organization name entities; S3-3. Experimental results and analysis: Based on the classic dataset MSRA in the current Chinese NER task and the RPNER dataset constructed in this paper, a number of comparative experiments and ablation experiments were carried out. The comparative experiments include: comparing with the current mainstream Chinese NRE task model, exploring the integration of different k words for comparison, and comparing different adversarial training methods in the NLP field; and the ablation experiment is to verify the impact on the overall effect of the model by removing the newly introduced modules in this chapter. The specific experimental design is as follows: a. Comparative experiment with current mainstream models: This chapter uses the mainstream models in Chinese NER tasks in recent years to compare with the CWAFC model proposed in this chapter on the Chinese classic MSRA dataset to verify the universality and effectiveness of the model in this chapter; b. Explore different A comparative experiment on word pair models: This experiment uses different The words are integrated into the embedding layer to explore the impact on the model effect; c. Comparative experiment based on different adversarial training methods: This experiment mainly uses the three mainstream adversarial training methods in the field of natural language processing to test the effects on the model in this chapter; d. Ablation experiment: This experiment removes the modules introduced in this paper to solve the problems existing in this model, so as to verify the impact of the introduced modules on the model in this chapter.