Text structure element automatic identification method based on hierarchical neural network

Abstract

The invention discloses an automatic text structural element recognition method based on a hierarchical neural network, and the method achieves the efficient and automatic recognition of structural elements in a target document abstract through integrating the context information of a word level and a sentence level, and optimizing a tag sequence through an attention mechanism and a conditional random field. According to the method, a Chinese scientific and technical literature structural abstract data set CSL Structured 2K is constructed, published and published, the Weighted F1 value is 93.38% and the Macro F1 value is 88.4% on the basis of experimental verification of a published English data set PubMed 20K, and the Weighted F1 value is 88.69% and the Macro F1 value is 88.86% on the basis of verification of a CSL Structured 2K data set. By comparing models such as BiLSTM-CRF, BERT, SciBERT, HSLN, BERT-HSLN, SciBERT-HSLN and the like, the performance on a reference data set exceeds the current most advanced result by 0.6%-1.3%, and it is verified that the method has good performance in automatic recognition tasks of abstract structural elements of scientific and technical literatures.

Description

Technical Field

[0001] The present application relates to the technical field of automatic identification of structural elements of scientific and technological literature abstracts, and in particular to a method for automatic identification of text structural elements based on a hierarchical neural network. Background Art

[0002] As a core information summary of scientific research results, scientific and technological literature abstracts usually include structural elements such as the background, purpose, methods, results and conclusions of the research. These abstracts are the key for researchers to quickly obtain key information, but in actual applications, the vast majority of literature abstracts are presented in an unstructured form, resulting in low efficiency in information extraction and increasing the time cost of researchers to screen and analyze literature.

[0003] Due to the lack of clear structural labels, traditional machine learning models do not perform well when extracting specific information (such as research objectives, experimental methods, etc.) from unstructured text. Most existing natural language processing (NLP) models rely on manually defined rules or shallow machine learning algorithms. These methods require a lot of domain knowledge support and are difficult to deal with literature abstracts in different fields or multiple languages. Imperfect label sequence optimization: Some models have difficulty handling the sequential dependencies between sentences, resulting in the identified structural element labels not conforming to the logical order of the abstract text, affecting the accuracy of automated information extraction. Lack of high-quality Chinese datasets: Although there are some annotated datasets in the English field (such as PubMed 20k), there is a lack of similar structured datasets in the Chinese field, which limits the development and application of information extraction technology for Chinese scientific literature.

[0004] The current methods for identifying the structural elements of scientific literature abstracts can be divided into three categories:

[0005] Rule-based approach: Relies on grammatical patterns and keyword rules manually defined by experts, but the rules are prone to conflict and are not well adaptable to changes in domains and text formats.

[0006] Methods based on traditional machine learning: A lot of feature engineering is required, and the optimization ability of label sequences is limited, making it difficult to capture long-distance dependencies in the context.

[0007] Models based on simple deep learning: such as LSTM, BiLSTM and other methods, although they can capture contextual information to a certain extent, they still have shortcomings in high-order semantic understanding and label optimization. Summary of the invention

[0008] The present application provides a method for automatically identifying text structure elements based on a hierarchical neural network, aiming to solve the problem that the prior art relies on a large amount of manual feature extraction or predefined rules and has limited performance when facing complex semantic structures.

[0009] A method for automatically identifying text structure elements based on a hierarchical neural network, the method comprising:

[0010] Input each sentence of the target scientific and technological literature abstract text into the word embedding layer, and encode each word in the sentence into a vector representation based on the pre-trained language model of the BERT model; wherein the target scientific and technological literature abstract includes Chinese scientific and technological literature abstract and English scientific and technological literature abstract;

[0011] Generate a representation rich in contextual information between words within a sentence through a bidirectional long short-term memory network in the sentence encoding layer, and generate a sentence vector through an attention pooling mechanism;

[0012] Input the sentence vector into the context information enrichment layer, make each sentence vector contain context information from surrounding sentences through the bidirectional long short-term memory network, and learn the high-order semantic information of the sentence through the multi-head attention mechanism;

[0013] Combining residual connections and layer normalization within the contextual information enrichment layer;

[0014] The label scores are obtained through a feedforward neural network, and the predicted label sequences are optimized through conditional random fields, and finally the structural elements in the identified scientific literature abstracts are obtained, wherein the structural elements include background, purpose, method, result and conclusion.

[0015] In the above scheme, optionally, the word embedding layer adopts two pre-trained language models: BERT-Base-Chinese and SciBERT;

[0016] Among them, the BERT-Base-Chinese is used to process Chinese scientific literature abstracts, and the SciBERT is used to process English scientific literature abstracts.

[0017] In the above scheme, optionally, the sentence encoding layer generates a contextual representation of the sentence by the following steps:

[0018] Use a bidirectional long short-term memory network to scan the word sequence in the sentence from forward and backward directions respectively to generate a hidden state sequence;

[0019] Based on the attention pooling mechanism, the weights of words are dynamically adjusted according to context information to generate sentence-level representation vectors.

[0020] In the above scheme, optionally, the multi-head attention mechanism process of the context information enrichment layer includes the following steps:

[0021] The sentence encoding vector sequence generated by the sentence encoding layer is used as input, and the context information between sentences is learned through a bidirectional long short-term memory network;

[0022] Divide each sentence vector into multiple subspaces;

[0023] Perform self-attention calculations in each subspace, calculate query matrix, key matrix, and value matrix, and capture the dependencies between sentences through weighted calculations between these matrices;

[0024] The attention outputs of each subspace are concatenated into the overall high-order semantic representation of the sentence.

[0025] In the above scheme, optionally, the context information enrichment layer uses residual connection to add input and output, alleviates the gradient vanishing problem, and maintains the stability of network training through layer normalization.

[0026] In the above solution, optionally, the conditional random field performs label sequence optimization at the sentence level, specifically including:

[0027] Construct a transfer matrix to describe the sequential dependencies between the labels of structural elements;

[0028] Generate the optimal label sequence based on the transition probability of the sentence label and the label score predicted by the model.

[0029] In the above scheme, optionally, the method is trained using a Chinese structured summary dataset CSL Structed2K, which contains 2,000 scientific and technological literature abstracts, a total of 8,584 annotated sentences, covering the fields of engineering and medicine.

[0030] In the above scheme, optionally, the method performs model verification by the following steps:

[0031] The performance is verified on the PubMed 20k English dataset, which contains 20,000 biomedical literature abstracts.

[0032] Performance verification is conducted on the CSL Structed 2K Chinese dataset to evaluate the performance of the model in the task of identifying structural elements of Chinese summaries.

[0033] In the above scheme, optionally, the hierarchical neural network model is optimized by the following hyperparameters:

[0034] Maximum text length: defines the maximum length of a sentence, which is 64 for the English dataset and 128 for the Chinese dataset;

[0035] Batch size: defines the number of samples for each training, set to 64;

[0036] Learning rate: Set the initial learning rate to 3e-5 and set the learning rate decay parameter to 0.9;

[0037] Dropout ratio: set to 0.5 on the PubMed 20k dataset and 0.3 on the CSL Structed 2K dataset.

[0038] In the above scheme, optionally, the method adopts a semi-automatic annotation strategy to construct the Chinese structured summary dataset CSL Structed 2K, and the annotation strategy includes:

[0039] Use the preset large language model to perform preliminary automatic annotation of scientific literature abstracts to identify structural elements such as background, purpose, method, results and conclusions;

[0040] Based on the preliminary annotation, the manual review and correction team verifies the annotation results generated by the large language model to ensure the accuracy of the annotation content;

[0041] Remove redundant information and correct multiple annotation errors to ensure that each sentence is correctly annotated as only one structural element;

[0042] The final annotation results are imported into the CSL Structed 2K dataset, and the dataset is publicly released for further research and model training.

[0043] Compared with the prior art, this application has at least the following beneficial effects:

[0044] Based on further analysis and research on the problems of the prior art, the present application recognizes that the prior art relies on a large number of manual feature extractions or predefined rules, and has limited performance when faced with complex semantic structures. The present application inputs each sentence of the target scientific literature abstract text into a word embedding layer, and encodes each word in the sentence into a vector representation based on a pre-trained language model of the BERT model; wherein the target scientific literature abstract includes Chinese scientific literature abstracts and English scientific literature abstracts; a representation rich in contextual information between words within the sentence is generated through a bidirectional long short-term memory network in the sentence encoding layer, and a sentence vector is generated through an attention pooling mechanism; the sentence vector is input into a contextual information enrichment layer, and the sentence vector is passed through a bidirectional long short-term memory network so that each sentence vector contains contextual information from surrounding sentences and learns the high-order semantic information of the sentence through a multi-head attention mechanism; residual connections and layer normalization are combined in the contextual information enrichment layer; the predicted label sequence is optimized through a conditional random field, and the structural elements in the scientific literature abstract are automatically identified, and the structural elements include background, purpose, method, result and conclusion.

[0045] This application captures the deep semantic information of sentences and improves the recognition accuracy of abstract structural elements through a pre-trained language model based on the BERT model. A bidirectional long short-term memory network is used to represent the contextual information between words within a sentence and perform attention pooling. Then, a bidirectional long short-term memory network is used in combination with a multi-head attention mechanism to capture contextual information at the sentence level and adapt to abstract texts in different fields and languages. The conditional random field (CRF) is introduced to optimize the label sequence to ensure that the logical order of the labels is consistent with the content of the abstract, avoiding conflicts and errors in label prediction. The CSL Structed 2K dataset was constructed using a semi-automatic annotation strategy and publicly released, providing high-quality training data for the recognition of structural elements of Chinese scientific and technological literature abstracts.

[0046] The technical solution proposed in this application can effectively solve the deficiencies in the existing technology, automatically and efficiently identify structural elements from unstructured abstracts, and improve the accuracy and efficiency of document information extraction. The model's multi-language support and label sequence optimization functions make it more adaptable. In addition, the annotation strategy based on a combination of large language models and manual review fills the gap in Chinese data sets and promotes the development of automatic identification technology for structural elements of Chinese scientific literature abstracts. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic diagram of a process flow of a method for automatically identifying text structure elements based on a hierarchical neural network provided in one embodiment of the present application;

[0048] Figure 2 A schematic diagram of a large model annotation result provided in one embodiment of the present application;

[0049] Figure 3 A schematic diagram of a prompt template for a method for automatically identifying text structure elements based on a hierarchical neural network provided in one embodiment of the present application;

[0050] Figure 4 A schematic diagram of a single abstract annotation of a method for automatic identification of text structure elements based on a hierarchical neural network provided in one embodiment of the present application;

[0051] Figure 5 A schematic diagram of a single abstract of a scientific document of a method for automatically identifying text structure elements based on a hierarchical neural network provided in one embodiment of the present application;

[0052] Figure 6 A model structure diagram of a method for automatically identifying text structure elements based on a hierarchical neural network provided in one embodiment of the present application;

[0053] Figure 7 A schematic diagram of the PubMed 20k experimental confusion matrix provided for one embodiment of the present application;

[0054] Figure 8 A schematic diagram of a confusion matrix of a CSL Structed 2k experiment provided for one embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] In one embodiment, Figure 1 As shown, a method for automatically identifying text structure elements based on a hierarchical neural network is provided, comprising the following steps:

[0057] Input each sentence of the target scientific and technological literature abstract text into the word embedding layer, and encode each word in the sentence into a vector representation based on the pre-trained language model of the BERT model; wherein the target scientific and technological literature abstract includes Chinese scientific and technological literature abstract and English scientific and technological literature abstract;

[0058] The sentence vector is generated by converting it into a representation rich in contextual information between words within the sentence through a bidirectional long short-term memory network and performing attention pooling; the sentence vector is input into the contextual information enrichment layer, and the sentence vector passes through the bidirectional long short-term memory network so that each sentence vector contains contextual information from surrounding sentences and learns the high-order semantic information of the sentence through a multi-head attention mechanism;

[0059] Combining residual connections and layer normalization within the contextual information enrichment layer;

[0060] Labels are predicted through linear transformation, and the predicted label sequences are optimized through conditional random fields to automatically identify structural elements in scientific literature abstracts, including background, purpose, method, result and conclusion.

[0061] In this embodiment, the word embedding layer uses two pre-trained language models, BERT-Base-Chinese and SciBERT;

[0062] Among them, the BERT-Base-Chinese is used to process Chinese scientific literature abstracts, and the SciBERT is used to process English scientific literature abstracts.

[0063] In this embodiment, the sentence encoding layer generates a contextual representation of a sentence by the following steps:

[0064] Use a bidirectional long short-term memory network to scan the word sequence in the sentence from forward and backward directions respectively to generate a hidden state sequence;

[0065] Based on the attention pooling mechanism, the weights of words are dynamically adjusted according to context information to generate sentence-level representation vectors.

[0066] In this embodiment, the bidirectional long short-term memory network process of the context information enrichment layer includes the following steps:

[0067] Learn the contextual information of surrounding sentences through a bidirectional long short-term memory network

[0068] In this embodiment, the multi-head attention mechanism process of the context information enrichment layer includes the following steps:

[0069] Divide each sentence vector into multiple subspaces;

[0070] Perform self-attention calculations in each subspace, calculate query matrix, key matrix, and value matrix, and capture the dependencies between sentences through weighted calculations between these matrices;

[0071] The attention outputs of each subspace are concatenated into the overall high-order semantic representation of the sentence.

[0072] In this embodiment, the context information enrichment layer uses residual connection to add input and output, alleviates the gradient vanishing problem, and maintains the stability of network training through layer normalization.

[0073] In this embodiment, the conditional random field performs label sequence optimization at the sentence level, specifically including:

[0074] Construct a transfer matrix to describe the sequential dependencies between the labels of structural elements;

[0075] Generate the optimal label sequence based on the transition probability of the sentence label and the label score predicted by the model.

[0076] In this embodiment, the method is trained using the Chinese structured abstract dataset CSL Structed 2K, which contains 2,000 scientific and technological literature abstracts, a total of 8,584 annotated sentences, covering the fields of engineering and medicine.

[0077] In this embodiment, the method performs model verification by the following steps:

[0078] The performance is verified on the PubMed 20k English dataset, which contains 20,000 biomedical literature abstracts.

[0079] Performance verification is conducted on the CSL Structed 2K Chinese dataset to evaluate the performance of the model in the task of identifying structural elements of Chinese summaries.

[0080] In this embodiment, the hierarchical neural network model is optimized by the following hyperparameters:

[0081] Maximum text length: defines the maximum length of a sentence, which is 64 for the English dataset and 128 for the Chinese dataset;

[0082] Batch size: defines the number of samples for each training, set to 64;

[0083] Learning rate: Set the initial learning rate to 3e-5 and set the learning rate decay parameter to 0.9;

[0084] Dropout ratio: set to 0.5 on the PubMed 20k dataset and 0.3 on the CSL Structed 2K dataset.

[0085] In this embodiment, the method adopts a semi-automatic annotation strategy to construct the Chinese structured formula summary dataset CSLStructed 2K, and the annotation strategy includes:

[0086] Use the preset large language model to perform preliminary automatic annotation of scientific literature abstracts to identify structural elements such as background, purpose, method, results and conclusions;

[0087] Based on the preliminary annotation, the manual review and correction team verifies the annotation results generated by the large language model to ensure the accuracy of the annotation content;

[0088] Remove redundant information and correct multiple annotation errors to ensure that each sentence is correctly annotated as only one structural element;

[0089] The final annotation results are imported into the CSL Structed 2K dataset, and the dataset is publicly released for further research and model training.

[0090] The present application inputs each sentence of the target scientific literature abstract text into the word embedding layer, and encodes each word in the sentence into a vector representation based on the pre-trained language model of the BERT model; wherein the target scientific literature abstract includes Chinese scientific literature abstract and English scientific literature abstract; generates sentence-level context information representation through the bidirectional long short-term memory network in the sentence encoding layer, and generates sentence vectors through the attention pooling mechanism; inputs the sentence vector into the context information enrichment layer, and the sentence vector passes through the bidirectional long short-term memory network so that each sentence vector contains context information from surrounding sentences and learns the high-order semantic information of the sentence through the multi-head attention mechanism; combines residual connection and layer normalization in the context information enrichment layer; predicts labels through a feedforward neural network, optimizes the predicted label sequence through a conditional random field, and automatically identifies the structural elements in the scientific literature abstract, wherein the structural elements include background, purpose, method, result and conclusion.

[0091] This embodiment captures the deep semantic information of sentences and improves the recognition accuracy of abstract structural elements through a pre-trained language model based on the BERT model. The context information representation at the sentence level is generated through the bidirectional long short-term memory network in the sentence encoding layer, and the sentence vector is generated through the pooling mechanism of attention; the use of a bidirectional long short-term memory network combined with a multi-head attention mechanism can capture context information at the sentence level and adapt to abstract texts in different fields and languages. The conditional random field (CRF) is introduced to optimize the label sequence to ensure that the logical order of the labels is consistent with the content of the abstract, avoiding conflicts and errors in label prediction. The CSL Structed 2K dataset was constructed using a semi-automatic annotation strategy and publicly released, providing high-quality training data for the recognition of structural elements of Chinese scientific and technological literature abstracts.

[0092] The technical solution proposed in this embodiment can effectively solve the deficiencies in the prior art, automatically and efficiently identify structural elements from unstructured abstracts, and improve the accuracy and efficiency of document information extraction. The model's multi-language support and label sequence optimization functions make it more adaptable. In addition, the annotation strategy based on a combination of a large language model and manual review fills the gap in Chinese data sets and promotes the development of automatic identification technology for structural elements of Chinese scientific and technological literature abstracts.

[0093] In one embodiment, in order to improve the efficiency of scientific researchers in searching and analyzing documents and the utilization rate of knowledge, and to promote the application of models in Chinese scientific and technological literature corpus, this embodiment proposes a hierarchical neural network model (BERT-MHA-Res-HSLN) for unstructured abstracts of scientific and technological literature to identify structural elements in unstructured abstracts. The model first encodes the original corpus into a vector representation through a word embedding method, and then fully mines the sequence feature information between words in the sentence through a long short-term memory network (LSTM), and learns reasonable feature weights in combination with a self-attention pooling mechanism, and then further captures the contextual information between sentences through a bidirectional long short-term memory network, and obtains high-order semantics in combination with a multi-head attention mechanism, and finally optimizes the predicted label sequence through a conditional random field CRF. Experimental results show that the model proposed in this embodiment has high performance. On an English public data set, the Weighted F1 value is 93.38% and the Macro F1 value is 88.4%. Compared with the BERT model alone, the Weighted F1 value is increased by 3.45%. On the publicly released CSL Structed 2K Chinese dataset, the model's accuracy reached 88.68%, the WeightedF1 value was 88.69%, and the Macro F1 value was 88.86%. Comprehensive comparison, the model's performance surpassed the most advanced results by 0.6%-1.3%.

[0094] Against the backdrop of the rapid development of deep learning and large language models, large-scale, high-quality datasets remain the core cornerstone of natural language processing (NLP) progress. With the introduction of more and more high-quality open source text datasets, researchers have access to richer data resources, providing a solid foundation for technological innovation and algorithm optimization.

[0095] The PubMed 20k dataset covers about 20,000 annotated scientific abstracts from the biomedical field. The data is derived from the PubMed database. Due to its large size and high-quality annotations, PubMed20k has become a benchmark dataset for evaluating the performance of machine learning models in the biomedical field. The abstracts in this dataset are preprocessed and segmented into single sentences. Each sentence is divided into the following five categories according to its content: Background, Objective, Methods, Results, and Conclusions.

[0096] The NICTA-PIBOSO dataset covers 1,000 medical literature abstracts, and each sentence is classified into the following six categories according to its role and content in the abstract: Background, Intervention, Study, Population, Outcome, and Other. Researchers have widely used this dataset to develop and test various machine learning models to help automatically analyze and process key information in medical literature, especially in the classification of medical research conclusions and results.

[0097] The ART corpus dataset further enriches the annotation system of scientific literature. The corpus annotates core scientific concepts of 225 scientific papers, covering the fields of physical chemistry and biochemistry, and covers more than 1 million words and 35,040 sentences. The annotation categories include Background, Motivation, Goal, Hypothesis, Object, Model, Method, Experiment, Result, Observation, and Conclusion.

[0098] Although many high-quality open source datasets have been developed, most of them are still in English, while Chinese datasets are very scarce. To fill this gap, this embodiment constructs the CSL Structured 2K dataset based on the Chinese Scientific Literature Dataset (CSL

[20] ). The dataset contains 2,000 abstracts of scientific and technological literature from the fields of engineering and medicine, totaling 8,584 sentences. Each sentence is labeled into one of the following five categories according to its content: background, objective, method, result, and conclusion. This dataset provides a basis for further advancing Chinese natural language processing tasks, especially in the fields of automatic identification of abstract structural elements. It has important application value.

[0099] Large language models have shown impressive results in natural language understanding and other tasks with their rich knowledge reserves and powerful reasoning capabilities. The emergence of advanced large language models represented by GPT-4 has provided unprecedented opportunities for automated annotation of complex data. In recent years, with the emergence of large models such as ChatGPT, GPT-4, ChatGLM, Wenxin Yiyan, and Tongyi Qianwen, a variety of options have been provided for data annotation.

[0100] Prompt Engineering refers to the design and construction of appropriate prompts or instructions to guide large language models to generate specific types of outputs. It plays a key role in improving model performance, controlling output quality, and meeting specific needs. However, LLM may have "hallucination phenomena" and generate inaccurate information. Therefore, in this study, in the process of building the CSL Structed 2K dataset, we adopted a strategy that combines the automated processing capabilities of large language models with the accuracy of manual review. This strategy significantly improves the efficiency of data annotation, while also enhancing the reliability of data annotation, thereby ensuring the quality of the dataset.

[0101] For the automatic identification of structural elements of unstructured abstracts of scientific literature, the current main research methods can be divided into three categories: rule-based, machine learning-based, and deep learning-based methods.

[0102] Rule-based methods identify specific information in text by defining a series of rules. These rules are usually manually formulated by domain experts and involve grammatical patterns, sentence structures, and specific keywords or phrases. The formulation and maintenance of rules require a lot of domain knowledge and manpower input, and are more sensitive to new fields or text formats that change greatly, and are prone to conflicts between rules. Houngbo et al. analyzed the grammatical patterns of sentences describing methods in papers and used regular expressions to represent syntactic rules to extract research methods. Zheng Mengyue et al. analyzed the writing characteristics of unstructured abstracts, divided the text into three elements at the sentence level: purpose, method, result, or conclusion, and manually summarized and counted the sentence patterns, clue words, and positions in the abstracts to establish a relevant rule library.

[0103] Machine learning-based methods need to convert text into numerical features and rely on high-quality feature engineering. The performance of the model is limited by the selection of features and the quality of data annotation. Yamamoto uses the SVM method to automatically identify structural elements; Ding Liangping et al. further analyzed the impact of factors such as sample size, N value in N-gram, whether to remove stop words, and word frequency weighting on the effect of the support vector machine model on predicting the structure elements of the summary.

[0104] The method based on deep learning can automatically extract the potential features in the language sequence from the text data. Jin et al. proposed a hierarchical sequence labeling network (HSLN) for the classification of sequence sentences of scientific and technological literature abstracts in the medical field, which achieved the best performance on the PubMed large-scale data set. Among them, the hierarchical attention network (HAN) used is a neural network for document classification. It applies the attention mechanism at the character and sentence level, which provides an inspiring thinking for the construction of the model in this embodiment. Zhang Zhixiong et al. explored the recognition effect of different deep learning models on the language steps of scientific and technological paper abstracts, and compared the effects of deep models such as SVM, DNN, LSTM and Attention-BiLSTM in the field of language step recognition. The experiment showed that the method of combining the bidirectional long short-term memory network and the attention mechanism can more accurately identify the structural elements such as purpose, method, and result in the abstract. The attention mechanism can help the model better utilize the effective information in the input, and provide a certain explanatory ability for the behavior of the neural network model. Google's BERT model is another milestone in the field of text processing. As a pre-trained language model, BERT is trained in both directions (left and right) on large-scale text data, which can capture the semantic information in the text. Zhang Zhixiong et al. applied the BERT model to summary step recognition. In a large-scale sample experiment, the pre-trained model BERT achieved the best results. Liu Zhongbao et al. compared the effects of BiRNN, BiLSTM, BiLSTM-CRF and BERT models on text summary structural element recognition, indicating that the BERT model is most suitable for summary structural element recognition tasks. Guo Hangcheng et al. combined sentence embedding and CRF to propose the Paragraph-BERT-CRF structural element recognition method, which fully utilized the paragraph context information in the text.

[0105] Based on the above work, it is shown that the rule-based method requires high labor costs and relies on manually defined rules. The machine learning-based method relies on high-quality feature engineering. The deep learning-based method can effectively mine text feature information from massive documents, is suitable for processing large-scale data, and has strong automatic feature extraction capabilities and good adaptability. It performs well in natural language processing tasks and has the potential to further improve the performance of methods in this field. Therefore, this embodiment conducts a more in-depth study on the application of deep learning methods in this field, and improves the performance of the method through model design and improvement.

[0106] In order to construct a high-quality dataset of structured abstracts of Chinese scientific literature, the following principles are mainly followed during construction and annotation: (1) logical correctness and executable annotation tasks, with clear annotation goals. (2) Detailed auxiliary prompt information so that the model can better understand and execute tasks.

[0107] Based on the above principles, a semi-automatic method combining manual verification and large model annotation is used to construct the dataset. The annotation process is as follows: Figure 2 As shown in the figure. After preprocessing the data and building the prompt template, the large model will output the preliminary results, which will be manually verified to ensure the accuracy and quality of each sentence. Finally, a high-quality scientific literature structured abstract dataset named CSL Structed 2K was constructed.

[0108] This embodiment uses the Chinese scientific literature open source dataset (CSL) as a data source, and screens and retains abstracts with longer texts in the fields of engineering and medical technology.

[0109] During the data preprocessing process, it was observed that Chinese symbols and English symbols coexisted in the documents, the data format was not unified, and contained a large number of special characters, such as typesetting characters. In order to avoid confusion and ambiguity, this embodiment unified the data format and symbols, converted English punctuation marks into Chinese symbols, and removed special symbols.

[0110] A well-designed prompt template can reduce the trouble of parsing the output. This embodiment constructs prompt templates to make GLM-4 as a structural element recognition model for unstructured abstracts of scientific and technological literature, thereby assisting in the construction of the CSL Structed 2K dataset. Figure 3 As shown, it specifically includes five parts: task definition, structural element definition, structural element identification description, output requirements and input content.

[0111] In this example, a total of 2,000 scientific and technological literature abstracts were annotated, of which 70% were in the field of engineering and 30% were in the field of medicine. The structural elements extracted from each abstract were manually verified, and a total of 8,584 sentences were marked. The sentence statistics are detailed in Table 1. During the annotation process, problems such as annotation errors, redundant output content, annotation content exceeding the specified structural element range, and the same sentence being incorrectly annotated as multiple structural elements occurred. After correction, the annotation results of each structural element were finally obtained, as shown in Table 2. At the same time, Figure 4 Shows an example of annotation for an abstract in the dataset.

[0112] Table 1 Statistics of CSL Structed 2K dataset

[0113]

[0114] Table 2 Labeling results of different structural elements

[0115]

[0116] Abstracts of scientific and technological literature usually contain multiple structural elements, and each structural element usually corresponds to multiple sentences, such as Figure 5 As shown, the structural elements in the abstracts of scientific and technological documents usually have a strong sequential nature. For example, "Conclusion" is unlikely to be before "Method". Therefore, this embodiment regards the problem of automatic identification of structural elements in unstructured abstracts of scientific and technological documents as a sequence labeling task of dividing labels according to sentences at the abstract level.

[0117] In order to fully learn the contextual information between words and sentences in the summary and the high-order semantics of the sentences, and to obtain the optimal label sequence, this embodiment proposes a hierarchical neural network model, such as Figure 6 As shown in the figure, it consists of four parts: word encoding layer, sentence encoding layer, context information enrichment layer and label sequence optimization layer.

[0118] The word embedding layer performs batch encoding on scientific literature abstract texts.

[0119] The sentence encoding layer is converted into a representation rich in contextual information between words within the sentence through BiLSTM and attention pooling is performed to generate a sentence vector.

[0120] In the context information enrichment layer, the sentence vectors are transformed through BiLSTM so that each sentence vector contains the context information from the surrounding sentences and learns the high-order semantic information of the sentence through the multi-head attention mechanism.

[0121] The label sequence optimization layer predicts labels through linear transformation and optimizes label sequences through CRF.

[0122] Since the model cannot process text directly, and considering the characteristics of different languages ​​and word segmentation requirements, for English and Chinese datasets, different pre-trained language model word segmenters are used to perform batch encoding on the text.

[0123] For the English data set, this embodiment uses the word segmenter of the pre-trained language model SciBERT to perform batch encoding processing on the text. SciBERT is a pre-trained language model based on BERT introduced by Beltagy et al., which aims to solve the problem of lack of high-quality, large-scale annotated scientific data. SciBERT uses the same architecture as BERT, but is pre-trained on a large corpus of 1.14 million scientific documents, which mainly come from computer science in the biomedical field. In order to better match the training corpus, SciBERT has established a special vocabulary (SciVOCAB). According to the vocabulary, the word segmenter divides the sentences in the text into a series of tokens and maps these tokens to corresponding indexes in the vocabulary, so that the text can be digitized and input into the model for processing.

[0124] In order to improve the performance of the model in Chinese natural language processing tasks, this embodiment adopts a pre-trained language model BERT-Base-Chinese for the Chinese data set. The model is pre-trained on the Chinese corpus, and its parameters are optimized and adjusted.

[0125] The BERT embedding layer consists of the following three parts: token embeddings, which convert each word into a vector with a fixed dimension; segment embeddings, which are used to distinguish different sentences; and position embeddings, which are used to represent the position relationship of words and capture the word order information in the text. i The word embedding output is denoted as w i (w i,1 ,···,w i,m ) Where m: the number of words in a sentence; d: the dimension of the embedding vector.

[0126] This layer takes the embedding vector of each token of the sentence in the word encoding layer as input, passes through a bidirectional long short-term memory network (BiLSTM) and attention pooling, and generates a sentence representation vector to encode the sentence.

[0127] In a single sentence, the appearance of a word is closely related to contextual information. The unidirectional LSTM model can only consider the impact of previous input information on the current content, but cannot simultaneously consider the impact of subsequent information. Therefore, a bidirectional long short-term memory network (BiLSTM) is added to the representation of a single sentence. The BiLSTM consists of a forward LSTM that scans the previous information from front to back and a backward LSTM that scans the following information from back to front, so as to learn the contextual information and semantics between internal words in a single sentence. Sentence S i The word embedding w i After BiLSTM processing, a hidden state sequence h is output i (h i,1 ,...,h i,m ) Where h is the size of the hidden layer. LSTM calculates represents the hidden state of the forward LSTM; the backward LSTM calculates Represents the hidden state of the backward LSTM. The outputs in both directions are concatenated, and the hidden layer output at position t is h i,t , see formula (1);

[0128]

[0129] In order to generate the representation vector e of the sentence i , using an attention-based pooling method, attention pooling learns the weight of each word, enabling the model to dynamically adjust the weight according to the context, thereby better capturing important features.

[0130] A i =softmax(U i tanh(W i h i +b i ) (2)

[0131] e i =A i H i T (3)

[0132] Among them, h i =[h i,1 ,…,h i,m ], W i Soft alignment transformation matrix, b i : bias term, Where u: hidden vector used to represent context information in the attention mechanism; U i : The word-level context matrix is ​​used to measure the relevance or importance of each token relative to the entire sentence. A: Attention matrix, After dimension transformation, the output is finally recorded as E(e1,...,e n ), Where, s: the number of attention weight vectors.

[0133] This layer innovatively combines sentence-level bidirectional long short-term memory networks, multi-head attention mechanisms, residual connections, and normalization to effectively capture the contextual information and high-order semantics between sentences in the summary, while alleviating the degradation problem in deep networks. Specifically, the encoding vector sequence of n sentences generated by the sentence encoding layer is used as input, and each vector corresponds to a sentence. After being processed by the context information enrichment layer, a new vector sequence A(a1,...,a n ).

[0134]

[0135] In the abstract, the appearance of a sentence is closely related to the context information. Through the bidirectional long short-term memory network, the context information of the surrounding sentences is learned, and the output is recorded as B(b1,...,b n )

[0136] The vector sequence B after bidirectional LSTM is input into the multi-head attention mechanism. Multi-Head Attention is an extended form of the self-attention mechanism. When encoding the information of the current position, the self-attention mechanism will over-focus on its own position. The multi-head attention mechanism can solve this problem. It divides the input data into multiple heads, performs self-attention calculation on each head, and finally splices the results of multiple heads to obtain the final output. The output contains the encoded representation information in different subspaces, thereby enhancing the expressiveness of the model.

[0137] Divide the vector sequence into multiple heads. Assume there are l heads with a dimension of Each head undergoes linear transformation to obtain the query matrix Q, key matrix K and value matrix V, see formula (4);

[0138]

[0139] in, is the parameter matrix obtained through learning.

[0140] For the i-th head i Perform self-attention calculations respectively, see formula (5);

[0141]

[0142] Splice l heads together, see formula (6);

[0143] MultiHead(Q,K,V)=concat(head1,head2,…,head l )W o (6)

[0144] Finally, the output is recorded as M(m1,...,m n ),

[0145] In order to alleviate the degradation problem and improve performance when the network depth is increased, the model uses residual connections to effectively train deep networks. The output after residual connection is recorded as A, A = M + B. Use the LayerNorm function to perform layer normalization, calculate the new Z vector, and input it to the label sequence optimization layer. This layer takes the Z vector generated by the context information enrichment layer as input, and processes it through the feedforward neural network to obtain L(l1,...,l n ). i Each component in contains the corresponding label score.

[0146] In general sentence classification problems, each sentence is classified separately. However, in scientific literature abstracts, each sentence should be assigned specific structural elements in order, and the structural elements of the sentence are usually related to the order of its context, such as sentences describing "Method" usually appear before sentences describing "Result". BiLSTM is good at processing long-distance text information, and the multi-head attention mechanism can obtain high-order semantics of vectors, but neither can handle the dependencies between adjacent labels. The conditional random field can generate the optimal prediction sequence by considering the relationship between adjacent labels. Therefore, this embodiment relies on the conditional random field to model the interdependence between labels, and constrains the structural elements of the abstract by designing a transfer matrix to make it conform to the sequential dependency and improve the structural element recognition performance.

[0147] The data sources are structured abstract datasets CSL Structed 2K and PubMed 20k. The PubMed20k dataset is a text dataset consisting of 20,000 biomedical research abstracts extracted from PubMed. The detailed abstract of each document includes five structural elements: Background, Objective, Methods, Results, and Conclusions. The dataset contains 15,000 abstracts in the training set, 2,500 abstracts in the test set, and 2,500 abstracts in the validation set. Table 3 shows an example abstract of an annotated structural element, and Table 4 shows the statistical information of each structural element.

[0148] Table 3 PubMed 20k abstract number 24290286 information

[0149]

[0150]

[0151] Table 4 Statistical results of different structural elements of PubMed 20k

[0152]

[0153] The experimental environment used in this embodiment is shown in Table 5 below:

[0154] Table 5 Experimental environment configuration

[0155]

[0156] This embodiment is tested under the above experimental environment, and the experimental parameters are shown in Table 6 and Table 7:

[0157] Table 6 PubMed 20k model parameters

[0158]

[0159]

[0160] Table 7 CSL Structed 2K model parameters

[0161]

[0162] Considering the average number of words in a single sentence in the summary of the Pubmed 20k dataset, this embodiment sets the input length of a single sentence to 64, fills sentences with a length less than 64, and truncates sentences with a length greater than 64. Considering the average sentence length of CSL Structed 2K, the input length of a single sentence is set to 128. Since BERT has performed a large number of parameter updates during the pre-training process, it is usually necessary to use a smaller learning rate to avoid damage to the pre-training parameters. The learning rate is set to 3e-5. After multiple experiments, it was verified that when the Dropout of the Pubmed 20k dataset was 0.5 and the Dropout of the CSL Structed 2K dataset was 0.3, the experimental effect reached the best.

[0163] This embodiment selects accuracy, weighted F1, and macro F1 as model evaluation indicators. Accuracy refers to the proportion of samples with correct classification prediction to the total number of samples; Weighted F1 considers the number of samples in each category and weights the F1 value of each category; Macro F1 calculates the F1 value of each category and then calculates the arithmetic average of the scores.

[0164]

[0165] Among them, i: i-th category; n i : the number of samples of class i; N: the total number of samples; TP: positive samples with positive prediction results; FN: positive samples with negative prediction results; FP: negative samples with positive prediction results; TN: negative samples with negative prediction results. The structural element confusion matrix of this model on the two data sets is as follows Figure 7 , Figure 8 The experimental results F1 values ​​are shown in Table 8 and Table 9;

[0166] Table 8 F1 values ​​of each structural element in PubMed 20k

[0167]

[0168]

[0169] Table 9 F1 values ​​of various structural elements of CSL Structed 2K

[0170]

[0171] By observing the confusion matrix and F1 value of each structural element of the PubMed 20k dataset, the F1 values ​​of Background and Objective are relatively low, 0.791 and 0.720 respectively, while the recognition performance of other structural elements Methods, Results, and Conclusions is very good, with F1 values ​​above 0.96. There is a lot of confusion between Background and Objective categories. Analysis of the reasons: In PubMed 20k, the proportion of each structural element is relatively different, the data is relatively unbalanced, and the amount of Background and Objective data is relatively small.

[0172] By observing the confusion matrix and F1 value of each structural element in the CSL Structed 2K dataset, except for the slightly lower F1 values ​​of 0.843 and 0.854 for Objective and Conclusion respectively, the F1 values ​​of other structural elements are close to or exceed 0.86, and the recognition effects of each structural element are relatively small. Analysis of the reasons: The proportion of structural elements in the CSL Structured 2K dataset is more balanced, and there is no significant difference in the number of samples between categories, which enables the model to learn the characteristics of each category more fully during training.

[0173] In order to better compare the structural element recognition effects of different models, the PubMed 20k dataset uses BilLSTM-CRF, SciBERT, HSLN, and SciBERT-HSLN as comparison models. CSL Structed2K uses BilLSTM-CRF, BERT, HSLN, and BERT-HSLN as comparison models. The experimental results are shown in Tables 10 and 11:

[0174] Table 10 PubMed 20k comparison experimental results

[0175]

[0176]

[0177] Table 11 CSL Structed 2K comparison test results

[0178]

[0179] The experimental results show that the performance of the model proposed in this embodiment is significantly better than the other three models, and it has achieved better Accuracy, Weighted F1, and Macro F1 values. In the Pubmed 20k dataset, the Accuracy and Weighted F1 values ​​have better results than CSL Structed 2K. The reason for this is that in the recognition of Methods and Resultes structural elements, the data volume is large, which can better learn context and features. Although there is no obvious gap in Weighted F1 and Macro F1 in the CSL Structured 2K dataset, the overall data volume is relatively small, and the model cannot fully learn its features on complex categories, and its performance is slightly worse than PubMed 20k.

[0180] In order to further verify the effectiveness of each component of the model, this embodiment conducted two sets of ablation experiments on two data sets. The experimental results are shown in Tables 12 and 13.

[0181] Table 12 PubMed 20k ablation experiment results

[0182]

[0183] Table 13 CSL Structed 2K ablation test results

[0184]

[0185] Ablation experiments on two datasets show that the performance of the model is significantly improved by adding the multi-head attention mechanism (MHA) module. The multi-head attention mechanism helps the model to capture long-distance dependencies and global features. On this basis, the introduction of residual connections further improves the performance of the model. Although the improvement is small, residual connections help alleviate the gradient vanishing problem in deep networks, further enhance the model's ability to recognize complex structures, and play a role in improving gradient propagation and enhancing stability in deep networks.

[0186] In order to solve the problem of lack of Chinese scientific literature structured abstract datasets, this embodiment constructs the first Chinese scientific literature structured abstract dataset CSL Structed2K by combining a large language model with manual verification and evaluation.

[0187] This embodiment proposes the BERT-MHA-RES-HLSN model for the unstructured summary text in scientific and technological literature, in the automatic identification of structural information of unstructured summary of scientific and technological literature. The model applies bidirectional long short-term memory network at word level and sentence level respectively to effectively capture the contextual information between words and sentences. In addition, a multi-head attention mechanism is introduced to capture high-order semantic features, and residual connection and normalization are combined to optimize the performance of the network in the deep level. Finally, the conditional random field is used to optimize the label sequence, so as to fully consider the order relationship between sentences and further improve the performance of deep neural networks.

[0188] In future research, the method proposed in this embodiment can be further generalized so that it can adapt to text analysis in multiple fields and effectively improve the efficiency of automatic identification of structural elements in a more complex information background.

[0189] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A method for automatically identifying text structure elements based on hierarchical neural networks, characterized in that: The method comprises: Input each sentence of the target scientific and technological literature abstract text into the word embedding layer, and encode each word in the sentence into a vector representation based on the pre-trained language model of the BERT model; wherein the target scientific and technological literature abstract includes Chinese scientific and technological literature abstract and English scientific and technological literature abstract; Generate context information and semantics between internal words in a single sentence through the bidirectional long short-term memory network in the sentence encoding layer, and generate sentence vectors through the attention pooling mechanism; Input the sentence vector into the context information enrichment layer, learn the context information of surrounding sentences through a bidirectional long short-term memory network in the enrichment layer, and generate sentence representations with high-order semantic features through a multi-head attention mechanism; Applying residual connections and layer normalization within the contextual information enrichment layer; The corresponding label scores are obtained through a feedforward neural network, and the predicted label sequence is optimized through a conditional random field. The structural elements include background, purpose, method, result and conclusion.

2. The method for automatic identification of text structure elements based on hierarchical neural network according to claim 1 is characterized in that: The word embedding layer uses two pre-trained language models, BERT-Base-Chinese and SciBERT; Among them, the BERT-Base-Chinese is used to process Chinese scientific literature abstracts, and the SciBERT is used to process English scientific literature abstracts.

3. The method for automatic identification of text structure elements based on hierarchical neural network according to claim 1, characterized in that: The sentence encoding layer generates a contextual representation of the sentence by the following steps: Use a bidirectional long short-term memory network to scan the word sequence in the sentence from forward and backward directions respectively to generate a hidden state sequence; Based on the attention pooling mechanism, the weights of words are dynamically adjusted according to context information to generate sentence-level representation vectors.

4. The method for automatic identification of text structure elements based on hierarchical neural network according to claim 1, characterized in that: The bidirectional long short-term memory network of the context information enrichment layer comprises the following steps: The sentence encoding vector sequence generated by the sentence encoding layer is used as input, and the context information between sentences is learned through a bidirectional long short-term memory network; The multi-head attention mechanism process of the contextual information enrichment layer includes the following steps: Divide each sentence vector into multiple subspaces; Perform self-attention calculations in each subspace, calculate query matrix, key matrix, and value matrix, and capture the dependencies between sentences through weighted calculations between these matrices; The attention outputs of each subspace are concatenated into the overall high-order semantic representation of the sentence.

5. The method for automatic identification of text structure elements based on hierarchical neural network according to claim 1, characterized in that: The context information enrichment layer uses residual connection to add input and output, alleviates the gradient vanishing problem, and maintains the stability of network training through layer normalization.

6. The method for automatic identification of text structure elements based on hierarchical neural network according to claim 1, characterized in that: The conditional random field performs label sequence optimization at the sentence level, specifically including: Construct a transfer matrix to describe the sequential dependencies between the labels of structural elements; Generate the optimal label sequence based on the transition probability of the sentence label and the label score predicted by the model.

7. The method for automatic identification of text structure elements based on hierarchical neural network according to claim 1, characterized in that: The method is trained using a Chinese structured abstract dataset CSL Structed 2K, which contains 2,000 scientific and technological literature abstracts, a total of 8,584 annotated sentences, covering the fields of engineering and medicine.

8. The method for automatic identification of text structure elements based on hierarchical neural network according to claim 1, characterized in that: The method performs model validation through the following steps: The performance was verified on the PubMed 20k English dataset, which contains 20,000 biomedical literature abstracts. Performance verification is conducted on the CSL Structed 2K Chinese dataset to evaluate the performance of the model in the task of identifying structural elements of Chinese summaries.

9. The method for automatic identification of text structure elements based on hierarchical neural network according to claim 1, characterized in that: The hierarchical neural network model is optimized with the following hyperparameters: Maximum text length: defines the maximum length of a sentence, which is 64 for the English dataset and 128 for the Chinese dataset; Batch size: defines the number of samples for each training, set to 64; Learning rate: Set the initial learning rate to 3e-5 and set the learning rate decay parameter to 0.9; Dropout ratio: set to 0.5 on the PubMed 20k dataset and to 0.3 on the CSL Structed 2K dataset.

10. The method for automatic identification of text structure elements based on hierarchical neural network according to claim 1, characterized in that: The method adopts a semi-automatic annotation strategy to construct the Chinese structured summary dataset CSL Structed 2K, and the annotation strategy includes: Use the preset large language model to perform preliminary automatic annotation of scientific literature abstracts to identify structural elements such as background, purpose, method, results and conclusions; Based on the preliminary annotation, the manual review and correction team verifies the annotation results generated by the large language model to ensure the accuracy of the annotation content; Remove redundant information and correct multiple annotation errors to ensure that each sentence is correctly annotated as only one structural element; The final annotation results are imported into the CSL Structed 2K dataset, and the dataset is publicly released for further research and model training.

Images