Literature reading guiding method, system and equipment and medium

Abstract

The invention provides a literature reading guiding method, system, equipment and medium, and belongs to the technical field of retrieval, and the method comprises the following steps: extracting a literature system structure of a to-be-retrieved literature, and decomposing the to-be-retrieved literature into a plurality of structure units with different granularities according to chapters; dividing the plurality of different structural units into different categories according to the subtitles; adding a prefix vector in front of each attention layer of an encoder of the GLM-6B generative language model to obtain a PT-GLM-6B model, and using the prefix vector of the PT-GLM-6B model to perform trunk content extraction on the paragraph corresponding to the subtitle of each category; a sparse matrix SM and a soft hybrid expert model Soft Moe are introduced into the BERT model, Soft Moe-SM-BERT is obtained, and identification of citation intention is carried out; according to the method, efficient literature guidance can be performed on the user according to the extracted trunk content and citation intention.

Description

Technical Field

[0001] The present invention belongs to the field of retrieval technology, and in particular relates to a document guide method, system, device and medium. Background Art

[0002] Scientific research is an important means of exploring the future and innovative development in unknown areas. It is not only the cornerstone of scientific development, but also an important driving force for human civilization and national economic and social development. Scientific and technological breakthroughs and innovations are all based on learning from, summarizing, absorbing, and transforming the experience of predecessors. Literature is the carrier of scientific inheritance and scientific ethics, the basis and basis for topic selection and decision-making, and the raw material for writing good papers. Literature introduction can quickly obtain key information and core content of literature, grasp the breadth and depth of research, avoid duplication of work, prevent intellectual property disputes, find breakthroughs for research, and improve research efficiency and quality. Existing literature reading tools are all designed to help users manage, read and organize literature more effectively, and improve the efficiency and experience of academic research. These common points reflect the basic functional requirements of literature reading tools, which can be divided into the following two categories according to their functions:

[0003] The first category focuses on document management, retrieval, and reading tools. For example, Aminer, a search and analysis platform for academic literature developed by the Institute of Computing Technology of the Chinese Academy of Sciences, offers intelligent retrieval, author information analysis, and citation analysis, helping users quickly discover relevant literature and understand academic trends. Readcube is a document management and reading software that provides features such as paper discovery, PDF organization, annotation, and sharing, helping users efficiently manage their academic literature.

[0004] The second category is systems that focus on generating document summaries and breaking down document content. These include Semantic Schola, a free academic search engine developed by the Allen Institute for AI. It provides intelligent features such as paper summaries, key concepts, and citation analysis to enhance users' understanding of document content. CORE, an open-access paper search engine developed by a university, offers intelligent document recommendations, relevance analysis, knowledge graphs, and other features to help users discover valuable academic achievements. Scholarcy, an AI-based document summary generation tool developed by Scholarcy Ltd., utilizes natural language processing technology to automatically analyze PDF files, extract key sentences and paragraphs, and generate concise paper summaries.

[0005] In summary, existing literature management and analysis systems primarily focus on document retrieval, organization, and reading assistance, and offer some automatic summary generation capabilities. However, these systems are not specifically designed for rapid understanding of literature content. They fail to extract key information frameworks from the main text, thus failing to deeply explore the underlying information within the literature, thus failing to achieve efficient reading and general understanding. Furthermore, while reading, people often consider the relevant research directions of the citations. This background information is also valuable and should not be ignored. However, existing literature navigation systems fail to tap into this information, thus failing to truly achieve rapid and efficient literature comprehension and information extraction. Summary of the Invention

[0006] In order to overcome the above-mentioned deficiencies in the prior art, the present invention provides a method for document reading, comprising the following steps:

[0007] Obtain the documents to be searched;

[0008] The Yolo++ algorithm is used to extract the document architecture of the document to be retrieved, and the document to be retrieved is decomposed into multiple structural units of different granularity according to chapters. Multiple different structural units are divided into different categories according to subtitles. A prefix vector is added before each attention layer of the encoder of the GLM-6B generative language model to obtain a prefix-tuned PT-GLM-6B model. The prefix vector of the PT-GLM-6B model is used to extract the main content of the paragraph corresponding to the subtitle of each category.

[0009] Taking the BERT model as the base model, the sparse matrix SM and the soft hybrid expert model SoftMoe are introduced into the BERT model to obtain the sparse soft hybrid expert model SoftMoe-SM-BERT. SoftMoe-SM-BERT is used to identify the citation intent of the retrieved documents.

[0010] Provide users with literature guidance based on the extracted main content and citation intention.

[0011] Preferably, the method uses the Yolo++ algorithm to extract the document system structure of the document to be retrieved, and after decomposing the document to be retrieved into multiple structural units of different granularity according to chapters, it also includes extracting the text, pictures and tables of each structural unit as leaf nodes, and constructing a hierarchical structure of the document text based on the extracted leaf nodes; classifying the subtitles of each structural unit according to the hierarchical structure, and replacing the subtitles that cannot clearly reflect the structural unit.

[0012] Preferably, the method of using Soft Moe-SM-BERT to identify the citation intention of the documents to be retrieved specifically includes the following steps: converting the input sample sequence of the documents to be retrieved into multiple token sequences, and marking the multiple token sequences; each token is mapped to a corresponding vector representation in the vocabulary;

[0013] Use BERT in SoftMoe-SM-BERT to add a type embedding to each token and generate a position embedding for each token; add the vector representation of each token to its type embedding and position embedding to obtain the final embedding representation of each token;

[0014] The final embedding representation of each token is input into Soft Moe, which captures contextual relationships and semantic representations, and dynamically weighs and integrates them among different experts. At the same time, the weight matrix of each expert is sparsely constructed to identify the citation intention.

[0015] Preferably, the token sequence includes multiple tokens, a token is the smallest unit processed by SoftMoe-SM-BERT, and a token includes words and punctuation marks.

[0016] Preferably, the Soft Moe is located between the self-attention mechanism and the feedforward neural network of the BERT encoding layer.

[0017] The present invention also provides a document guide system, comprising:

[0018] Data acquisition module, used to obtain documents to be retrieved;

[0019] The backbone content extraction module uses the Yolo++ algorithm to extract the document architecture of the document to be retrieved, decomposes the document into multiple structural units of different granularity according to chapters, and divides the multiple structural units into different categories according to subtitles. It adds a prefix vector to each attention layer of the encoder of the GLM-6B generative language model to obtain a prefix-tuned PT-GLM-6B model. The prefix vector of the PT-GLM-6B model is used to extract the backbone content of the paragraphs corresponding to the subtitles of each category.

[0020] The citation intent recognition module is used to identify the citation intent of the retrieved documents using the BERT model as the base model. The sparse matrix SM and the soft hybrid expert model Soft Moe are introduced into the BERT model to obtain the sparse soft hybrid expert model Soft Moe-SM-BERT. The Soft Moe-SM-BERT is used to identify the citation intent of the retrieved documents.

[0021] The introductory module is used to introduce the sparse matrix SM and the soft mixed expert model Soft Moe into the BERT model based on the BERT model to obtain the sparse soft mixed expert model SoftMoe-SM-BERT. The Soft Moe-SM-BERT is used to identify the citation intention of the retrieved documents.

[0022] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the literature guide method.

[0023] The literature reading method provided by the present invention has the following beneficial effects:

[0024] The present invention can extract the document system structure by using the Yolo++ algorithm and decompose the document to be retrieved into multiple structural units of different granularities according to chapters; divide multiple different structural units into different categories according to subtitles, and divide the document to be retrieved into a complete hierarchical structure; use the prefix-tuned PT-GLM-6B model to extract the main content of the paragraphs corresponding to the subtitles of each category. This process only requires fine-tuning a small number of prefix parameters to adapt to different tasks, and this method allows PT-GLM-6B to perform customized parameter adjustments for different categories of tasks, thereby better matching task requirements, which makes PT-GLM-6B have stronger cross-task generalization capabilities and can achieve excellent performance on various different categories of tasks, thereby being able to propose the main content in different subtitles; the present invention can accelerate the calculation speed by constructing a sparse matrix in the Soft Moe-SM-BERT model, and can better capture long-distance semantic dependencies by introducing Soft Moe, thereby identifying the citation intention; the present invention combines the extracted main content and citation intention of the document to be retrieved to enable users to efficiently read and understand document information. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] To more clearly illustrate the embodiments of the present invention and its design, the following briefly introduces the drawings required for this embodiment. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be derived from these drawings without inventive effort.

[0026] Figure 1 is a flow chart of the present invention;

[0027] Figure 2 This is the PT-GLM-6B architecture diagram;

[0028] Figure 3 This is the GLM-6B architecture diagram;

[0029] Figure 4 Encode 2D position information;

[0030] Figure 5 is the self-attention mask;

[0031] Figure 6 Annotated examples of prefix adjustment for encoder-decoder models;

[0032] Figure 7 This is a comparison chart of different categories of effects; Figure 7 (a) is UniLM RG-1, Figure 7 (b) is UniLM RG-2, Figure 7 (c) is UniLM RG-L; among them, Figure 7 (d) is PT-GLM-6BRG-1, Figure 7 (e) is PT-GLM-6B RG-2, Figure 7 (f) is PT-GLM-6B RG-L; Figure 7 (g) is LLaMARG-1, Figure 7 (h) is LLaMARG-2, Figure 7 (i) is LLaMA RG-L;

[0033] Figure 8 This is a comparison chart of extraction effects; among them, Figure 8 (a) is RG-1, Figure 8 (b) is RG-2, Figure 8 (c) is RG-L;

[0034] Figure 9 This is a comparative ablation experiment for different categories; Figure 9 (a) is GLM-6B RG-1, Figure 9 (b) is GLM-6BRG-2, Figure 9 (c) is GLM-6B RG-L; among them, Figure 9 (d) is PT-GLM-6B RG-1, Figure 9 (e) is PT-GLM-6BRG-2, Figure 9 (f) is PT-GLM-6BRG-L;

[0035] Figure 10 This is a CAC dataset comparison experiment diagram; among them, Figure 10 (a) is RG-1, Figure 10 (b) is RG-2, Figure 10 (c) is RG-L;

[0036] Figure 11 This is the Soft Moe-SM-BERT model structure diagram;

[0037] Figure 12 is the matrix sparsification process; where Figure 12 (a) is a schematic diagram of the structure for screening the first position. Figure 12 (b) is a schematic diagram of the structure for screening the second position. Figure 12 (c) is a schematic diagram of the structure for screening the nth position;

[0038] Figure 13 Compute the matrix for self-attention;

[0039] Figure 14 This is the Soft Moe structure diagram;

[0040] Figure 15 This is the distribution map of CCCF citation data;

[0041] Figure 16 is the experimental result of model comparison; Figure 16 (a) is the accuracy, Figure 16 (b) is the accuracy, Figure 16 (c) is the recall rate, Figure 16 (d) is the F1 value;

[0042] Figure 17 The results of the Loss comparison experiment of the citation intent classification model are given;

[0043] Figure 18 is the multi-label classification indicator iteration graph; where, Figure 18 (a) is the accuracy, Figure 18 (b) is the accuracy, Figure 18 (c) is the recall rate, Figure 18 (d) is the F1 value;

[0044] Figure 19 is the Ren-CECps data distribution;

[0045] Figure 20 The comparative experimental results of the Ren-CECps dataset are shown in Figure 2. Figure 20 (a) is the accuracy, Figure 20 (b) is Micro F1, Figure 20 (c) is Macro F1;

[0046] Figure 21 is the ablation experiment result on CCCF citation dataset; Figure 21 (a) is the accuracy, Figure 21 (b) is the accuracy, Figure 21(c) is the recall rate, Figure 21 (d) is the F1 value;

[0047] Figure 22 Iteration diagram of ablation experiment;

[0048] Figure 23 is the result of sparsity comparison; Figure 23 (a) is the accuracy, Figure 23 (b) is the accuracy, Figure 23 (c) is the recall rate, Figure 23 (d) is the F1 value. DETAILED DESCRIPTION

[0049] In order to enable those skilled in the art to better understand the technical solution of the present invention and to be able to implement it, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention and are not intended to limit the scope of protection of the present invention.

[0050] Example

[0051] The present invention provides a method for guiding reading of documents. Figure 1 As shown, the following steps are included:

[0052] Step 1: Obtain the literature to be searched.

[0053] Step 2: Use the document architecture extraction Yolo++ algorithm to decompose the document to be retrieved into multiple structural units of different granularity according to chapters; divide multiple different structural units into different categories according to subtitles; add a prefix vector before each attention layer of the encoder of the GLM-6B generative language model to obtain the PT-GLM-6B model, and use the prefix-tuned PT-GLM-6B model to extract the main content of the paragraphs corresponding to the subtitles of each category.

[0054] The present invention mainly introduces a method for extracting the main structure of an article based on extracting subheadings and their corresponding text content. Compared with the method of directly extracting abstracts, this method can better retain the main content and semantic structure of the original text. Specifically, the method includes two steps: first, the Yolo++ algorithm (Document Hierarchy Extraction) is used to extract the original document system architecture, and then the subheadings are classified to facilitate the subsequent focused extraction of the main structure, and useless subheadings are replaced with other subtopics to fit the document theme; then, from the paragraph content under each subheading, the sentences that best summarize the paragraph information are extracted. These sentences can retain the original semantic information to the greatest extent and are displayed as the main structure of the article. The present invention uses a general language model (PT-GLM-6B) optimized by prefix tuning (PT) to implement the above-mentioned main structure extraction task.

[0055] Identify visual elements such as pictures and tables in the text and extract them as leaf nodes. For the main text paragraphs of plain text, they will also be regarded as leaf nodes. Through the division of this structural unit and the extraction of leaf nodes, the algorithm constructs a complete hierarchical structure of the document content. This representation method based on the document architecture can avoid directly processing the entire lengthy article sequence, but instead uses more fine-grained paragraph-level semantic information, which helps to improve the performance of trunk extraction. When the main content of the document is finally displayed, the hierarchical structure of the original text can also be clearly presented, making it convenient for readers to quickly grasp the main content of the article. When extracting the architecture of the document, it should be noted that not all titles need to be extracted. The present invention only extracts important titles that are obviously related to the document theme as nodes, and excludes some titles with weak relevance such as related work, introduction, summary, method, etc. This is because these titles cannot truly reflect the core information of the document. In order to solve this problem, the present invention classifies titles based on rules and sets differentiated extraction tendencies according to titles of different categories. Specifically, this method categorizes the introduction as background information and extracts sentences describing the existing problem and the article's solution. These sentences are then used as nodes to replace the original introduction title. This not only avoids the appearance of irrelevant nodes, making the final document hierarchy more consistent with the document content, but also simplifies the task of extracting the main text, allowing for targeted extraction of key information from titles of different categories.

[0056] In the autoregressive model, the input text is divided into fixed-length sequences for training. However, for document-level long text tasks, using fixed-length input sequences will make it difficult to capture the overall semantic structure and contextual information of the text, which may affect the performance of the model in long text generation tasks. Therefore, it is necessary to divide the document into different parts according to subtitles, retaining the complete paragraphs of each part to shorten the text length. After dividing the document structure, the question categories can also be divided according to the subtitle categories, and then combined with the GLM-6B generative language model for paragraph trunk extraction. It uses the autoregressive blank filling method for pre-training, and by introducing improvements to two-dimensional position encoding, it better captures the dependencies between different positions in the text. The PT-GLM-6B architecture is as follows: Figure 2 shown.

[0057] PT-GLM-6B can also better model the contextual relationships in long texts of subtitle paragraphs by allowing predictions in arbitrary orders, thereby improving the accuracy and coherence of generation. Ordinary fine-tuning requires modifying and storing all parameters of the entire pre-trained language model, which has high storage costs and a large memory burden. The fine-tuning process will also change the overall parameter distribution of the language model, destroying the original knowledge and making it easy to overfit to specific tasks. In response to the problems of high fine-tuning costs and easy overfitting, the present invention introduces prefix tuning to solve this problem. Compared with traditional fine-tuning methods, prefix tuning only needs to optimize a small number of prefix parameters, which greatly reduces storage and computational overhead. In addition, prefix tuning performs better on examples extrapolated to different topics. In the question-answering task of the present invention, the good generalization of prefix tuning is relied upon for different questions.

[0058] Regarding the problem of extracting the main body of the paragraph under the subheading, considering that the text main body extraction has the characteristics of both natural language understanding (NLU) and natural language generation (NLG), it can be said to be a typical NLP comprehensive task. It is necessary to understand the semantic information of the original text and generate the core original sentence of the article. This invention uses the Transformer-based pre-training model GLM to solve this problem, because this type of model has the ability of semantic understanding and text generation at the same time. The architecture is as follows Figure 3 As shown in the figure, it can be seen that the model is mainly composed of two layers of loops in the inference phase: The first layer of loop: This loop continues and generates a new token at each iteration. The loop termination condition is that the model generates a specific end token. <eos>Second loop: In this fixed number of loops, GLMBlock runs sequentially. In each loop, the most likely token ID is determined based on the attention scores and used to generate the next token.

[0059] (1) Pre-training objectives.

[0060] In the GLM pre-training framework, autoregressive blank filling is used as the pre-training objective. The blank filling task is implemented using the following methods: input text structure; 2D position encoding; and the model's attention mask matrix.

[0061] Input Text Composition: The goal of autoregressive blank filling is to randomly select consecutive text segments in the input text and replace them with special "[MASK]" tokens to form a corrupted text. The model needs to predict these replaced text segments in an autoregressive manner based on the corrupted text sequence. When predicting each text segment, the model has access to the corrupted text and previously predicted segments to fully capture the interdependencies between different segments.

[0062] 2D position encoding: such as Figure 4 Position1 and Position2 shown are representations for two-dimensional encoding of fragments in the input text. The first is used to record the relative order in PartA, and the second is used to record the relative order in the masked text fragment PartB. All tokens in PartA have a relative position encoding of 0, indicating that they do not belong to any masked span. The tokens in PartB have relative position encodings from 1 to the span length, indicating their relative positions within the masked span. The advantage of this 2D position encoding method is that for the tokens in PartB, they do not need to know the length of the masked span, and the model can directly perform autoregressive generation based on the relative position information without manually setting the number of [MASK] like BERT. 2D encoding can better capture the position information of the token in the entire sequence and local span, which is very helpful for autoregressive generation.

[0063] The model's attention mask matrix: Based on the first [MASK], x2 and x3 are decoded sequentially, and based on the second [MASK], x6 is decoded sequentially. Here, x2, x3, and x6 are text blocks, and 2, 3, and 6 are serial numbers. So how do we decode variable-length sequence codes from [MASK]? The GLM model cleverly uses the special [START] and markers to handle variable-length masked text segments. The model autoregressively generates masked text from [START] until the marker is generated, adapting to masked sequences of arbitrary length. Furthermore, GLM uses a custom attention mask to unify the bidirectional encoder and unidirectional decoder.

[0064] like Figure 5 As shown, a custom self-attention mask is required to achieve the following: words in Part A of the bidirectional encoder are mutually visible, as indicated by the green box in the figure; words in Part B of the unidirectional decoder are unidirectionally visible, as indicated by the yellow box in the figure; Part B is visible to Part A, while the rest is invisible, as indicated by the gray box in the figure. In the task of generating a set of sentences from text, the input text is divided into two parts: Part A and Part B. Part A constitutes the input to the model, with a mask added at the end. This allows Part A to be visible to each other, similar to the encoder (BERT), which focuses on information extraction. The model generates the text content of Part B through autoregression, with Part B only seeing what has come before it, similar to the decoder (GPT), which focuses on generation. The overall effect is conditional generation using the extracted information as a condition. This achieves understanding of the original text in the dataset and generation of the sentence set based on this understanding. In unconditional generation, the model only needs to be given an initial context to generate subsequent text content. For example, if the input is "represents Google," the mask mark is added to the end of the mapping, and the next sentence can be generated through the GLM autoregressive model. In conditional generation tasks, the model can be fine-tuned to generate corresponding text based on specific conditions, such as generating answers to given questions or generating the main body of an article based on a specific context.

[0065] (2) Model fine-tuning.

[0066] Models that don't include prompts have several key drawbacks during training and application, including a lack of targeted guidance and an inability to provide clear generation goals. The generated content lacks overall purpose and logic, making it difficult to achieve human-level coherence. Each task requires a fully fine-tuned model replica, which is not conducive to multitasking and also increases deployment and storage costs. This paper proposes a lightweight fine-tuning method called prefix-tuning. Prefix-tuning introduces trainable prefix vectors to provide clear task guidance for the language model, making the generated content more cohesive while also achieving good modularity and space efficiency, thus overcoming the limitations of traditional fine-tuning methods. Prefix-tuning differs from prompt-tuning in the location of prompt injection. Prefix-tuning injects prompts into the input of each attention layer, while prompt-tuning only injects prompts at the word embedding layer. Therefore, prefix-tuning allows for more fine-grained control of model behavior because it provides prompts for each attention layer, better capturing task-related information and providing stronger expressive power. Therefore, the present invention selects prefix-tuning for fine-tuning optimization and improvement.

[0067] Differentiated prefix tuning: For inputs containing sub-labels and paragraphs of different categories, the prefix vector can guide the language model's extraction focus to tend to the extraction target of the category to which the paragraph belongs, making the extraction process targeted. For example, for background paragraphs, it is necessary to extract the problems existing in the existing methods, while for outlook paragraphs, this extraction mode is not applicable, so it is necessary to perform differentiated extraction on paragraphs of different categories. The present invention targets the task of extracting the main trunk of paragraphs under subtitles, and divides them into five categories according to the nature of the subtitles. At the same time, five types of prefix fine-tuning are used, and the selection of the type of pre-tuning is used before the extraction task to ensure that the extraction tendency is applicable to the subtitle. In addition to selecting the type of pre-tuning, each prefix tuning is inserted in the same way, such as Figure 6 shown.

[0068] This prefix is ​​inserted into the input of the model and interacts with the original input through a continuous self-attention mechanism, thereby affecting the final generated output. Compared with directly fine-tuning the pre-trained model, fine-tuning only the prefix can significantly reduce the number of parameters while achieving similar or even better performance. In the Transformer encoder of GLM, a learnable prefix vector is inserted in front of the original input. This prefix vector can interact with the original input through a self-attention mechanism. Examples of introducing prefix tuning in the encoder-decoder are as follows: Figure 6 As shown, the left figure focuses on the encoder's prefix adjustment and input processing, and the right figure focuses on the decoder's prefix update and output generation, together reflecting the model's staged optimization process.

[0069] The principle of prefix tuning is that providing appropriate contextual information in prefix tuning can guide the language model to complete the task without modifying its parameters. For example, if you want the language model to generate a word (such as "Yao Ming"), you can add common collocations of the word (such as "basketball star") in front of it as context. In this way, the language model will be more inclined to generate the desired word. Extending this to beyond generating single words or sentences, it is hoped to find a context to guide the language model to solve natural language generation tasks. Intuitively, this context can affect the encoding of x by guiding what information to extract from the input x, and can also affect the generation of the output y by guiding the distribution of the next generated token. Natural language task instructions (such as "summarize this table in one sentence") may guide experts to complete the task, but they will fail for most pre-trained language models.

[0070] Optimizing instructions to continuous word embeddings propagates their effects across all activation layers of the GLM and influences subsequent token generation. This is more expressive than using discrete prompts, which need to match the embeddings of real words. This approach is also simpler than directly modifying all activation layers, avoiding long-range dependencies and reducing the number of adjustable parameters. Therefore, prefix tuning optimizes the parameters of the entire prefix.

[0071] The PT-GLM-6B algorithm in our invention extracts content from each paragraph in the original title based on prescribed rules. This process includes extracting background, technical framework, outlook, juxtaposition, and other key content, enabling a deeper understanding of the core research information. A total of 4,096 pieces of data were successfully extracted and aligned. The data was divided into training, test, and validation sets with a total size of 3,276, 409, and 409, respectively.

[0072] As shown in Table 1, the GPT model successfully extracted original sentences with negative sentiment from the background section, as well as original sentences describing the key aspects of the solution. This helps fully understand the research motivation and key elements of the solution. Regarding the technical framework, original sentences describing the most important technological innovations and implementation methods were extracted, along with sentences identifying existing technical shortcomings and addressing these shortcomings. Regarding the outlook, original sentences proposing the future of the technology were extracted. This is crucial for understanding the long-term impact and potential development directions of the research. Furthermore, original sentences containing parallel words were extracted to capture the parallel relationships between various aspects of the research. For other key information, GPT was used to extract original sentences from the original text regarding the original methods and techniques, which are unique highlights and innovations of the research. Using GPT-assisted content extraction, the present invention successfully delved deeply into each segment of the research title and extracted key information. This information not only helped the present invention understand the core of the research but also provided rich material for further analysis and research. GPT's intelligent analysis provides an efficient and accurate tool for information extraction, enhancing the depth and comprehensiveness of the entire research.

[0073] Table 1 Title classification settings

[0074]

[0075] Evaluation metrics: RG-1, RG-2, and RG-L are different variants of the ROUGE evaluation metric used to evaluate the performance of automatically extracted sentences. They measure the overlap between the extracted sentences and the reference sentences.

[0076] ROUGE-1: Measures the overlap between the words (unigrams) in the extracted sentences and the words in the reference sentence set.

[0077] ROUGE-2: Measures the overlap between two consecutive words (bigrams) in the extracted sentence and two consecutive words in the reference sentence set.

[0078] ROUGE-L: measures the length of the longest common subsequence between the extracted sentence and the reference sentence, where the subsequence can be discontinuous, indicating their semantic similarity.

[0079] These ROUGE metrics all range in value from 0 to 1, where 1 indicates that the extracted sentence and the reference sentence are completely identical in terms of the corresponding overlap ratio. These metrics are often used in automatic evaluation to measure the performance of sentence set generation systems. Using multiple ROUGE metrics simultaneously can provide a more comprehensive assessment of different aspects of the overlap between the extracted and reference sentences.

[0080] Experimental environment: The specific hardware and software used in the experiment are shown in Table 2.

[0081] Table 2 Hardware and software information

[0082] software and hardware property operating system Windows 11 CPU model Intel(R)Core(TM)i5-11700@3.00GHz GPU Model GeForce RTX 3060 12G Memory 16G Python 3.8.10

[0083] Experimental plan: The effectiveness of the PT-GLM-6B model was verified through experiments. These experiments aimed to answer the following questions:

[0084] Experiment 1: Model comparison experiment, compare with other large models and explore the optimization effect.

[0085] Experiment 2: An ablation experiment is performed to investigate the effect of the Prefix-Tuning module on the overall model.

[0086] Experiment 3: The impact of different prefix lengths on the model.

[0087] Experiment 4: Comparison of the effectiveness of the model on the Chinese public dataset LCSTS.

[0088] Experiment 5: Analyze the prediction effects of different models through case study.

[0089] To verify the effectiveness of the proposed PT-GLM-6B model, we compared it with the classic large model UniLM and the latest LLaMA model. UniLM (Unified Language Model) is a large language model based on the Transformer architecture and using a bidirectional Transformer encoder-decoder structure. LLaMA is a large language model constructed based on the Transformer architecture and combined with improvements such as pre-normalization, the Swiglu activation function, and rotation position embedding.

[0090] Experiment 1: Model comparison experiment.

[0091] like Figure 7 As shown, Figure 7 This is a comparison chart of different categories of effects; Figure 7 (a) is UniLM RG-1, Figure 7 (b) is UniLM RG-2, Figure 7 (c) is UniLM RG-L; among them, Figure 7 (d) is PT-GLM-6B RG-1, Figure 7 (e) is PT-GLM-6B RG-2, Figure 7 (f) is PT-GLM-6BRG-L; Figure 7 (g) is LLaMARG-1, Figure 7 (h) is LLaMARG-2, Figure 7 (i) is LLaMARG-L. From a category analysis, the background category shows relatively high values ​​across all categories. This is due to its relatively clear extraction method, which extracts relevant negative impact words as deficiencies and related solutions. This category contains less complex information than the semantically complex technical paragraphs, and the main stems that need to be extracted often have corresponding results. Compared to other categories, the requirements are relatively clear, resulting in higher values ​​for this category in RG-1, RG-2, and RG-L. From a trend analysis, UniLM and LLaMA personality indicators show similar structures, indicating that the two models do not clearly differentiate across different data categories. In contrast, PT-GLM-6B has unique effects across various categories, with significant differences between indicators. This indicates that pre-tuning significantly influences question tendencies across different subheading categories, confirming the effectiveness of various pre-tuning methods. Overall, LLaMA and PT-GLM-6B both outperform UniLM in RG-1, with similar differences between the two. In RG-2, LLaMA outperforms PT-GLM-6B, with UniLM having the lowest values. PT-GLM-6B outperforms LLaMA on RG-L, indicating that this targeted fine-tuning method is more conducive to the backbone extraction task. In addition, the GLM model has both semantic understanding and text generation, making it more suitable for the tasks in this section.

[0092] like Figure 8 As shown, Figure 8 This is a comparison chart of extraction effects; among them, Figure 8 (a) is RG-1, Figure 8 (b) is RG-2, Figure 8 (c) is RG-L. An overall analysis of the mean values ​​of various indicators reveals that UniLM performs worst in RG-1, RG-2, and RG-L. Compared to PT-GLM-6B, PT-GLM uses 6 billion parameters, while UniLM does not. Due to its larger number of parameters, PT-GLM is better able to capture key information in the article stem extraction task, outperforming UniLM. UniLM requires different attention masks to switch between different pre-training objectives, while GLM uses a single autoregressive Transformer network. GLM uses an encoder and decoder to handle different pre-training objectives, combining natural language understanding (NLU) and generation (NLG) tasks. This model structure makes GLM better suited to article sentence extraction tasks, resulting in superior performance compared to UniLM. The LLaMA model is slightly inferior to the PT-GLM-6B model on RG-1 and RG-L. The LLaMA model is based on a collection of basic language models. In comparison, the autoregressive pre-training objective of PT-GLM-6B can better capture semantic and contextual information and effectively share parameters between different tasks. However, the LLaMA model cannot fully utilize parameter sharing between different tasks. Therefore, the repetition rate of single words in real results and the repetition rate of the longest common subsequence are higher than the LLaMA model. On RG-2, it is slightly higher than the PT-GLM-6B model. The LLaMA model has more parameters than the PT-GLM-6B model, so it has some advantages in the repetition rate of two consecutive words. However, the semantic connection between individual words is weak, which only shows that the large model parameters are numerically stacked to achieve the model's excellent effect.

[0093] Experiment 2: Ablation experiment.

[0094] like Figure 9 As shown, Figure 9 This is a comparative ablation experiment for different categories; Figure 9 (a) is GLM-6BRG-1, Figure 9 (b) is GLM-6B RG-2, Figure 9 (c) is GLM-6B RG-L; among them, Figure 9 (d) is PT-GLM-6B RG-1, Figure 9 (e) is PT-GLM-6B RG-2, Figure 9 (f) is PT-GLM-6B RG-L. It can be seen that the background class has a relatively good extraction effect, which is related to its accurate extraction method, that is, extracting negative words as the shortcomings of the literature, so that the model can clearly understand the intention. The technical framework class has a lower effect than other indicators. Analysis shows that the text under this type of sub-titles has a high proportion of professional terms and relatively complex semantic information, resulting in its effect being inferior to other categories. Judging from the trend of the bar chart, the trend of GLM-6B is relatively flat, indicating that the extraction results for different categories are not targeted. On the other hand, PT-GLM-6B has different effects on the extraction of corresponding categories by introducing prefix tuning for different categories, and the effect is mostly better than GLM-6B without prefix tuning, which further shows that the introduction of prefix tuning is conducive to the extraction of sub-titles of different categories.

[0095] As shown in Table 3 , it can be seen that GLM-6B is slightly worse than PT-GLM-6B after taking the average of each indicator.

[0096] Table 3 GLM ablation experiment

[0097]

[0098] The advantage of prefix tuning lies in the fact that PT-GLM-6B employs a prefix tuning method that only requires fine-tuning a small number of prefix parameters to adapt to different tasks. This method also allows PT-GLM-6B to customize parameter adjustments for different categories of tasks, thereby better matching task requirements. This gives PT-GLM-6B stronger cross-task generalization capabilities, enabling it to achieve outstanding performance across a wide range of tasks. Regarding the backbone extraction task, PT-GLM-6B optimizes consecutive prefixes, helping the GLM better understand the input text and extract a more accurate backbone. This tuning process can be performed by optimizing the embedding vectors of consecutive prompts to better represent their task-related features. This allows the GLM to extract a more accurate and relevant set of sentences based on these optimized prompts.

[0099] Experiment 3: Comparison experiment of different prefix lengths.

[0100] To verify the impact of prefix length on task performance, the average RG value of each category is used as the result to clearly demonstrate the comparison of prefix effects. As shown in Table 4, it can be seen that a prefix length of 110 is the best for this task, while the effect slowly decreases when it reaches 128. Although longer prefixes mean more trainable parameters and are therefore more expressive, as the prefix length increases to the threshold of 110, performance improves and then decreases slightly. Analyzing the reasons, longer prefixes can provide richer contextual information, thereby better guiding the model to generate relevant outputs. Too short prefixes may not fully convey the task requirements. However, if the prefix is ​​too long, it is too flexible compared to the entire model parameters, may be too dependent on the training data, and may not be well generalized to other categories.

[0101] Table 4 Prefix length experiment

[0102] PrefixLength RG-1(%) RG-2 (%) RG-L (%) 64 30.2 12.2 28.1 110 37.9 17.7 35.8 128 35.4 15.6 34.2

[0103] Experiment 4: Comparative experiment on CAC dataset.

[0104] Most existing Chinese datasets are generative summarization datasets, so this experiment uses the Chinese Abstractive Corpus (CAC), a dataset of 24,526 historical articles from mainstream vertical media in the education and training industry. This corpus is primarily designed for training abstraction models. Each piece of data has two fields: summary and text, and both fields are annotated by the author.

[0105] This dataset is similar to the dataset of the present invention and has manually corrected data. In order to make the model more effective on public datasets, prefix tuning is directly introduced on this dataset for comparison without classification.

[0106] like Figure 10 As shown, Figure 10 This is a CAC dataset comparison experiment diagram; among them, Figure 10 (a) is RG-1, Figure 10 (b) is RG-2, Figure 10 (c) is RG-L. It is clear that PT-GLM-6B outperforms the other two models in RG-1 and RG-L, indicating that PT-GLM-6B has higher single-word overlap and semantic and syntactic similarity. Prefixes effectively capture task-related information, allowing rapid adaptation to specific tasks without compromising pre-training capabilities. GLM's bidirectional attention mechanism enables the model to more comprehensively understand the text context, thereby extracting higher-quality sentences. This combination enables GLM-6B to achieve the best performance in both word overlap (RG-1) and semantic similarity (RG-L). The LLaMA model, on the other hand, has a higher phrase overlap rate, attributed to its pre-normalization design, which normalizes the input of each transformer sublayer rather than the output. This mechanism better learns the semantic structure of the input text, thereby better capturing key phrases when generating sentence sets. This is the most distinctive aspect of GLM-6B's optimization approach.

[0107] Experiment 5: Case analysis and comparison experiment, analysis of typical cases.

[0108] Paragraph: Traditional dynamic random access memory has been used as the main memory of computer systems for decades. However, due to the limited scalability and high refresh energy consumption of DRAM, it is difficult to meet the needs of future computer systems. Therefore, people have developed new types of non-volatile memory, such as phase change memory (PCM) and resistive random access memory (ReRAM). Due to their advantages such as high scalability and low energy consumption, they have been proposed as the storage medium of the next generation of main memory to replace or supplement DRAM. The non-volatility of NVM also allows data to be stored persistently in main memory to achieve instantaneous system failure recovery. The byte-addressable nature of NVM and the access latency close to that of DRAM allow NVM to be directly connected to the memory bus and accessed through the CPU's read and write instructions, avoiding the overhead of traditional block-based interfaces.

[0109] In order to ensure the random validity of the experiment, random sampling was performed from all data sets. As can be seen from Table 5, in the case, it can be seen that the UniLM extraction result omitted the negative sentiment sentence that needed to be extracted in the extraction result because there was no prefix tuning. This sentence shows the shortcomings of the background paragraph, and the other parts that need to be extracted are not the sentences that should be extracted; in the LLaMA model, its extraction result is close to the correct result, indicating that the fine-tuning effect is effective, which is also due to the powerful understanding ability of the model. However, the sentences extracted from the negative sentiment sentences are incomplete, and other sentences with negative emotions are omitted, and the sentences describing the specific content of the solution are missing. This is due to the fact that no prefix tuning was introduced in the fine-tuning.

[0110] Table 5 Case study comparison experiment table

[0111]

[0112] The extraction tendency of the corresponding category paragraphs was lost in the first stage, and the effect of changing the extraction tendency by manual prompts alone was not ideal; GLM-6B had a better extraction effect on negative emotional sentences, and the extracted content was more complete than LLaMA, but the solution extracted the sentence incorrectly, and the extraction tendency was unclear due to the lack of prefix tuning, indicating that the extraction tendency was still restricted by manual prompts; in PT-GLM-6B, it can be seen from the results that the model extracted relatively complete content and structure, which proved the effectiveness of prefix tuning for extraction tendency, and the extraction effect was slightly improved compared with the original model, which improved the extraction results.

[0113] The content of this invention mainly introduces the article trunk extraction task and related models and methods. The traditional trunk extraction method mainly uses an extractive sentence set, but it is not applicable to the task of this invention. Therefore, a trunk Yolo++ extraction method based on subtitle structure and paragraph is proposed, and a prefix-tuned PT-GLM-6B model is proposed to solve the trunk extraction task. In the PT-GLM-6B model, autoregressive blank filling is used as the pre-training target, and the model architecture and fine-tuning process are introduced. In the experiment and result analysis part, the effect of the model is compared and analyzed by experiments such as ablation experiments. The results show that the PT-GLM-6B model performs well in the key original sentence extraction task, and the effect is even better after prefix tuning.

[0114] Step 3: Taking the BERT model as the base model, introduce the sparse matrix SM and the soft mixed expert model Soft Moe into the BERT model to obtain the sparse soft mixed expert model Soft Moe-SM-BERT, and use SoftMoe-SM-BERT to identify the citation intent of citations.

[0115] In the task of identifying citation intention, it is difficult to establish a unified standard and feature system because the citation intention is often hidden in the deep layer. The present invention uses the Multi-Label Text Classification (MLTC) method to solve the problem of citation intention and establish a clear label range. The present invention proposes the Soft Moe-SM-BERT model to identify the citation intention of the extracted citation data. In the face of the problem that increasing the training time will lead to a decrease in the ability to understand the context, the training speed is optimized by adding sparsification based on BERT to accelerate the convergence speed; and then the token is weighted averaged by introducing SoftMoe to better capture long-distance semantic dependencies. Then, experiments and control experiments on multi-label classification of citation intention are carried out, and the performance is compared with other classic and latest models to prove the performance superiority of the model.

[0116] The citation intention recognition task is specifically defined as: there is a sample space X∈R d , d is the word vector dimension, R is the real number field, there are n sample sequences, let the sequence set be y i ∈Y,x i is a sentence with a quotation mark, N is the total number of sentences with a quotation mark, and citation involves multiple intentions or topics. The intention is modeled as multiple labels, and the citation intention recognition task is transformed into a multi-label classification task. Let the label set be T = {t1, t2, ..., t h }, h represents the total number of labels, assuming that the label space contains m label samples: S = {(s1, s2, ..., s m )},s i ∈ T. The meaning of the set is shown in Table 6.

[0117] Table 6 Collection Description

[0118] gather describe X A SequenceSet is a collection of text sequences. T LabelSet refers to the set of labels that are classified in a given problem or task. S Label Space refers to the abstract space used to describe possible labels in a problem. Y ResultSet, the correct label set corresponding to X

[0119] The goal of this task is to train a prediction model. During the training process, s i Converge to the true label y i To achieve the prediction effect. The model corresponds to the function f(x, y) can be expressed as the confidence that y∈Y is the correct prediction of x, which can be converted into a ranking rank(x, y), where x represents a sentence with a reference tag and y represents the intent label. The higher the ranking, the higher the score.

[0120] This paper primarily addresses the multi-label text classification task. The data is long and semantically rich. Considering its contextual understanding and semantic learning capabilities, the paper uses BERT as the underlying model. However, the BERT model requires significant computing resources and training time when processing semantically rich long text data. Furthermore, overly long texts may contain noise or irrelevant information, making it impossible to capture all contextual information, thereby reducing the BERT model's ability to understand the text.

[0121] To this end, the present invention innovatively proposes the SoftMoe-SM-BERT model, the architecture of which is as follows: Figure 11 As shown, in order to solve the problem of low training efficiency, a technology based on sparse matrix (SM) is used to utilize the sparse nature of the neural network activation pattern to improve the computational efficiency and speed of training without losing or even improving the model performance. In order to solve the problem of contextual semantic understanding, the present invention proposes to introduce a fully trainable hybrid expert model, namely a soft hybrid expert model. In SoftMoe (Soft Mixture of Experts, Soft Moe), the weighted average of all tags is calculated, and these weighted combinations are passed to each expert. This soft assignment method allows each expert to process a subset of tags based on the relevance or importance of the context. By considering multiple weighted combinations of tags, Soft Moe can capture richer contextual information, thereby better modeling the citation context relationship.

[0122] Figure 11 This section shows the basic architecture of the Soft Moe-SM-BERT model. The architecture consists of three modules: the vector input conversion module, the SM-BERT module, and the SM-Soft Moe module.

[0123] The vector input module takes a sample sequence as input, such as the sentence sample "represents Google" from the CCF paper dataset. The sentence is converted into a token sequence. The sentence is broken down into tokens, the smallest unit processed by Soft Moe-SM-BERT. These tokens can be words, punctuation marks, and so on, and each token corresponds to an index in the vocabulary. After token segmentation, a special [CLS] marker is added to the beginning of the sentence to indicate the start of the classification task. A special [SEP] marker is added to the end of the sentence to indicate its end.

[0124] According to the content of the quoted sentence, each token is mapped to the corresponding vector representation in the vocabulary, namely word embeddings. For sentence pair tasks (such as text classification, sentence relationship judgment, etc.), BERT will add a type embedding (segment embedding) to each token to distinguish the differences between sentences. In order to encode the position information of each token, BERT will generate a position embedding (position embedding) for each token. The dimension of the position embedding is the same as that of the word embedding, but each position corresponds to a different position encoding. The position encoding of each word is calculated according to the dimension of the embedding vector using formulas (1) and (2). Where pos represents the position of the word; i represents the dimension index of the word embedding vector; d model is the dimension of the word embedding vector; PE(pos, 2i) represents the position encoding of even positions; PE(pos, 2i+1) represents the position encoding of odd positions.

[0125]

[0126] For BERT, long text will lead to increased computation and storage requirements for all weight matrices in the BERT model, increasing memory pressure and computing resource consumption, affecting training efficiency and model performance. Sparse matrices only store non-zero elements and their corresponding indexes, so they can greatly reduce storage space. In addition, since most elements in a sparse matrix are zero, the sparse property can be used to speed up calculations in calculations such as matrix multiplication, thereby reducing computational costs and better capturing semantics. This invention applies sparse matrices to BERT. Sparse process such as Figure 12 As shown, Figure 12 (a) is a schematic diagram of the structure for screening the first position. Figure 12 (b) is a schematic diagram of the structure for screening the second position. Figure 12 (c) is a schematic diagram of the structure for screening the nth position.

[0127] The specific transformation process is as follows: there is a neural network whose input layer X has N neurons, H is a hidden layer with M neurons, and Y is an output layer with K neurons. Then formula (3) represents the feedforward neural network process. Here, only the gradient of the hidden layer is derived, h j is the calculation result of the jth neuron, x i is the neuron in the input layer X, w ij is a weight matrix, where i∈N, i={1,...,n}, j∈M, j={1,...,m}, and formula (3) is as follows:

[0128]

[0129] Back propagation is to update the weight matrix, and calculate the partial derivative of w by calculating the loss function E, using Δw ij Represents the position weight, and the weight matrix derivative formula (4) is obtained after calculation:

[0130] Δw ij =ηδ j ·x i (4);

[0131] Among them, η is the learning rate, the error term δ j is the partial derivative of the loss function with respect to the activation value of the neuron, as shown in formula (5):

[0132]

[0133] Among them, δ q is the error term of the output layer, x j is the output of neuron j, u j is the weighted output of neuron j, It is represented by the activation function f on its input u j If the input matrix X or the error matrix δ is sparse, the matrix multiplication calculation will be greatly reduced, thus saving the amount of calculation. So in formula (:3), x i With w ij The calculation result of the hidden layer of the feedforward neural network is obtained by multiplication, so at this time w ij Changing to a sparse matrix will greatly reduce Δw ij The calculation of makes the gradient drop quickly.

[0134] In the hidden layer, although many neurons will not be exactly equal to 0 or 1, they are still very close to 0 or 1. If the difference between the two quantities is less than a predefined threshold γ h , then they are very close. So here the present invention sets T h is the subset of activated neurons in all hidden layers, as shown in formula (6).

[0135] T h ={j|1-y cj >γ h and y cj >γ h } (6);

[0136]

[0137] Among them, y cl is the weight of the neuron.

[0138] From the updated formula (7), only the h Only neurons that participate in the calculation will be recorded as zero.

[0139] So we can get a matrix in the update expert with sparsity that meets the threshold condition. Then according to the zero update rule, the original weight matrix w ij Become a sparse matrix. Thus a sparse matrix is ​​realized.

[0140] For each token, its word embedding, position embedding and token type embedding are added together to get the final representation of the token. The token is fed into the encoder and the attention mechanism is used to capture the dependencies and importance between different parts of the input sequence. The matrix T composed of tokens is added to the weight matrix W. Q 、W K 、W V Multiply, where And convert the weight matrix into a sparse matrix. The conversion effect is as follows Figure 13 As shown in Figure 2, after obtaining the corresponding query matrix Q, key matrix K, and value matrix V, we use these Q, K, and V matrices to calculate the attention score and use them to perform weighted summation on the value matrix V to obtain the final attention weighted vector representation. Let the target word be "Google" x i Calculate the similarity between the query and each key, at the i-th position in the sentence, then its attention score at the j-th position is s ij , then his overall score is shown in formula (8), and formula (9) reveals that the attention total is multiplied by the value representation V, and then the similarity score is converted into a probability distribution through the softmax function, where Is the dimension of the key vector, which is used to scale the attention weight. As shown in formula (10), multi-head attention is an extension of the attention mechanism. The outputs of each attention head are spliced ​​together to obtain the weighted sum of the values ​​to obtain the representation of the current query, where

[0141] S=QK T (8);

[0142]

[0143] MH(Q,K,V)=Concat(h1,h2,...,h n ) (10);

[0144] Among them, W i Q 、W i K 、W i V They are query vector, key vector, and value vector respectively.

[0145] According to the given attention matrix, the value of each position represents the degree of attention or importance of the model to each position in the input sequence. As shown in Table 7, the probability score matrix is ​​obtained and connected to the two-layer feedforward neural network and recorded as x MH , then connect it through the Add&Layer Norm module, that is, residual connection, layer normalization.

[0146] Table 7 Example of probability score matrix of self-attention

[0147] generation surface yes valley Song generation 0.5 0.3 0.1 0.05 0.05 surface 0.3 0.4 0.1 0.14 0.16 yes 0.1 0.1 0.5 0.12 0.1 valley 0.05 0.14 0.12 0.4 0.29 Song 0.05 0.06 0.1 0.29 0.5

[0148] As shown in the normalization formula (11), μ is the mean of the input, σ is the standard deviation of the input, γ and β are learnable parameters, and ∈ is a very small number to prevent the denominator from being zero. Such 12 Encoder layers constitute the complete BERT structure.

[0149]

[0150] The final loss function is as follows:

[0151]

[0152] Where n is the number of samples, Indicates that when the true label y i The loss when y is 1, which penalizes the model for predicting a low probability on the positive class (y=1). Indicates that when the true label (y i ) is 0, which penalizes the model for predicting a higher probability on the negative class (y=0). This formula measures the difference between the true label and the predicted probability, encouraging the model to improve its predictions to minimize the overall loss.

[0153] In order to solve the problem of ambiguity in context understanding that may occur when BERT processes long text data, the present invention introduces the Soft Mixture of Experts (Soft Moe) method. Soft Moe is a combination model based on an expert network that can better adapt to complex data structures. In text classification tasks, especially for semantically rich long text data, SoftMoe can more comprehensively consider the training results of each expert model, thereby better capturing the multifaceted characteristics of the data. Since the dataset used in the present invention is composed of long texts, the introduction of SoftMoe can better adapt to this complex data structure and improve the model's ability to understand citation context information.

[0154] SM-Soft Moe module: In the present invention, a soft hybrid expert module (SoftMoe) is used to add it before the Feed Forward of the BERT encoding layer. Each token passes through the multi-head attention mechanism of the Transformer layer and the residual network normalization before entering SoftMoe. In this way, the characteristics of the soft hybrid expert can be integrated into the BERT model. In the BERT encoder, the self-attention mechanism is used to calculate the correlation between each word and other words in the input sequence to obtain a better citation context representation. The feedforward neural network is used to perform nonlinear transformation and feature extraction on the output of the self-attention mechanism. Placing Soft Moe between the self-attention mechanism and the feedforward neural network allows Soft Moe to receive the output of the self-attention mechanism as input, and to infer and combine according to different experts. Such a position allows Soft Moe to better utilize contextual relationships and semantic representations, and to dynamically weigh and integrate between different experts, while sparseening the weight matrix of each expert to speed up the training process. As Figure 14 As shown, the expert layer consists of multiple experts, each of which is a simple feedforward neural network that selects a sparse combination of experts to process each input. All parts of the network are jointly trained through backpropagation. The sample sequence "representing Google" enters the SoftMoe module and uses As the input of the sequence, n is the number of tokens and d represents the dimension of the token, such as Figure 14 As shown, for the Soft Moe module, a set of expert networks corresponding to each token is used, which can be expressed as The learnable parameters of each slot are a parameter matrix containing weights. This parameter matrix will be linearly transformed with the input token vector to generate a weight logits vector related to the number of slots. This logits vector represents the weight of each slot for the input token. These assigned weights determine which slot processes each input in the SoftMoe model. The weight Logit is obtained through feedforward calculation. After obtaining the weight Logit matrix, the columns are normalized because the columns correspond to the number of slots. Each slot calculates the linear combination of all input tokens based on these weights. Each expert is responsible for processing 2 slots, where all slots have corresponding parameter vectors. This parameter vector is also d-dimensional and is expressed as Θ∈R d×(n·c) ,use It represents the result of linear combination of input n tokens. As shown in formula (13):

[0155]

[0156] Where X is the input feature, Θ is the weight, and m is the sequence length.

[0157] Here the scheduling weight E i，j Only the columns of XΘ are softmaxed, not the rows. After forming the input slot, it is sent to the expert for calculation to obtain the output slot, as shown in formula (14):

[0158]

[0159] Where c is the slot number, i is the expert number, and i / c means that the i-th expert is applied to the c-th slot.

[0160] Finally, the same original logic as shown in formula (21) is used to normalize each token (i.e., row by row), and the output tokenY is calculated as (n·p) output slots The weight calculation method of combine is shown in formula (15).

[0161]

[0162] Here, Q is called the combination weight, which is the result of applying softmax to the rows of XΘ. The final output is obtained by weighting and combining the representations of each slot according to the Combine Weights matrix. Each column of the Combine Weights matrix corresponds to a slot and determines the contribution of each slot to the final result.

[0163] The test dataset used in this paper is from the Communications of the China Computer Association, covering all papers in the journal since 2005, totaling 3,273 articles. Using the Yolo v8 model to identify and parse the paper PDFs, we first manually annotated the coordinates of various sections of the document, including the main text, various headings, and citations. We then used the PDFPlumer tool to extract the citations within the document. After manual collation and review, the raw data was screened, cleaned, and filtered. Finally, we extracted a total of 16,845 citations. The data is shown in Table 8:

[0164] Table 8 CCCF citation data statistics

[0165] title Subtitle Citation References 1341 3321 16845 18885

[0166] By analyzing the existing classification of citation intent and combining it with the structure and characteristics of the dataset, we propose the most appropriate classification strategy for this data. The classification strategy is shown in Table 9:

[0167] Table 9 CCCF citation classification model

[0168] model explain Phenomenon Describe the occurrence of a simple phenomenon of citation application The content mentions application, or the literature is applied question Describe the source of the problem evidence Describe the proof relationship in the content Examples Explanation that has no other function, citation ordinary Vague results important Detailed, effective, and successful results Inspiration The inspirational conclusions from this article

[0169] According to the results of classifying and predicting 16,845 pieces of data, the amount of data for each category is as follows: Figure 15 As shown, it can be seen that the common and important categories of the result class account for a larger proportion, while the supporting and inspiration classes have less data. This may result in better training prediction results for the common and important categories of the result class, while other categories have relatively fewer predictions. The number of training sets, test sets, and validation sets divided by the data of this invention are 13476, 1685, and 1685 respectively.

[0170] Table 10 Evaluation indicator confusion matrix

[0171]

[0172] As shown in Table 10, the evaluation indicators used in this invention are accuracy, precision, recall, training time (Per-Epoch time), and Micro F1 value (F-Measure). Among them, TP is the number of true positive examples, TN is the number of true negative examples, FP is the number of false positive examples, and FN is the number of false negative examples.

[0173] As shown in formula (22), the present invention uses the average Jaccard coefficient as Accuracy, which is mainly used for similarity calculation between sets, where D is the test set, s represents each sample in D, and G s is the true output of sample s, P s is the model prediction output of sample s. This indicator reflects the overall similarity between the model prediction output and the reference output. The F1 value can comprehensively consider the precision and recall rate of the model and is a relatively balanced performance evaluation indicator. In the present invention, due to the uneven distribution of samples, the Micro F1 value is used as the evaluation indicator because it takes into account the contribution of each sample and is more suitable for the case of uneven data distribution. The calculation method of the indicator is as shown in formulas (16)-(19), where Recall micro , Precision micro They represent the total Precision and Recall of all categories respectively, i represents the i-th category, and the following experiments are all represented by Recall, Precision, and F1.

[0174]

[0175] TP indicates the number of positive classes predicted as positive, FP indicates the number of negative classes predicted as positive, FN indicates the number of positive classes predicted as negative, and n indicates the total number of classes.

[0176] The Macro F1 metric for Experiment 3 is shown in formulas (20)-(22). In multi-label classification tasks, different labels may have different importances. Some labels may appear more frequently, while others may be very rare. Macro-F1 simply averages the F1 scores of each label, thus ignoring the imbalance of labels. This may lead to insufficient evaluation of rare labels. Therefore, the experiment uses Micro F1 as the evaluation criterion globally and uses this metric as a reference only when testing the public dataset in Experiment 3:

[0177]

[0178] Experimental plan: The effectiveness of the proposed model optimization was verified through experiments. These experiments aimed to answer the following research questions:

[0179] Experiment 1: How does the proposed model compare with the classic model and the latest model?

[0180] Experiment 2: Convergence effect of four indicators on different models.

[0181] Experiment 3: How effective is Soft Moe-SM-BERT in public datasets?

[0182] Experiment 4: Verify the effectiveness of the module through ablation experiments.

[0183] Experiment 5: Explore the impact of different sparsity on the model effect.

[0184] The experiment of this invention selects TextCnn, BERT+TextCnn, Soft Moe-SM-BERT, and RoBERTa-MA models for experimental comparison.

[0185] Experiment 1: Model comparison experiment.

[0186] The results of the multi-label classification experiments of each model on the citation dataset of the present invention are shown in the figure below: Figure 16 As shown, Figure 16 (a) is the accuracy, Figure 16 (b) is the accuracy, Figure 16 (c) is the recall rate, Figure 16 (d) is the F1 value.

[0187] In the test results, the F1 value of the classic classification model TextCnn is compared with SoftMoe-SM-BERT, and the difference is 5%. The data set is long text data. The context understanding ability of TextCNN is far inferior to other models. Its input is fixed length, which will truncate the long text, resulting in information loss and limiting the ability to capture semantics. Soft Moe-SM-BERT is 3.9% higher than BERT+Textcnn, and the effect is significantly improved by comparing soft hybrid experts (Soft Moe) and TextCNN. Soft hybrid experts can obtain prediction results by combining different experts and organically combining weights to reduce the bias and variance of the model to extract more comprehensive and diverse feature information and improve the performance of the model. TextCNN mainly focuses on local features, and the processing of global information is relatively weak. In the classification task of the present invention, global information is very important for the correct classification of long texts, so it is weaker than Soft Moe-SM-BERT in performance. From Figure 16 The results also show that the BERT+TextCNN model achieves a 3.2% higher precision than the SoftMoe-SM-BERT model. This is because the model's limited understanding of context may lead to more conservative predictions, favoring negative examples. This results in higher accuracy when predicting positive examples, i.e., higher precision. SoftMoe-SM-BERT achieves a slight 0.3% higher precision than RoBERTa-MA. RoBERTa-MA builds on the basic output of RoBERTa by adding a Multi-Attention layer to the final output. This introduces multiple attention mechanisms for different labels, capturing relationships between labels. However, in some cases, this may overemphasize these relationships and overlook the importance of predicting each label independently. The SoftMoe-SM-BERT model utilizes a trainable soft routing mechanism, avoiding rigid label assignments. This allows for better utilization of all input information and avoids label omissions or imbalances. This is further supported by the slightly improved F1 score of the SoftMoe-SM-BERT model compared to the latest RoBERTa-MA.

[0188] Depend on Figure 17 It can be seen that the loss of the Soft Moe-SM-BERT model decreases faster than that of RoBERTa-MA, which shows that the Soft Moe-SM-BERT model has advantages in training efficiency or performance. The Soft Moe-SM-BERT model can reduce the loss value faster and learn and adjust the model parameters more effectively during training.

[0189] Experiment 2: Multi-index model convergence experiment.

[0190] The convergence process of each model in the multi-label classification task experiment on the citation dataset of the present invention is as follows: Figure 18 As shown, Figure 18 (a) is the accuracy, Figure 18 (b) is the accuracy, Figure 18 (c) is the recall rate, Figure 18 (d) is the F1 value.

[0191] Depend on Figure 18 As can be seen, the accuracy tends to level off after 12 epochs, with RoBERT-MA having a slightly higher accuracy. BERT+TextCNN has the highest value. In RECALL, Soft Moe-SM-BERT converges faster than RoBERTa-MA, with RoBERTa-MA having the highest value and TextCNN having the lowest value. In terms of F1, Soft Moe-SM-BERT is slightly higher than RoBERTa-MA. In terms of accuracy, recall, and F1, it can be seen that Soft Moe-SM-BERT converges the fastest.

[0192] Experiment 3: Public dataset effect verification experiment.

[0193] The model of Experiment 1 was used for comparison and tested on a public dataset. The dataset used is Ren-CECps, which consists of sentences selected from Chinese blogs. These sentences have been manually annotated with 8 basic emotions: anger, anxiety, expectation, hatred, joy, love, sadness, and surprise. This Chinese blog corpus contains rich emotional annotation information. The data distribution and the number of categories are similar to the data of this invention, and can be used as a reference for model optimization. The data distribution is as follows: Figure 19 shown.

[0194] Figure 20 The comparative experimental results of the Ren-CECps dataset are shown in Figure 2. Figure 20 (a) is the accuracy, Figure 20 (b) is Micro F1, Figure 20 (c) is Macro F1, from Figure 20 As can be seen, Soft Moe-SM-BERT outperforms RoBERTa-MA in accuracy by 2.2 percentage points. In Micro F1, Soft Moe-SM-BERT outperforms RoBERTa-MA by 0.4 percentage points. In Macro F1, Soft Moe-SM-BERT outperforms RoBERTa-MA. Micro F1 is calculated by combining the true and predicted labels of all samples. Therefore, it focuses more on labels that appear frequently in the dataset, indicating that Soft Moe-SM-BERT performs better at identifying common labels in the dataset, resulting in a higher Micro F1 score than RoBERTa-MA. Macro F1 focuses more on the model's performance on each label, indicating that RoBERTa-MA performs worse than Soft Moe-SM-BERT when handling rare or challenging labels in the dataset.

[0195] Experiment 4: Ablation experiment.

[0196] In order to verify the effectiveness of the optimization, an ablation experiment was conducted on the CCCF citation dataset of the present invention. The experimental results are as follows: Figure 21 As shown, Figure 21 (a) is the accuracy, Figure 21 (b) is the accuracy, Figure 21 (c) is the recall rate, Figure 21 (d) is the F1 value.

[0197] Soft Moe-SM-BERT achieves the highest accuracy, surpassing SM-BERT by 0.5 percentage points. Soft Moe-BERT achieves the highest precision at 57.2%, while BERT-base achieves the highest recall at 49.2%. This is due to the fact that, during BERT's pre-training task, BERT-base is trained to predict masked tokens, which can have multiple possible answers. To improve accuracy, BERT-base tends to select the most common or plausible answer, ignoring other possibilities. The SoftMoE layer performs a weighted combination of all input tokens with each expert to generate the output token. This trainable distribution mechanism enables Soft Moe-BERT to more comprehensively consider different possibilities and make predictions more flexibly.

[0198] Compared with BERT, SM-BERT has a slightly improved effect, mainly by reducing the amount of computation and accelerating the inference process, as shown in Table 11:

[0199] Table 11 CCCF citation dataset training speed results

[0200]

[0201] As can be seen from the Soft Moe-SM-BERT effect in the figure, Soft Moe soft hybrid experts solve this problem and can train more key information. The high accuracy of Soft Moe-SM-BERT shows that the model can evaluate global performance and achieve a balance in the classification process. The F1 value of the model is also the best, indicating that the soft hybrid experts enhance the decision-making ability of the model. Hybrid experts can make decisions through voting, weighted averaging, etc., thereby improving the accuracy and robustness of the model. The iterative process is as follows Figure 22 shown.

[0202] Experiment 5: Sparsity comparison experiment

[0203] Depend on Figure 22 It can be seen that SoftMoe-SM-BERT has the fastest iteration speed. The introduction of sparse soft hybrid experts enables the model selection experts to learn data features in a targeted manner, which can accelerate the convergence process to a considerable extent.

[0204] Figure 23 is the result of sparsity comparison; Figure 23 (a) is the accuracy, Figure 23 (b) is the accuracy, Figure 23 (c) is the recall rate, Figure 23 (d) is the F1 value. Figure 23 It can be seen that the effect is relatively poor when the sparsity is 0.25 and 0.5. The model effect gradually increases with sparsity between 0.5 and 0.8, and the F1 value is the best at 0.8. When the sparsity is 1, the effect drops significantly. Analysis shows that when the model sparsity is too high, such as a sparsity of 0.25, the connections between some neurons are cut or weakened. The model cannot use these cut connections to transmit important information, resulting in information loss. If the sparsity is too low, such as a sparsity of 1, redundant information will exist in the network and overfitting of the training data will occur, resulting in the loss of key information and reduced generalization ability of the model. When choosing sparsity, it is necessary to strike a balance between information retention and model performance. In this model, a sparsity of 0.8 is the best.

[0205] The present invention mainly introduces the content of identifying the citation intention of the extracted citations and extracting labels for the references. The application of citations in articles has multiple functions, including increasing the credibility of the article, supporting arguments and viewpoints, emphasizing research innovation and increasing verifiability. The present invention proposes the Soft Moe-SM-BERT model for citation intention multi-classification experiments. The experimental results show that the Soft Moe-SM-BERT model has advantages in accuracy and F1 value, especially in terms of loss reduction rate and training speed. It is significantly better than other models. Ablation experiments and instance-based case learning further verified the optimization effect and performance improvement of the model, indicating that the Soft Moe-SM-BERT model can better handle multi-label classification tasks.

[0206] The present invention can extract the document architecture by using the Yolo++ algorithm and decompose the document to be retrieved into multiple structural units of different granularities according to chapters; divide multiple different structural units into different categories according to subtitles, and divide the document to be retrieved into a complete hierarchical structure; use the prefix-tuned PT-GLM-6B model to extract the main content of the paragraphs corresponding to the subtitles of each category. This process only requires fine-tuning a small number of prefix parameters to adapt to different tasks, and this method allows PT-GLM-6B to perform customized parameter adjustments for different categories of tasks, thereby better matching task requirements. This makes PT-GLM-6B have stronger cross-task generalization capabilities and can achieve excellent performance on various different categories of tasks, thereby being able to propose the main content in different subtitles; the present invention can accelerate the calculation speed by constructing a sparse matrix in the Soft Moe-SM-BERT model; by introducing Soft Moe to perform weighted averaging on tokens, it can better capture long-distance semantic dependencies; the present invention constructs a hyponym and hyponym network, which can query existing hypernyms and perform auxiliary prediction on new words that do not exist, thereby extracting research directions and methods of the document to be retrieved. Combined with the main content of the extracted documents to be retrieved, the citation intention, research direction and method, matching documents are recommended to users.

[0207] The present invention also provides a document guide system, comprising:

[0208] Data acquisition module, used to obtain documents to be retrieved;

[0209] The backbone content extraction module uses the Yolo++ algorithm to extract the document architecture of the document to be retrieved, decomposes the document into multiple structural units of different granularity according to chapters, and divides the multiple structural units into different categories according to subtitles. It adds a prefix vector to each attention layer of the encoder of the GLM-6B generative language model to obtain a prefix-tuned PT-GLM-6B model. The prefix vector of the PT-GLM-6B model is used to extract the backbone content of the paragraphs corresponding to the subtitles of each category.

[0210] The citation intent recognition module is used to identify the citation intent of the retrieved documents using the BERT model as the base model. The sparse matrix SM and the soft hybrid expert model Soft Moe are introduced into the BERT model to obtain the sparse soft hybrid expert model Soft Moe-SM-BERT. The Soft Moe-SM-BERT is used to identify the citation intent of the retrieved documents.

[0211] The introductory module is used to introduce the sparse matrix SM and the soft mixed expert model Soft Moe into the BERT model based on the BERT model to obtain the sparse soft mixed expert model SoftMoe-SM-BERT. The Soft Moe-SM-BERT is used to identify the citation intention of the retrieved documents.

[0212] The present invention also provides a computer device, comprising a memory and a processor; the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the document guide method.

[0213] The present invention also provides a computer-readable storage medium, which stores a computer program. The computer program is suitable for being loaded by a processor to execute the literature guide method.

[0214] The above-described embodiments are only preferred specific implementation methods of the present invention, and the protection scope of the present invention is not limited thereto. Any simple changes or equivalent replacements of the technical solutions that can be obviously obtained by any technician familiar with the field within the technical scope disclosed in the present invention fall within the protection scope of the present invention.< / eos>

Claims

1. A method for reading a document, characterized in that: include: Obtain the literature to be searched; The Yolo++ algorithm is used to extract the document architecture of the document to be retrieved, and the document to be retrieved is decomposed into multiple structural units of different granularity according to chapters. Multiple different structural units are divided into different categories according to subtitles. A prefix vector is added before each attention layer of the encoder of the GLM-6B generative language model to obtain a prefix-tuned PT-GLM-6B model. The prefix vector of the PT-GLM-6B model is used to extract the main content of the paragraph corresponding to the subtitle of each category. Taking the BERT model as the base model, the sparse matrix SM and the soft mixed expert model Soft Moe are introduced into the BERT model to obtain the sparse soft mixed expert model Soft Moe-SM-BERT. Soft Moe-SM-BERT is used to identify the citation intent of the retrieved documents. Provide users with literature guidance based on the extracted main content and citation intention.

2. The method for reading literature according to claim 1, wherein: The method further comprises extracting the document architecture of the document to be retrieved using the Yolo++ algorithm and decomposing the document to be retrieved into a plurality of structural units of different granularities according to chapters, extracting the text, pictures and tables of each structural unit as leaf nodes, and constructing a hierarchical structure of the document text according to the extracted leaf nodes; The subheadings of each structural unit are classified according to the hierarchical structure, and subheadings that do not clearly reflect the structural unit are replaced.

3. The method for reading literature according to claim 1, wherein: The method of using Soft Moe-SM-BERT to identify the citation intention of the documents to be retrieved specifically includes the following steps: converting the sample sequence of the documents to be retrieved into multiple token sequences, and marking the multiple token sequences; each token is mapped to a corresponding vector representation in the vocabulary; Use BERT in SoftMoe-SM-BERT to add a type embedding to each token and generate a position embedding for each token; add the vector representation of each token to its type embedding and position embedding to obtain the final embedding representation of each token; The final embedding representation of each token is input into Soft Moe, which captures contextual relationships and semantic representations, and dynamically weighs and integrates them among different experts. At the same time, the weight matrix of each expert is sparsely constructed to identify the citation intention.

4. The method for reading literature according to claim 3, wherein: The token sequence includes multiple tokens. A token is the smallest unit processed by SoftMoe-SM-BERT and includes words and punctuation marks.

5. The method for reading literature according to claim 1, wherein: The Soft Moe is located between the self-attention mechanism and the feedforward neural network of the BERT encoding layer.

6. A literature reading system, characterized in that: include: Data acquisition module, used to obtain documents to be retrieved; The backbone content extraction module uses the Yolo++ algorithm to extract the document architecture of the document to be retrieved, decomposes the document into multiple structural units of different granularity according to chapters, and divides the multiple structural units into different categories according to subtitles. It adds a prefix vector to each attention layer of the encoder of the GLM-6B generative language model to obtain a prefix-tuned PT-GLM-6B model. The prefix vector of the PT-GLM-6B model is used to extract the backbone content of the paragraphs corresponding to the subtitles of each category. The citation intent recognition module is used to identify the citation intent of the retrieved documents using the BERT model as the base model. The sparse matrix SM and the soft hybrid expert model Soft Moe are introduced into the BERT model to obtain the sparse soft hybrid expert model Soft Moe-SM-BERT. The Soft Moe-SM-BERT is used to identify the citation intent of the retrieved documents. The introductory module is used to introduce the sparse matrix SM and the soft mixed expert model Soft Moe into the BERT model based on the BERT model to obtain the sparse soft mixed expert model SoftMoe-SM-BERT. The Soft Moe-SM-BERT is used to identify the citation intention of the retrieved documents.

7. A computer device, characterized in that: The method comprises a memory and a processor; the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the literature guide method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the literature guide method according to any one of claims 1 to 5.

Images