A device and method for classifying titles of academic papers based on Bert

Abstract

The application relates to a Bert-based academic paper title grading device and method, the grading device comprising a data reading module, a feature fusion module, a grading prediction module, a dynamic context expansion module and a structured title tree construction module; wherein the feature fusion module extracts layout features and normalizes the layout features, then linearly projects the layout features, copies the layout features to embedding positions corresponding to all tokens after tokenization of corresponding title text blocks, finally splices layout feature vectors after copying with text embedding vectors of corresponding tokens in feature dimensions, and dimensionally reduces and fuses high-dimensional features after splicing to output embedding vectors of fused multi-modal information; the grading prediction module internally applies a YARN algorithm to a grading model trained in stages, the grading model adopts a training strategy in stages, different base frequency parameters and scaling factors of rotary position coding are set for different target context length ranges. The application has high classification accuracy and fast classification speed.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing and artificial intelligence, and specifically relates to a BERT-based device and method for classifying academic paper titles. Background Technology

[0002] With the development of artificial intelligence technology, using large language models for literature reading is becoming an important part of academic research. However, due to the limitations of large language models, parsing academic papers in PDF format remains a key task.

[0003] Currently, the difficulties in parsing PDF papers include:

[0004] (1) General document models (such as LayoutLMv3) do not capture the contextual semantics of the heading chain well enough;

[0005] (2) The rule-based tools have a high probability of misidentification and cannot cover all cases;

[0006] (3) Traditional layout models cannot parse multi-level headings;

[0007] (4) It is difficult to handle when there are too many titles. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention provides a method and apparatus for classifying academic paper titles based on BERT. By using a BERT model based on three-stage fine-tuning of YARN, it achieves accurate classification of multi-level titles in academic papers with high speed and accurate classification results.

[0009] The objective of this invention is achieved through the following technical solution:

[0010] A BERT-based academic paper title classification device, comprising:

[0011] The data reading module is used to acquire the text blocks and layout information after parsing PDF academic papers;

[0012] The feature fusion module is used to format and tokenize the text information of title text blocks identified as Title type according to their natural order of appearance in the document; it also generates a corresponding normalized layout feature vector for each title text block, and after linear projection, copies it to the embedding positions of all tokens after word segmentation of the corresponding title text block; finally, it concatenates the copied layout feature vector with the text embedding vector of the corresponding token in the feature dimension, and the concatenated high-dimensional features are dimensionality reduced and fused through a linear projection layer to output an embedding vector that fuses multimodal information;

[0013] The hierarchical prediction module is used to perform title classification and output the hierarchical category of each title text block. The hierarchical prediction module has a built-in hierarchical model trained in stages using the YARN algorithm. The hierarchical model is a model formed by adding no less than two linear layers after the output layer of the BERT model. The input of the hierarchical model is an embedding vector that integrates multimodal information, and the output is the hierarchical category of each title text block.

[0014] The dynamic context extension module is used to temporarily adjust the scaling factor in the rotational position encoding when the length of the input sequence of the hierarchical model is greater than 2048, expand the range supported by the position encoding through interpolation, and realize long sequence inference; and after the prediction is completed, the scaling factor is reset to the system default value and the fundamental frequency parameter of the original rotational position encoding is restored.

[0015] The structured title tree construction module is used to convert the hierarchical probability distribution of each title text block output by the hierarchical prediction module into the final predicted label, and construct a structured title tree according to the page order and parent-child relationship.

[0016] Furthermore, the normalized layout feature vector includes six normalized values, namely the page normalized value of the horizontal coordinate of the top left corner of the text block, the page normalized value of the vertical coordinate of the top left corner of the text block, the ratio of the text block width to the page width, the ratio of the text block height to the page height, the normalized position of the current page number in the full text, and the normalized value of the total number of pages in the document.

[0017] Furthermore, the feature fusion module is used to format and tokenize the text content of the title text block identified as Title type according to its natural order of appearance in the document, specifically through the following operations:

[0018] Each consecutive title text is concatenated using the [SEP] delimiter and marked with a [CLS] tag at the beginning to form a unified input sequence; then the sequence is encoded by the Tokenizer of the hierarchical model to generate text word embedding vectors.

[0019] Furthermore, the phased training involves setting different fundamental frequency parameters and scaling factors for rotational position encoding for different target context length ranges; the phased training of the hierarchical model using the YARN algorithm specifically comprises the following three phases of training:

[0020] Phase 1: Target context length range 0-512, rotation position encoding base frequency parameter ROPE_base=10000, scaling factor scale=1.0;

[0021] Phase 2: Target context length range 512-1024, rotation position encoding base frequency parameter ROPE_base=20000, scaling factor scale=5.0;

[0022] Phase 3: Target context length range 1024-2048, rotation position encoding base frequency parameter ROPE_base=50000, scaling factor scale=10.0.

[0023] Furthermore, the specific process by which the feature fusion module obtains the final input embedding vector that fuses multimodal information is as follows:

[0024] (1) The 6-dimensional normalized layout features of the normalized layout feature vector are mapped to a 768-dimensional vector through a linear layer, so that they are aligned with the Bert word dimension;

[0025] (2) Expand the mapped layout feature vector in the sequence dimension to align it with the embedding vector of each text word;

[0026] (3) The text word embedding vector and the layout embedding vector are concatenated along the feature dimension to form a 1536-dimensional joint feature.

[0027] (4) Compress it back to 768 dimensions through another linear layer to generate an enhanced lexical representation, which is the final input embedding vector that integrates multimodal information.

[0028] Furthermore, the text information of the title text block of type Title includes the text content, the position information of the text block on the page, and the page information where the text block is located.

[0029] A BERT-based method for classifying academic paper titles, comprising the following steps:

[0030] S1: Parse PDF academic papers and extract the set of continuous text blocks that make up the title chain and their layout attributes;

[0031] S2: Connect all text information of title text blocks identified as Title type in the entire academic paper into a single long text sequence according to their natural order of appearance in the document, using the [SEP] delimiter, and add the [CLS] mark at the beginning of the sequence to form the single input sequence of the BERT model;

[0032] S3: Generate a corresponding normalized layout feature vector for each title text block; perform linear projection on the normalized layout feature vector, and copy the projected normalized layout feature vector to the embedding positions corresponding to all tokens after word segmentation of the title text block; then concatenate the copied normalized layout feature vector with the text embedding vector of the corresponding token in the feature dimension; reduce the dimension of the concatenated high-dimensional features through a linear projection layer and fuse them to obtain the final input embedding vector that fuses multimodal information.

[0033] S4: A hierarchical model is constructed by adding no less than two linear layers after the output layer of the Bert model. The input of the hierarchical model is an embedding vector that integrates multimodal information, and the output is the hierarchical category of each title text block. The hierarchical model is trained in stages using the YARN algorithm. Different fundamental frequency parameters and scaling factors for rotation position encoding are set for different target context length ranges to obtain the trained hierarchical model.

[0034] S5: Use the trained hierarchical model for inference to obtain the hierarchical probability distribution of each title text block output by the trained hierarchical model; and when the length of the input sequence is greater than 2048, temporarily adjust the scaling factor in the rotation position encoding, expand the position encoding support range through interpolation, realize long sequence inference, and after the prediction is completed, reset the scaling factor to the system default value and restore the fundamental frequency parameter of the original rotation position encoding.

[0035] S6: Transform the hierarchical probability distribution of each title text block into the final predicted label, and construct a structured title tree according to the page order and parent-child relationship.

[0036] Furthermore, a hierarchical model is constructed by adding three linear layers after the output layer of the BERT model.

[0037] An electronic device, comprising:

[0038] One or more processors;

[0039] A storage device for storing one or more programs that, when executed by the electronic device, enable the electronic device to implement a Bert-based academic paper title classification method.

[0040] A computer-readable storage medium having a program stored thereon that, when executed by a processor, implements a BERT-based method for classifying academic paper titles.

[0041] The beneficial effects of this invention are as follows:

[0042] (1) The academic paper title classification device and method of the present invention innovatively integrates text semantic features and key page layout features, based on the global attention mechanism of BERT, and based on the input representation method of continuous title [SEP] concatenation, so that the model can accurately understand all title content and layout; and adopts the YARN algorithm to expand the position encoding in stages and combined with the dynamic interpolation deployment strategy, so that the model can be applied to the case of very long title text sequences and has high classification accuracy.

[0043] (2) The present invention can realize fully automatic processing from the original PDF to the structured title tree, which significantly improves the efficiency of document parsing and information extraction. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 The overall flowchart of the BERT-based academic paper title classification method provided by this invention;

[0046] Figure 2 This is a schematic diagram of multimodal feature fusion in the fine-tuned Bert model provided by the present invention;

[0047] Figure 3 This is a schematic diagram of multi-stage fine-tuning based on YARN provided by the present invention;

[0048] Figure 4 The device structure block diagram provided by the present invention;

[0049] Figure 5 This is a structural diagram of the electronic device provided by the present invention. Detailed Implementation

[0050] The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments. The purpose and effects of the present invention will become clearer. It should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0051] Explanation of technical terms:

[0052] BERT (Bidirectional Encoder Representations from Transformers): A bidirectional pre-trained language model based on the Transformer architecture, capable of simultaneously utilizing contextual information to perform semantic modeling of words or sentences, and widely used in natural language understanding tasks.

[0053] ROPE (Rotary Position Embedding): A positional encoding method for Transformer models. It injects positional information into word vectors through a rotation matrix, and has better extrapolation and long sequence modeling capabilities compared to traditional absolute positional encoding.

[0054] YARN (Yet Another RoPE-based NTK-aware Method): A positional encoding extrapolation optimization algorithm based on ROPE. By scaling and interpolating the rotation fundamental frequency, it expands the model's ability to process long text sequences without retraining the model, while maintaining the performance of short texts.

[0055] The inventive concept of this invention is as follows: Based on the Arxiv open-source paper, a training dataset is constructed, and the input features are constructed by utilizing BERT's global attention mechanism and the multimodal characteristics of the title; the YARN-optimized ROPE algorithm is applied to extrapolate the model context window, and a dynamic interpolation deployment method is designed to avoid the impact of extremely long texts on normal input.

[0056] On the one hand, this invention provides a BERT-based method for classifying academic paper titles, such as... Figure 1 As shown, the method includes the following steps:

[0057] Step 1: Parse the PDF academic paper and extract the set of continuous text blocks that make up the title chain and their layout attributes.

[0058] PDF academic papers can originate from academic databases or user uploads.

[0059] In this embodiment, a structurally complete academic paper is obtained from open-source platforms such as arXiv, and its PDF file and corresponding LaTeX source code are downloaded. The LaTeX source code is used to extract the real title hierarchy structure (such as commands like \section{} and \subsection{}) as ground truth labels for supervised training. The YOLOv10 layout analysis model is then used to detect text blocks in the PDF and identify candidate regions of type "Title". Subsequently, PaddleOCR is used to extract the text content and record its bounding box coordinates and page information.

[0060] Here, "contents" represents the extracted text content;

[0061] The bbox represents the position information of the text box on the page. It consists of 8 values, represented as bbox[x0,y0,x1,y1,x2,y3,x3,y3]. Each pair of values ​​represents the horizontal and vertical coordinates of a point, and the four points represent the coordinates of a rectangle.

[0062] The page information of a text block is represented by page_info, which consists of three values, denoted as page_info[page_id, page_height, page_width], where page_id represents the page ID, page_height represents the page height, and page_width represents the page width.

[0063] Step 2: Text Concatenation: All text information of title text blocks identified as Title type in the entire academic paper are concatenated into a single long text sequence by using the [SEP] delimiter according to their natural order of appearance in the document, and a [CLS] mark is added to the beginning of the sequence to form a single input sequence for the BERT model.

[0064] In this embodiment, the specific operation is as follows: each consecutive title text is concatenated using the "[SEP]" separator, and a "[CLS]" marker is added at the beginning to form a unified input sequence. Subsequently, this sequence is encoded using a pre-trained BERT tokenizer to generate a token embedding vector (token_emb), which will serve as the basic input for subsequent multimodal fusion. This design enables the model to simultaneously perceive the semantic relationships between multiple titles, improving the contextual awareness capability of hierarchical judgment.

[0065] In code terms:

[0066] input_text = "[CLS]Heading 1[SEP]Heading 2[SEP]Heading 3...[SEP]"

[0067] token_emb=encoding(input_text)

[0068] Step 3: Generate a normalized layout feature vector for each title text block; perform linear projection on the normalized layout feature vector, and copy the projected normalized layout feature vector to the embedding positions corresponding to all tokens after word segmentation of the title text block; then concatenate the copied normalized layout feature vector with the text embedding vector of the corresponding token in the feature dimension; reduce the dimension of the concatenated high-dimensional features through a linear projection layer and fuse them to obtain the final input embedding vector that fuses multimodal information.

[0069] The layout feature vector consists of six normalized values, represented as [x_norm, y_norm, w_norm, h_norm, page_id_norm, page_total_norm], where:

[0070] x_norm represents the page-normalized value of the x-coordinate of the top-left corner of the text block;

[0071] y_norm represents the page-normalized value of the top-left y-coordinate of the text block;

[0072] w_norm represents the proportion of the text block width to the page width;

[0073] h_norm represents the proportion of the text block height to the page height;

[0074] page_id_norm represents the normalized position of the current page number in the full text;

[0075] page_total_norm represents the normalized value of the total number of pages in the document. It helps the model perceive the overall length of the current document, thereby more accurately determining the relative importance of the headings in the document.

[0076] The specific calculation methods for each feature are as follows:

[0077] x_norm = bbox[0][0] / page_width, that is, divide the x-coordinate of the top left corner of the text block by the page width to get its relative position (between 0 and 1) in the horizontal direction of the page.

[0078] y_norm = bbox[0][1] / page_height, that is, divide the ordinate of the top left corner of the text block by the page height to get its relative position (between 0 and 1) in the vertical direction of the page.

[0079] h_norm = (bbox[2][1]- bbox[0][1]) / page_height, that is, calculate the height of the text block (by subtracting the y-coordinate of the top left corner from the y-coordinate of the bottom right corner), and then divide by the page height to get the height ratio;

[0080] w_norm = (bbox[2][0]- bbox[0][0]) / page_width, that is: calculate the width of the text block (by subtracting the x-coordinate of the top left corner from the x-coordinate of the bottom right corner), and then divide by the page width to get the width ratio;

[0081] page_id_norm = page_id / max_pages, which means: divide the current page number by the total number of pages in the document to get the relative position of the page containing the text block in the whole text (e.g., page 5 of 100 → 0.05).

[0082] `page_total_norm = max_pages / 500` means dividing the total number of pages in the document by the preset global maximum number of pages, 500, to normalize the document length and help the model determine the relative importance of the title in the current document.

[0083] The above normalization process aims to eliminate the inconsistency in feature scale caused by differences in page size and total number of pages among different PDF documents, enabling the model to generalize across documents and improve the robustness of title level prediction.

[0084] like Figure 2 As shown, when performing multimodal fusion of layout features and text features, the 6-dimensional normalized layout features (x_norm, y_norm, w_norm, h_norm, page_id_norm, page_total_norm) are first mapped to a 768-dimensional vector (aligned with the BERT terminology dimension) through a linear layer. Then, this layout vector is extended along the sequence dimension to align with each terminology. Next, the text terminology embeddings and layout embeddings are concatenated along the feature dimension to form a 1536-dimensional joint feature. Finally, it is compressed back to 768 dimensions through another linear layer to generate the enhanced terminology representation (enhanced_emb). This design allows the model to utilize both semantic and spatial typographic information simultaneously, significantly improving the accuracy of title hierarchy classification. The specific code implementation is as follows:

[0085] 1)layout_emb = Linear(6, 768)(layout_features)

[0086] 2)layout_emb = layout_emb.unsqueeze(1).expand_as(token_emb)

[0087] 3)combined = torch.cat([token_emb, layout_emb], dim=-1)

[0088] 4)enhanced_emb = Linear(768 * 2, 768)(combined)

[0089] Step 4: Construct a hierarchical model by adding at least two linear layers after the output layer of the BERT model. The input of the hierarchical model is an embedding vector that integrates multimodal information, and the output is the hierarchical category of each title text block. Apply the YARN algorithm to the hierarchical model for phased training, and set different fundamental frequency parameters and scaling factors for rotation position encoding for different target context length ranges to obtain the trained hierarchical model.

[0090] To achieve title level classification prediction, the title representation vector (title_states) output by BERT is first subjected to Dropout regularization; then, a non-linear transformation is performed through a fully connected layer fc1; next, layer normalization is performed using LayerNorm, and the model's expressive power is enhanced by the GELU activation function; after another Dropout, the probability distribution (logits) of each title block belonging to different levels is output through the final classification layer fc2.

[0091] The above prediction process can be represented in code as follows:

[0092] 1)x = self.dropout(title_states)

[0093] 2) x = self.fc1(x)

[0094] 3) x = self.layer_norm(x)

[0095] 4) x = F.gelu(x)

[0096] 5) x = self.dropout(x)

[0097] 6) logits = self.fc2(x)

[0098] The output hierarchy can also be expressed as: Bert Output → Linear(768,256) → ReLU → Linear(256,128) → Linear(128,4). Here, the output dimension "4" corresponds to four levels of heading classification (e.g., Level-1 to Level-4), which can be expanded according to actual needs. Cross-entropy loss is used for optimization, and argmax is taken as the final level label during prediction.

[0099] like Figure 3As shown, to balance the model's performance and training stability across sequences of different lengths, this invention employs a three-stage progressive fine-tuning strategy. This training method aims to gradually improve the model's ability to handle long title sequences (supporting up to 2048 tokens) without sacrificing short text modeling performance. The specific parameter settings for each stage are as follows:

[0100] Phase 1: Short Context Stable Training (0–512 tokens)

[0101] Objective: To strengthen the model's basic semantic understanding capabilities and avoid gradient oscillations in the early stages of training with long texts;

[0102] Parameter settings:

[0103] ROPE_base = 10000 (original Bert default value)

[0104] scale = 1.0 (no scaling, preserve original position encoding)

[0105] Training strategy: Freeze all parameters of all BERT Transformer layers, fine-tune only the classification head (three-layer MLP), learning rate = 1e-5, train for 3 epochs;

[0106] Data sampling: Only the title sequence within the first 512 tokens of each paper is extracted.

[0107] Phase 2: Medium-length extrapolation training (512–1024 tokens)

[0108] Objective: To initially expand the context window, introduce the YARN scaling mechanism, and adapt to cross-page title modeling;

[0109] Parameter settings:

[0110] ROPE_base = 20000 (Increases the base frequency and improves the stability of high-frequency components)

[0111] scale = 5.0 (scaling factor, used for interpolation extrapolation)

[0112] Training strategy: Unfreeze the parameters of the last two Transformer layers, freeze the parameters of the remaining layers, set the learning rate to 5e-6, and train for 3 epochs;

[0113] Data sampling: Sample header sequences of length 512–1024 tokens, padding if insufficient, and truncating if excessive.

[0114] Phase 3: Long text adaptation training (1024–2048 tokens)

[0115] Objective: To fully support long title chain modeling and optimize the capture of long-distance dependencies across multiple pages and multiple levels of titles;

[0116] Parameter settings:

[0117] ROPE_base = 50000 (Further expand the base frequency to suppress high-frequency oscillations)

[0118] scale = 10.0 (Significant scaling, supports extrapolation up to 2048 lengths)

[0119] Training strategy: Fine-tuning all parameters, learning rate = 1e-6, training for 3 epochs, using gradient clipping (clip_norm = 1.0) to ensure stability;

[0120] Data sampling: All title sequences of a single paper are fully preserved, with dynamic padding up to 2048.

[0121] Step 5: Use the trained hierarchical model for inference to obtain the hierarchical probability distribution of each title text block output by the trained hierarchical model; when the length of the input sequence is greater than 2048, temporarily adjust the scaling factor in the rotation position encoding, expand the range supported by position encoding through interpolation to achieve long sequence inference, and reset the scaling factor to the system default value after the prediction is completed, and restore the fundamental frequency parameter of the original rotation position encoding.

[0122] Step 6: Convert the hierarchical probability distribution of each title text block into the final predicted label, and construct a structured title tree according to the page order and parent-child relationship.

[0123] On the other hand, the present invention also provides a BERT-based academic paper title classification system, such as... Figure 4 As shown, its core lies in encapsulating model capabilities as modular components, supporting dynamic context expansion and seamless integration into the document parsing pipeline. The system includes the following key modules:

[0124] The data reading module receives text blocks identified as "Title" type data from upstream document parsing engines (such as YOLOv10 layout analysis + PaddleOCR). This data includes structured fields such as text content, bounding box coordinates (bbox), and page information (page_id, page_height, page_width). This module serves as the system entry point, ensuring standardized input data format and providing a foundation for subsequent feature construction.

[0125] The feature fusion module is used to format and tokenize the text information of title text blocks identified as Title type according to their natural order of appearance in the document. It also generates a corresponding normalized layout feature vector for each title text block, linearly projects it, and copies it to the embedding positions of all tokens after word segmentation. Finally, it concatenates the copied layout feature vector with the text embedding vector of the corresponding token along the feature dimension, and then fuses the concatenated high-dimensional features through a linear projection layer to output an embedding vector that integrates multimodal information. The feature fusion module fuses layout and text, enabling the hierarchical prediction module to simultaneously perceive "what is said" and "where it is placed," significantly improving the ability to distinguish title levels (e.g., centered large text for chapter titles, indented small text for subsection titles).

[0126] The hierarchical prediction module is used to perform title classification and output the hierarchical probability distribution of each title text block. The hierarchical prediction module has a built-in hierarchical model trained in stages using the YARN algorithm. The hierarchical model is formed by adding no less than two linear layers after the output layer of the BERT model. The input of the hierarchical model is an embedding vector that integrates multimodal information, and the output is the hierarchical category of each title text block. The staged training sets different fundamental frequency parameters and scaling factors for rotation position encoding for different target context length ranges.

[0127] The dynamic context extension module addresses the issue that academic paper title sequences may exceed the model's maximum length (2048 tokens) by introducing a YARN dynamic interpolation mechanism. Specifically, when the input sequence length of the hierarchical model exceeds 2048 tokens, it temporarily adjusts the scaling factor in the rotational position encoding, expanding the supported range of position encoding through interpolation to enable long sequence inference. After prediction, the scaling factor is immediately reset to the system default value (e.g., 1.0), and the original rotational position encoding fundamental frequency parameter is restored to ensure that subsequent model inference is unaffected. In practice, the system caches the default scale and ROPE_base parameters during initialization, modifies only the runtime copy during dynamic adjustment, and restores them from the cache after prediction, ensuring thread safety and state isolation.

[0128] The structured title tree construction module is used to convert the hierarchical probability distribution (logits) of each title text block output by the hierarchical prediction module into final predicted labels (such as Level-1, Level-2, etc.), and construct a structured title tree according to page order and parent-child relationships. This tree structure can be directly output as JSON or XML format, seamlessly integrated into downstream document management systems, knowledge graph construction tools, or reading assistance platforms, realizing fully automatic conversion from raw PDF to structured metadata.

[0129] The BERT-based academic paper title classification system of this invention is not only a title classifier, but also a core component in the intelligent parsing pipeline of academic documents, which significantly improves the automation rate of document processing and reduces the cost of manual annotation. It has been deployed and verified in a scientific research data platform.

[0130] It should be noted that the academic paper title classification system provided in the above embodiments can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the terminal or server into different functional modules to complete all or part of the functions described above.

[0131] The following is a specific implementation example to demonstrate the beneficial effects of the method of the present invention.

[0132] In this implementation case, since there is a lack of high-quality corpora in the current public academic datasets that cover the full text and have complete multi-level heading structure annotations (most of them only annotate a single page or the abstract), in order to support the training and evaluation of the model, the inventors in this case built and manually annotated 50 open-source Chinese and English journal papers covering multiple fields such as computer science, earth science, and engineering, and constructed a dedicated dataset that supports end-to-end multi-level heading hierarchy recognition.

[0133] Then, complete and formatted PDFs of academic papers are obtained from open platforms such as arXiv and PubMed Central. Professional annotators annotate the bounding boxes (bboxes) of each title text block and its level (such as level 1 chapter titles, level 2 section titles, level 3 subsection titles, etc.) to ensure that the annotation granularity is accurate to the paragraph level and covers all pages of the entire text.

[0134] Based on the above-mentioned title classification device and method, the title classification of the above 50 papers was predicted, and the overall accuracy rate (i.e., the proportion of all title block prediction levels that are completely consistent with the manual annotation) was calculated.

[0135] To verify the advantages of this invention, it was compared with the Qwen2.5 32B large model API (combined with manually designed prompts and heuristic rules such as text semantics, line height, and page numbers) used in the current mainstream industrial-grade solution, MinerU system. The comparison results are shown in Table 1.

[0136]

[0137] As shown in Table 1, this invention, through multimodal modeling that integrates semantics and layout, YARN-optimized long context support, and a continuous header block concatenation input mechanism, significantly outperforms the MinerU solution, which relies on Prompt engineering and rule post-processing. It demonstrates greater stability, especially when handling complex scenarios such as cross-page headers and nested hierarchical headers. Furthermore, this invention is based on a lightweight, fine-tuned BERT architecture, resulting in faster inference speeds and eliminating the need to call external large model APIs, thus avoiding network latency and cost overhead. This makes it more suitable for localized and batch deployment.

[0138] This comparative experiment fully verifies the comprehensive advantages of the invention in terms of accuracy, efficiency, and robustness, and it is especially suitable for academic document processing scenarios that require high-precision structured analysis.

[0139] Based on the same inventive concept, the present invention also provides an electronic device, such as... Figure 5 As shown, at the hardware level, in addition to the processor and memory, it also includes an internal bus, network interface, RAM, and other hardware required for business operations. The memory is non-volatile memory. The processor reads the corresponding computer program from the non-volatile memory into RAM and then runs it to implement the academic paper title classification method described above.

[0140] This invention is not limited to software implementation; it can also be implemented using application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or a hardware-software co-architecture. In other words, any apparatus or system that executes the method flow described in this invention falls within the protection scope of this invention.

[0141] It will be understood by those skilled in the art that the above descriptions are merely preferred examples of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the invention should be included within the scope of protection of the invention.

Claims

1. A BERT-based academic paper title classification device, characterized in that, include: The data reading module is used to acquire the text blocks and layout information after parsing PDF academic papers; The feature fusion module is used to format and tokenize the text information of title text blocks identified as Title type according to their natural order of appearance in the document; it also generates a corresponding normalized layout feature vector for each title text block, and after linear projection, copies it to the embedding positions of all tokens after word segmentation of the corresponding title text block; finally, it concatenates the copied layout feature vector with the text embedding vector of the corresponding token in the feature dimension, and the concatenated high-dimensional features are dimensionality reduced and fused through a linear projection layer to output an embedding vector that fuses multimodal information; The hierarchical prediction module is used to perform title classification and output the hierarchical category of each title text block. The hierarchical prediction module has a built-in hierarchical model trained in stages using the YARN algorithm. The hierarchical model is formed by adding no less than two linear layers after the output layer of the BERT model. The input of the hierarchical model is an embedding vector that integrates multimodal information, and the output is the hierarchical category of each title text block. The staged training sets different fundamental frequency parameters and scaling factors for rotation position encoding for different target context length ranges. The dynamic context extension module is used to temporarily adjust the scaling factor in the rotational position encoding when the length of the input sequence of the hierarchical model is greater than 2048, expand the range supported by the position encoding through interpolation, and realize long sequence inference; and after the prediction is completed, the scaling factor is reset to the system default value and the fundamental frequency parameter of the original rotational position encoding is restored. The structured title tree construction module is used to convert the hierarchical probability distribution of each title text block output by the hierarchical prediction module into the final predicted label, and construct a structured title tree according to the page order and parent-child relationship.

2. The BERT-based academic paper title classification device according to claim 1, characterized in that, The normalized layout feature vector includes six normalized values: the page normalized value of the horizontal coordinate of the top left corner of the text block, the page normalized value of the vertical coordinate of the top left corner of the text block, the ratio of the text block width to the page width, the ratio of the text block height to the page height, the normalized position of the current page number in the full text, and the normalized value of the total number of pages in the document.

3. The BERT-based academic paper title classification device according to claim 1, characterized in that, The feature fusion module is used to format and tokenize the text content of title text blocks identified as Title type according to their natural order of appearance in the document. This is achieved through the following operations: Each consecutive title text is concatenated using the [SEP] delimiter and marked with a [CLS] tag at the beginning to form a unified input sequence; then the sequence is encoded by the Tokenizer of the hierarchical model to generate text word embedding vectors.

4. The BERT-based academic paper title classification device according to claim 1, characterized in that, The application of the YARN algorithm to train the hierarchical model in stages involves the following three stages of training: Phase 1: Target context length range 0-512, rotation position encoding base frequency parameter ROPE_base=10000, scaling factor scale=1.0; Phase 2: Target context length range 512-1024, rotation position encoding base frequency parameter ROPE_base=20000, scaling factor scale=5.0; Phase 3: Target context length range 1024-2048, rotation position encoding base frequency parameter ROPE_base=50000, scaling factor scale=10.

0.

5. The BERT-based academic paper title classification device according to claim 1, characterized in that, The specific process by which the feature fusion module obtains the final input embedding vector that fuses multimodal information is as follows: (1) The 6-dimensional normalized layout features of the normalized layout feature vector are mapped to a 768-dimensional vector through a linear layer, so that they are aligned with the Bert word dimension; (2) Expand the mapped layout feature vector in the sequence dimension to align it with the embedding vector of each text word; (3) The text word embedding vector and the layout embedding vector are concatenated in the feature dimension to form a 1536-dimensional joint feature; (4) Compress it back to 768 dimensions through another linear layer to generate an enhanced lexical representation, which is the final input embedding vector that integrates multimodal information.

6. The BERT-based academic paper title classification device according to claim 1, characterized in that, The text information of the title text block of type Title includes the text content, the position information of the text block on the page, and the page information where the text block is located.

7. A BERT-based method for classifying academic paper titles, characterized in that, The method includes the following steps: S1: Parse PDF academic papers and extract the set of continuous text blocks that make up the title chain and their layout attributes; S2: Connect all text information of title text blocks identified as Title type in the entire academic paper into a single long text sequence according to their natural order of appearance in the document, using the [SEP] delimiter, and add the [CLS] mark at the beginning of the sequence to form the single input sequence of the BERT model; S3: Generate a corresponding normalized layout feature vector for each title text block; perform linear projection on the normalized layout feature vector, and copy the projected normalized layout feature vector to the embedding positions corresponding to all tokens after word segmentation of the title text block; then concatenate the copied normalized layout feature vector with the text embedding vector of the corresponding token in the feature dimension; reduce the dimension of the concatenated high-dimensional features through a linear projection layer and fuse them to obtain the final input embedding vector that fuses multimodal information. S4: A hierarchical model is constructed by adding no less than two linear layers after the output layer of the Bert model. The input of the hierarchical model is an embedding vector that integrates multimodal information, and the output is the hierarchical category of each title text block. The hierarchical model is trained in stages using the YARN algorithm. Different fundamental frequency parameters and scaling factors for rotation position encoding are set for different target context length ranges to obtain the trained hierarchical model. S5: Use the trained hierarchical model for inference to obtain the hierarchical probability distribution of each title text block output by the trained hierarchical model; and when the length of the input sequence is greater than 2048, temporarily adjust the scaling factor in the rotation position encoding, expand the position encoding support range through interpolation, realize long sequence inference, and after the prediction is completed, reset the scaling factor to the system default value and restore the fundamental frequency parameter of the original rotation position encoding. S6: Transform the hierarchical probability distribution of each title text block into the final predicted label, and construct a structured title tree according to the page order and parent-child relationship.

8. The BERT-based academic paper title classification method according to claim 7, characterized in that, A hierarchical model is constructed by adding three linear layers after the output layer of the BERT model.

9. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs that, when executed by the electronic device, cause the electronic device to implement the Bert-based academic paper title classification method as described in claim 7.

10. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the Bert-based academic paper title classification method as described in claim 7.

Images