Scientific and technological paper PDF intelligent analysis method based on multi-task learning

Abstract

The invention provides a science and technology paper PDF intelligent analysis method based on multi-task learning, and relates to the technical field of paper content mining, and the method comprises the steps: covering a picture position, a table position and an inter-line formula position, and obtaining a covered science and technology paper page picture; carrying out OCR (Optical Character Recognition) processing to obtain text contents and text positions; performing preprocessing to obtain text position features and text content features; and processing through a deep multi-task analysis model, and simultaneously obtaining a reading sequence recognition result, a text structure function recognition result, a paragraph detection result and a reference extraction recognition result. The method solves the technical problems that when a traditional text extraction method is used for processing a PDF format, complex typesetting elements such as formulas and charts cannot be accurately recognized generally, the layout difference of different types of content in a page is large, accurate structured recognition and accurate layout analysis are difficult to achieve through the traditional method, and the document analysis effect is poor.

Description

Technical Field

[0001] The present invention relates to the technical field of paper content mining, and in particular to a scientific paper PDF intelligent parsing method based on multi-task learning. Background Art

[0002] Currently, the object of content mining in scientific papers has gradually shifted from title-type information to full-text information. The fine-grained analysis of full-text content is crucial for many downstream tasks such as domain knowledge base construction and innovation evaluation.

[0003] The purpose of PDF content parsing of scientific papers is to extract the text content from PDF files and convert it into text storage formats such as JSON. It is necessary to identify the reading order of the page text, convert it into a one-dimensional linear structure, extract the pictures, tables, and formulas on the page, and identify the structural functions of the text, such as title, abstract, main text, captions, references, etc.

[0004] Existing general PDF parsing methods usually only focus on restoring the reading order of text in PDF and removing content such as headers and footers, which cannot meet the content structuring required by academic PDF parsing. Among the existing specialized academic PDF parsing methods, such as the GROBID project, the parsing effect on English journal PDFs is very good, but the parsing effect on Chinese academic PDFs is poor, and there are differences in the content layout of Chinese and English paper PDF files. Summary of the invention

[0005] This application provides a method for intelligent parsing of scientific paper PDFs based on multi-task learning, aiming to solve the technical problem that traditional text extraction methods are usually unable to accurately identify complex layout elements such as formulas and charts when processing PDF formats, and the layout of different types of content on the page varies greatly. Traditional methods find it difficult to achieve accurate structured recognition and precise layout analysis, resulting in poor document parsing effects.

[0006] The present application discloses a method for intelligent parsing of a scientific paper PDF based on multi-task learning, the method comprising: performing layout analysis on a scientific paper PDF file to obtain images and image positions, tables and table positions, interline formulas and interline formula positions, masking the image positions, the table positions and the interline formula positions to obtain masked scientific paper page images; performing OCR processing on the masked scientific paper page images to obtain text content and text positions; preprocessing the text content and text positions to obtain text position features and text content features; processing the text position features and the text content features through a deep multi-task parsing model to simultaneously obtain reading order recognition results, text structure function recognition results, paragraph detection results and citation extraction recognition results.

[0007] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0008] By analyzing the layout of the PDF files of scientific papers, we can accurately obtain the positions of images and images, tables and tables, and inter-line formulas and inter-line formula positions, and mask them. This preprocessing step reduces the interference of images, tables, and inter-line formulas on the text parsing process, thereby improving the accuracy of subsequent OCR processing and text content parsing; by combining the semantic features of the text with various positional features, we fully utilize the layout information of the PDF document and improve the accuracy of text structure function recognition, especially the recognition accuracy of image captions and table captions; by using regression functions to generate sorting scores for reading order detection, compared with the traditional pair-wise sorting method, the reasoning speed and accuracy of the text reading order recognition task are improved; by adopting a multi-task learning framework, the constructed parsing model can simultaneously perform four tasks: reading order detection, text function recognition, paragraph boundary recognition, and citation extraction, which not only improves the overall parsing efficiency, but also promotes each task to promote each other, thereby improving the overall parsing performance.

[0009] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A flowchart of a method for intelligent PDF parsing of scientific papers based on multi-task learning is provided for an embodiment of the present application.

[0011] Figure 2 A schematic flow chart of obtaining text position features and text content features in a method for intelligent parsing of a scientific paper PDF based on multi-task learning is provided for an embodiment of the present application.

[0012] Figure 3 A schematic diagram of the process of obtaining masked scientific paper page images in a scientific paper PDF intelligent parsing method based on multi-task learning is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0013] The embodiment of the present application provides a method for intelligent parsing of scientific paper PDFs based on multi-task learning, which solves the technical problem that traditional text extraction methods are usually unable to accurately identify complex layout elements such as formulas and charts when processing PDF formats, and the layout of different types of content on the page varies greatly. Traditional methods find it difficult to achieve accurate structured recognition and precise layout analysis, resulting in poor document parsing effects.

[0014] After introducing the basic principles of the present application, various non-limiting implementation methods of the present application will be specifically described below in conjunction with the drawings in the specification.

[0015] like Figure 1 As shown, the embodiment of the present application provides a method for intelligently parsing a scientific paper PDF based on multi-task learning, the method comprising:

[0016] Step S100: Analyze the layout of the scientific paper PDF file to obtain pictures and picture positions, tables and table positions, and interline formulas and interline formula positions, and mask the picture positions, table positions, and interline formula positions to obtain masked scientific paper page pictures;

[0017] Use the PDF to Image tool to convert each page in the scientific paper PDF file into an image to generate the first scientific paper page image. Regardless of whether the PDF file is a scanned type or a text-selectable type, the PDF page is uniformly converted into a pure image format first. This is because when the existing PDF parsing library (such as PDFminer) is directly used to process the text-selectable PDF file, the proposed text content is prone to garbled characters, and the images and formulas are not fully extracted, and it is even more impossible to process when encountering a scanned PDF file. Use the layout analysis model to analyze the converted page image, identify the location of the three types of elements, including pictures, tables, and interline formulas, and crop the image according to the location of the obtained pictures, tables, and interline formulas. Extract the area containing these three types of elements and save it as a separate image. For the cropped image area, use a preset color, such as gray or white, to cover the area of ​​the picture and table to obtain a masked scientific paper page image. At this time, the image only retains text and formula information.

[0018] Step S200: performing OCR processing on the masked scientific paper page image to obtain text content and text position;

[0019] The OCR model is used to process the masked scientific paper page image. The OCR model extracts the text content by analyzing the characters and text arrangement in the image. Specifically, the OCR model converts the characters in the image into machine-readable text content through text recognition, and provides the corresponding coordinate position of each text area, that is, the text box position, which provides a basis for further text analysis.

[0020] Step S300: preprocessing the text content and text position to obtain text position features and text content features;

[0021] Preprocess the text content and text position to obtain text character features, 1D position features, 2D position features, and page relative position features; preprocess special elements such as pictures, tables, and formulas obtained from layout analysis to obtain special element tokens and special element position features. Combine the extracted text character features and special element tokens to form a comprehensive content vector feature, which contains the content information of the text and special elements. The model will use these vectors for subsequent parsing tasks.

[0022] In addition to content vector features, position features are also very important. The spatial distribution of text and formulas on the page is very helpful for understanding the structure of the document. Therefore, the various extracted position features, including the 1D and 2D positions of text and the spatial features of formulas, are merged into a content position feature for subsequent model processing.

[0023] Step S400: Processing the text position features and the text content features through a deep multi-task parsing model, and simultaneously obtaining a reading order recognition result, a text structure function recognition result, a paragraph detection result, and a reference extraction recognition result.

[0024] The deep multi-task parsing model includes a reading order recognition task channel, a paragraph detection task channel, a structural function recognition task channel and a reference extraction task channel. Each channel is responsible for processing a certain aspect of the document. Each task channel will output specific results based on content vector features and position features. Each task channel has its own independent loss function, which is used for training and optimization. The key to multi-task learning is to improve the generalization ability of the model by sharing part of the network structure and utilizing the relationship between multiple tasks.

[0025] According to the reading order recognition task channel, the reading order recognition result is output, that is, the ranking score or label of each text line. The model will output the correct order of the text lines based on these labels; according to the paragraph detection task channel, the paragraph detection result is output, which is a binary classification result (1 or 0), where 1 indicates that the text line is the beginning of a paragraph and 0 indicates that it is not; according to the structural function recognition task channel, the text structure recognition result is output, indicating that the task recognizes different structural functions of the text, such as title, text, formula, reference, etc.; according to the citation extraction task channel, the citation extraction recognition result is output, that is, the citation information in the text, such as citations, formula citations, chart citations, etc.

[0026] Furthermore, if Figure 2 As shown, the text content and text position are preprocessed to obtain text position features and text content features, including:

[0027] Step S310: pre-processing the text content and text position to obtain text character features, 1D position features, 2D position features and page relative position features, and marking text tokens for the picture, the table and the inter-row formula;

[0028] Step S320: adding the text character feature and the text token into the text content feature;

[0029] Step S330: adding the 1D position feature, the 2D position feature and the page relative position feature into the text position feature.

[0030] Character information is extracted from each line of text output by OCR, and the text line content is encoded to obtain a vector representation of each character, namely, the text character feature.

[0031] The 1D position feature refers to the position of the text line in the PDF page. The position of each line of text can be regarded as the order of the line on the page, that is, the arrangement order from top to bottom. The content of the text line is encoded starting from 0 from the first position to generate the relative position of the text in the page. Usually, a numerical value is assigned to each text line in sequence, arranged from top to bottom, representing its sequence number in the page.

[0032] The 2D position feature refers to the spatial position information of the text line. Considering its horizontal and vertical coordinates on the page, the position of each line of text (such as the coordinates of the beginning and end of the line) is encoded. Specifically, the coordinates of the upper left corner and lower right corner of the text line position are normalized, and the width and height of the page are also normalized to a preset interval. For example, the coordinate value is mapped to between 0 and 1, or mapped to a fixed interval, such as [1,1024].

[0033] The page relative position feature refers to the overall position of a text line relative to a page. This feature takes into account the position of the text line in the page and the position of the page in the entire document, calculates the ratio of the page number of the page where the text line is located to the total number of pages in the entire document, and scales the ratio to a preset range.

[0034] Elements such as pictures, tables, and inline formulas need to be specially marked during the document parsing process. These elements do not directly contain processable text content, but are represented in the form of special tokens. These tokens serve as placeholders in the subsequent processing stage to help the model identify and distinguish these non-text elements.

[0035] The text character features and text tokens are combined to form a complete text content feature, which includes the semantic features of the text itself (such as text embedding) and the identification features of images, tables, etc., which is convenient for subsequent processing.

[0036] The 1D position feature, 2D position feature and page relative position feature are combined to form a complete text position feature. This feature helps the model to more accurately understand the spatial distribution of text lines on the page and their sequential relationship.

[0037] Furthermore, if Figure 3 As shown, obtain the covered scientific paper page image, including:

[0038] Step S110: receiving a scientific paper PDF file and converting it into an initial scientific paper page image;

[0039] Step S120: Analyze the initial scientific paper page image through the layout analysis model to obtain the image position, table position and inter-line formula position;

[0040] Step S130: cropping the initial scientific paper page image according to the image position, the table position and the inter-row formula position to obtain the image, the table and the inter-row formula;

[0041] Step S140: masking the initial scientific paper page image with a preset color according to the image position, the table position and the inter-row formula position to obtain a masked scientific paper page image.

[0042] Receive and load the scientific paper PDF file to be parsed. At this time, the file is still in a mixed format of text and images, and may contain text, formulas, pictures, tables, etc. Use software tools or scripts to convert each page into a high-quality image to obtain the initial scientific paper page image. This step converts PDF files into image formats for subsequent processing.

[0043] Layout analysis aims to identify and extract various elements from document images, such as text, images, tables, titles, etc., and understand their spatial and logical relationships. This technology uses a variety of methods, including rule-based, machine learning, and deep learning, to segment document images into different semantic regions, and classify and extract relationships between these regions. Layout analysis has a wide range of applications, from document digitization and information extraction to document understanding and robotic process automation. It plays a vital role and provides the basis for efficient processing and understanding of document information.

[0044] The layout analysis model is an algorithm used to extract and identify different elements from page images. The purpose is to divide the page into multiple areas and classify them according to the content of these areas. For example, deep learning methods such as convolutional neural networks (CNN) or regional convolutional neural networks (R-CNN) can efficiently and accurately extract elements from images and classify them. After model analysis, the image position, table position and the formula position are output, and these positions are usually represented in the form of rectangular boxes. Among them, pictures refer to images, charts, illustrations, etc. in the page. These pictures usually have specific identification, such as image resolution, color mode, etc.; tables refer to table elements in the page. These tables usually contain data, row and column structure and other information.

[0045] The image of the initial scientific paper page is cropped according to the image position, table position and inter-row formula position. For the cropping of the formula, it can be precisely cropped according to the rectangular area where the formula is located to ensure that only the formula content is retained. Similarly, for tables and pictures, they are also cropped according to their positions to avoid other unnecessary areas affecting subsequent processing. By cropping, the area containing the formula, picture or table is obtained.

[0046] In order to prevent the cropped area from affecting OCR processing, a preset color is selected to cover the cropped area, such as gray, white or other background colors. The purpose of this is to clear the unnecessary image area and keep only the text area, thereby avoiding interference with OCR recognition. After the covering is completed, a masked scientific paper page image is generated, which includes the covered area, while the text part remains intact.

[0047] Furthermore, the text content and text position are preprocessed to obtain text character features, 1D position features, 2D position features and page relative position features, and text tokens are marked for the pictures, the tables and the inter-line formulas, including:

[0048] Step S311: obtaining text line content and text line position according to the text content and the text position;

[0049] Step S312: Encode the text line content to obtain text character features;

[0050] Step S313: according to the content of the text line, start encoding from the first bit 0 to obtain the 1D position feature;

[0051] Step S314: performing 2D position encoding on the text line position to obtain the 2D position feature, including:

[0052] Step S3141: normalizing the upper left corner coordinates and the lower right corner coordinates of the text line position, and the width and height of the page to which the text line position belongs to a preset range to obtain the 2D position feature;

[0053] Step S315: Encoding the text line position and page position to obtain the page relative position feature, including:

[0054] Step S3151: calculating the ratio of the text line page number at the text line position to the total number of pages, and scaling the ratio to a preset interval to obtain the page relative position feature;

[0055] Step S316: marking the text tokens for the pictures, the tables and the inter-row formulas.

[0056] Use the OCR model to detect text on the PDF page and obtain all the text line information in the page, including the content and position of each line. Extract the character information in the OCR output by text line to form independent text line content. For example, different paragraphs and titles in a page will be divided into multiple lines, and the content of each line is a continuous character sequence; obtain the position information of the text line in the page. The position information of each line includes the upper left corner coordinates and the lower right corner coordinates. The text line position can represent the spatial range of the text in the page, providing a basis for subsequent position information encoding.

[0057] Convert the characters of the text content into feature vectors. First, segment the strings of each text line. WordPiece or SentencePiece can be used for segmentation. During the segmentation process, it is necessary to ensure that each number is separated separately. This is especially important for the subsequent reference extraction task. The features of each character can be represented by character embedding. The obtained text character features can help the model understand the semantic information of the text.

[0058] 1D position features are mainly used to represent the relative position of each character in the text line it is in. For each line of text, starting from the first character of the line, the characters are encoded in the order of the characters in the line, and each character is given a unique number (starting from 0). This number represents the relative position of the character in the text line. For example, the 1D position feature of the first character in a line of text is 0, the second character is 1, and so on. 1D position features help the model understand the order and structure of characters within a text line.

[0059] 2D position features are used to represent the two-dimensional spatial position of a text line on a page. For each line of text, the coordinates of the upper left and lower right corners of its bounding box on the page are extracted. These coordinates reflect the horizontal and vertical positions of the text line on the page. In order to eliminate the impact of page size differences on the features, the page size is first scaled to a fixed layout interval, such as [1200, 1800], so that one side of the page is the same as the length or width of the interval, and then the scaling ratio is calculated. Based on this scaling ratio value, the coordinate values ​​of all text lines, including width and height, are normalized and scaled to a preset interval [1200, 1800]. 2D position features help the model understand the specific position of the text line on the page, for example, whether the text line is in the upper left or lower right corner of the page, whether it is adjacent to other text lines, and the relative distance between them.

[0060] The page relative position feature is designed to represent the relative position of a text line in the entire PDF document. Specifically, for each line of text, we first obtain the page number where the text line is located, and then calculate the ratio of the page number to the total number of pages in the document. For example, if a line of text is located on page 3 of a document, and the document has a total of 10 pages, then the page relative position feature value of the text line is 3 / 10=0.3. This ratio can be regarded as a normalized position information that maps the position of the text line in the document to an interval of [0,1]. The closer the value is to 0, the closer the text line is to the beginning of the document; the closer the value is to 1, the closer the text line is to the end of the document.

[0061] The role of the page relative position feature is: 1) Providing global context information. The page relative position feature provides the position information of the text line in the entire document, which helps the model understand the context of the text. For example, the title usually appears at the beginning of the document, while the reference usually appears at the end of the document. 2) Distinguishing different types of content. The positions of different types of content in the document usually have certain regularities. For example, the abstract usually appears at the beginning of the document, while the conclusion usually appears at the end of the document. The page relative position feature can help the model distinguish these different types of content.

[0062] In addition to the construction of text element features, three types of special elements, pictures, tables, and interline formulas, also need to be taken into consideration. By introducing special element tags (tokens) and their position information, the model can distinguish different types of content and understand the spatial relationship between them and the text. For example, the character tag 21000 represents a table, 21001 represents a picture on the page, and 21002 represents an interline formula on the page. The information of special elements is crucial for the reading order recognition task. The cross-column tables and pictures on the page can change the original normal text reading order. After merging and splicing the tokens of special elements with the text character tags, a comprehensive content vector feature is formed, which includes the semantic information of the text and the structural information of the formula.

[0063] Different types of position features, such as the 1D position feature, the 2D position feature, the page relative position feature and the formula position feature, are combined into a unified content position feature. The splicing method can also be used to merge these position features. Through these features, the model can learn the spatial layout information of text and formulas in the page, including their relative position, arrangement, whether they are in the same paragraph, etc.

[0064] Furthermore, the deep multi-task parsing model includes:

[0065] A shared encoder for extracting representations of input features; a reading order recognition task channel, which constructs a regression function based on a multi-layer fully connected network to predict the ranking score of a text line; a text structure function recognition task channel, which constructs a multi-classifier based on a multi-layer fully connected network to predict the structural function type of a text line; a paragraph detection task channel, which constructs a binary classifier based on a multi-layer fully connected network to predict whether a text line is a paragraph start sentence; a reference extraction task channel, which is constructed based on a sequence annotation header to predict the reference position in the text.

[0066] Multi-task learning is a machine learning method that aims to improve the generalization ability of the model by learning multiple related tasks at the same time. Unlike single-task learning, which trains models separately, multi-task learning trains one model to complete multiple tasks. Specifically, the model uses a pre-trained causal language model as a shared text representation encoder. This embodiment uses the parameters of the pre-trained LayoutLMv3 as the initialization parameters of the model. The shared encoder is the core component of the model, which is used to uniformly encode the input multimodal features and extract global and local representations. The purpose of the shared encoder is to share representations between different tasks to enhance the collaborative learning ability between tasks and reduce redundant calculations.

[0067] The reading order recognition task channel is used to predict the reading order of the text. Reading order recognition refers to arranging the text in the PDF page into a one-dimensional linear sequence according to the normal human reading order. The reading order recognition task channel uses the semantic features and positional features of the text to infer its correct arrangement order. Usually, by analyzing the up-down, left-right position relationship of each text block on the page, the model can determine which text should appear first and which should appear later. Based on these rules, the model will establish a relationship between the text blocks on the page and output the reading order recognition result, which is a text sequence arranged in order. It rearranges the text blocks on the PDF page into a linear sequence to ensure that the text conforms to the order in which humans read documents. For the captions of images and tables, we distinguish them from the reading order of the main text. We believe that images and tables are generally presented as floating elements in the main text. Therefore, the model predicts the reading order of the main text and the caption text separately. The reading order of the caption text includes the captions of images and tables. When segmenting, the results of paragraph detection and structural function recognition are used together. If a text line is recognized as opening a new paragraph, it means that the text line is the caption text of the next element.

[0068] Structural function recognition refers to the classification of each line of text into the following 10 categories: table captions, image captions, text paragraphs, titles (including major titles and minor titles), abstracts, author names, keywords, institutions, references, and other auxiliary information (including headers, footers, copyright information, etc.). The structural function recognition channel uses deep learning models such as BERT, LSTM, CNN, etc., mainly using semantic features and auxiliary position features to determine which of the above 10 categories the text line belongs to. For each input text line, the model will output a category label to obtain the text structure recognition result, indicating which part of the document the text belongs to, such as title, text paragraph, table caption, etc.

[0069] Paragraph detection refers to correctly dividing the text into lines. A title belongs to a separate line, a paragraph belongs to a separate line, a reference belongs to a line, the abstract content belongs to a separate line, and the keyword content belongs to a separate line. The paragraph detection model infers which texts belong to the same paragraph and which texts should be divided into different paragraphs based on the spatial distance between text blocks, line spacing, and typesetting structure. Usually, there will be obvious intervals between paragraphs, and there are certain semantic differences between titles and text. The model distinguishes titles from text, text from references, etc. based on positional features. For example, titles are usually brief, while text is conventionally typeset, and references are usually located at the bottom of the page and separated from the text. After analysis, the paragraph detection result is output. This is a tokenized result that indicates whether each text block belongs to a paragraph. The paragraph detection task channel can not only identify text paragraphs, but also distinguish different content types such as titles, paragraphs, and references.

[0070] The input of the multi-layer fully connected neural network corresponding to the reading order recognition task, paragraph recognition task, and structural function recognition task is the first character of each line of text. The first character can be a letter, a number, or a separately added CLS character. The first character integrates all the information of the text line and can achieve the output of reading order recognition, paragraph detection, and structural function recognition tasks based on the features of the first character. In the citation extraction task channel, the encoding layer feature representation of all characters in the complete text line is input, not just the first character. The model uses all characters in the text line to classify and predict citation marks.

[0071] Citation extraction refers to the identification of citation locations in the PDF text of a document. For example, in the text "Zhang San et al. [1] proposed ...", [1] is the citation marker, and in the text "Maning (2018) studied the chemical properties of deep crustal fluids and the mechanisms that carry these fluids", Maning (2018) is the citation marker. The citation extraction task channel uses a deep learning model to process input features. The model analyzes the structure of the text and identifies possible citation markers, such as "[1]", "author, year", etc. Citation markers usually have fixed formats, so the model extracts citation information by identifying these formats and outputs the citation extraction recognition result, which is the identified citation information.

[0072] Furthermore, the deep multi-task parsing model is obtained by training with a multi-task loss function, and the multi-task loss function is:

[0073] Total loss =w1*Loss1+w2*Loss2+w3*Loss3+w4*Loss4

[0074] Among them, Total lossRepresents the overall loss value of multi-task parsing, w1, w2, w3, w4 represent the weights of each loss value, Loss1 represents the ListMLE loss function of the reading order recognition task, Loss2 represents the cross entropy loss function of the text structure function recognition task, Loss3 represents the cross entropy loss function of the paragraph detection task, and Loss4 represents the cross entropy loss function of the citation extraction task.

[0075] Specifically, the formula of the multi-task loss function is as follows:

[0076] Total loss =w1*Loss1+w2*Loss2+w3*Loss3+w4*Loss4;

[0077] Among them, Loss2, Loss3 and Loss4 are cross entropy loss functions, and Loss1 is the ListMLE loss function. The ListMLE loss function is a loss function used for ranking learning. It directly optimizes the likelihood probability of the entire sorted list. This method is based on the Plackett-Luce model, which regards each sorted list as a permutation and tries to maximize the observed permutation probability. In practical applications, ListMLE minimizes the loss by calculating the negative log-likelihood. Its goal is to make the ranking generated by the model as close to the ideal ranking as possible. Compared with the point-by-point and pair-by-pair methods, ListMLE takes into account the relative order relationship of the entire document sequence, so it can usually achieve better ranking results.

[0078] This method uses a multi-task learning method to train the constructed parsing model, and optimizes and learns the four tasks simultaneously. During the model training process, a variety of methods are used to set the four loss weights w1, w2, w3, and w4, including manually setting fixed weights, DWA and GradNorm methods to adaptively adjust and learn weights. According to the multi-task parsing loss function, the initial reading order recognition task channel, the initial paragraph detection task channel, the initial structure function recognition task channel, and the initial reference extraction task channel are trained to obtain the deep multi-task parsing model.

[0079] Furthermore, the output of the reading order recognition task channel is a sorting score of a text line, wherein the reading order recognition task includes a paragraph text sorting task and a caption text sorting task, the losses of the paragraph text sorting task and the caption text sorting task are calculated separately, and the loss of the reading order recognition task is the sum of the loss of the paragraph text sorting task and the caption text sorting task.

[0080] The main output of the reading order recognition task channel is the sorting score of each line of text. The sorting score is a continuous value that represents the reading order of the text line in the page. The higher the score, the higher the priority. Among them, the reading order recognition task includes the paragraph text sorting task and the caption text sorting task. The paragraph text sorting task is responsible for sorting the text lines of the body paragraph. The goal is to ensure that the text lines within the paragraph are arranged in the correct context order, thereby restoring the natural reading order; the caption text sorting task is responsible for sorting the captions of pictures and tables. The goal is to ensure that the caption text is correctly aligned with the associated picture or table and sorted in the correct reading order.

[0081] The paragraph text sorting task and the caption text sorting task each use the ListMLE sorting loss function to directly optimize the probability of the sorted list. By maximizing the ideal sorting, it ensures that the sorting result of the paragraph text is close to the actual reading order. The total loss of the final reading order recognition task is the sum of the paragraph text sorting task loss and the caption text sorting task loss. This independent loss calculation method ensures that the model can optimize the sorting quality of paragraph text and caption text separately, and at the same time can be jointly optimized through the total loss.

[0082] Furthermore, the output of the text structure function recognition task channel is the structural function type of the text line, and the predicted output includes at least 10 categories, the 10 categories include: table captions, picture captions, paragraph text, titles, abstracts, paper author names, keywords, institutions, references and no type, wherein the title includes the Chinese version of the paper title and chapter title, and the no type includes content from other articles, English version titles, English version abstracts, headers, and footers.

[0083] The goal of the text structure function recognition task channel is to predict the structural function category to which each line of text belongs, and the output is the specific function type label of each line of text. There are 10 categories in total, among which table captions represent explanatory text related to the table, which usually appears immediately after the table; picture captions represent explanatory text related to the picture, which usually appears immediately after the picture; paragraph text is an ordinary paragraph in the text, which is usually the main content of the article; titles include main titles and subtitles, such as chapter titles; abstracts are the abstract part of the paper, which is usually after the title and author information and is used to summarize the content of the paper; the name of the author of the paper is the name information of the author of the paper, which is usually located after the title or abstract; keywords refer to the keyword part of the paper, which is usually used to mark the core content or field of the research; institutions refer to the institution information to which the author belongs, which is usually located after the author's name; references refer to the list of references cited in the paper, which is usually located at the end of the article; untyped contains other text that does not belong to the above categories, including text from other article content, English version title, English version abstract, headers and footers, etc.

[0084] Furthermore, the output of the paragraph detection task channel is 1 or 0, where 1 represents that the text line is the starting sentence of a paragraph, and 0 represents that the text line is not the starting sentence of a paragraph.

[0085] The paragraph detection task channel uses a multi-layer fully connected network to build a binary classifier to determine the paragraph structure of the text. After obtaining the representation of each character in the encoding layer, only the first character of each text line is input into the binary classifier. The classifier output result of 0 indicates that the text line does not start a new paragraph, and the result of 1 indicates that the text line is the starting sentence of a paragraph and will start a new paragraph.

[0086] Furthermore, after obtaining the text character features, it also includes:

[0087] Step S460: Obtain the first character of the text line according to the text character feature;

[0088] Step S470: the text character feature is input into the first character of the text line in the reading order recognition task channel, the paragraph detection task channel, and the text structure function recognition task channel;

[0089] Step S480: the text character features are input into the reference extraction task channel to complete the text character features.

[0090] For each line of text, the first character of the line is extracted according to the character features to obtain the first character of the text line. The first character can be a letter, number or other symbol, depending on the format of the text. The first character provides information about the reading order, especially in documents with multi-column or multi-paragraph layouts. The first character helps determine the starting position of the text block.

[0091] The first character of a text line is input into the reading order recognition task channel, the paragraph detection task channel, and the text structure function recognition task channel. In the reading order recognition task channel, the first character helps determine the correct order of the text. The first character provides the starting mark of the text line and helps the model infer the relative order of each text line in the page; in the paragraph detection task channel, the first character helps identify the starting sentence of the paragraph. Usually, the first line of a paragraph has a unique layout (such as the first line indentation). The extraction of the first character can help the model identify paragraph separation; in the text structure function recognition task channel, the first character helps the model understand the structure of the text. For example, titles usually start with specific characters (such as capital letters), and body paragraphs usually have different layouts. The first character can provide these clues. Each task channel combines the first character of a text line with other text features for task-specific processing.

[0092] The complete text character features are input into the quotation extraction task channel. In the quotation extraction task channel, the complete text character features are input instead of just the first character. The model uses the text content extracted by OCR (including the entire sentence or paragraph) and combines the character features of the text to identify quotation marks.

[0093] In summary, the method for intelligently parsing a scientific paper PDF based on multi-task learning provided in the embodiment of the present application has the following technical effects:

[0094] By analyzing the layout of the PDF files of scientific papers, we can accurately obtain the positions of images and images, tables and tables, and inter-line formulas and inter-line formula positions, and mask them. This preprocessing step reduces the interference of images, tables, and inter-line formulas on the text parsing process, thereby improving the accuracy of subsequent OCR processing and text content parsing; by combining the semantic features of the text with various positional features, we fully utilize the layout information of the PDF document and improve the accuracy of text structure function recognition, especially the recognition accuracy of image captions and table captions; by using regression functions to generate sorting scores for reading order detection, compared with the traditional pair-wise sorting method, the reasoning speed and accuracy of the text reading order recognition task are improved; by adopting a multi-task learning framework, the constructed parsing model can simultaneously perform four tasks: reading order detection, text function recognition, paragraph boundary recognition, and citation extraction, which not only improves the overall parsing efficiency, but also promotes each task to promote each other, thereby improving the overall parsing performance.

[0095] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for intelligent PDF parsing of scientific papers based on multi-task learning, characterized in that: The method comprises: Performing layout analysis on the PDF file of scientific papers to obtain pictures and picture positions, tables and table positions, interline formulas and interline formula positions, masking the picture positions, table positions and interline formula positions, and obtaining masked scientific paper page pictures; Performing OCR processing on the masked scientific paper page image to obtain text content and text position; Preprocessing the text content and text position to obtain text position features and text content features; The text position features and the text content features are processed through a deep multi-task parsing model, and a reading order recognition result, a text structure function recognition result, a paragraph detection result and a citation extraction recognition result are obtained at the same time.

2. A method for intelligently parsing a scientific paper PDF based on multi-task learning as claimed in claim 1, characterized in that: Preprocessing the text content and text position to obtain text position features and text content features includes: Preprocess the text content and text position to obtain text character features, 1D position features, 2D position features and page relative position features, and mark text tokens for the picture, the table and the inter-row formula; Adding the text character feature and the text token into the text content feature; The 1D position feature, the 2D position feature and the page relative position feature are added to the text position feature.

3. A method for intelligently parsing PDF of scientific papers based on multi-task learning as claimed in claim 1, characterized in that: Perform layout analysis on PDF files of scientific papers to obtain images of covered pages of scientific papers, including: Receive scientific paper PDF files and convert them into initial scientific paper page images; Analyze the initial scientific paper page image through the layout analysis model to obtain the image position, table position and inter-row formula position; The initial scientific paper page image is cropped according to the image position, the table position and the inter-row formula position to obtain the image, the table and the inter-row formula; According to the image position, the table position and the inter-row formula position, the initial scientific paper page image is masked with a preset color to obtain a masked scientific paper page image.

4. A method for intelligently parsing a scientific paper PDF based on multi-task learning as claimed in claim 2, characterized in that: Preprocessing the text content and text position to obtain text character features, 1D position features, 2D position features, and page relative position features, and marking text tokens for the picture, the table, and the inter-line formula, including: According to the text content and the text position, obtain text line content and text line position; Encoding the text line content to obtain text character features; According to the content of the text line, encoding is started from the first position 0 to obtain the 1D position feature; Performing 2D position encoding on the text line position to obtain the 2D position feature includes: Normalizing the upper left corner coordinates and the lower right corner coordinates of the text line position, and the width and height of the page to which the text line position belongs to a preset range to obtain the 2D position feature; Encoding the text line position and page position to obtain the page relative position feature includes: Calculating the ratio of the text line page number at the text line position to the total number of pages, and scaling the ratio to a preset interval to obtain the page relative position feature; The text token is identified for the picture, the table, and the in-line formula.

5. The method for intelligently parsing a scientific paper PDF based on multi-task learning as claimed in claim 1, characterized in that: The deep multi-task parsing model includes: A shared encoder for extracting representations of input features; A reading order recognition task channel, wherein the reading order recognition task channel constructs a regression function based on a multi-layer fully connected network to predict the ranking score of the text line; A text structure function recognition task channel, wherein the text structure function recognition task channel builds multiple classifiers based on a multi-layer fully connected network to predict the structure function type of a text line; A paragraph detection task channel, wherein the paragraph detection task channel constructs a binary classifier based on a multi-layer fully connected network to predict whether a text line is a paragraph start sentence; The citation extraction task channel is built based on the sequence annotation header and is used to predict the citation position in the text.

6. A method for intelligently parsing a scientific paper PDF based on multi-task learning as claimed in claim 5, characterized in that: The deep multi-task parsing model is obtained by training with a multi-task loss function, where the multi-task loss function is: <h2 style=";text-align:left;direction:ltr">Total<h2 style=";text-align:left;direction:ltr"> loss <h2 style=";text-align:left;direction:ltr"> = w1*Loss1+w2*Loss2+w3*Loss3+w4*Loss4 Among them, Total loss Represents the overall loss value of multi-task parsing, w1, w2, w3, w4 represent the weights of each loss value, Loss1 represents the ListMLE loss function of the reading order recognition task, Loss2 represents the cross entropy loss function of the text structure function recognition task, Loss3 represents the cross entropy loss function of the paragraph detection task, and Loss4 represents the cross entropy loss function of the citation extraction task.

7. A method for intelligently parsing a scientific paper PDF based on multi-task learning as claimed in claim 5, characterized in that: The output of the reading order recognition task channel is the sorting score of the text line, wherein the reading order recognition task includes a paragraph text sorting task and a caption text sorting task, the losses of the paragraph text sorting task and the caption text sorting task are calculated separately, and the loss of the reading order recognition task is the sum of the loss of the paragraph text sorting task and the caption text sorting task.

8. A method for intelligently parsing a scientific paper PDF based on multi-task learning as claimed in claim 5, characterized in that: The output of the text structure function recognition task channel is the structural function type of the text line, and the predicted output includes at least 10 categories, the 10 categories include: table captions, picture captions, paragraph text, titles, abstracts, paper author names, keywords, institutions, references and no type, wherein the title includes the Chinese version of the paper title and chapter title, and the no type includes content from other articles, English version titles, English version abstracts, headers, and footers.

9. A method for intelligently parsing a scientific paper PDF based on multi-task learning as claimed in claim 5, characterized in that: The output of the paragraph detection task channel is 1 or 0. When it is 1, it represents that the text line is the starting sentence of a paragraph, and when it is 0, it represents that the text line is not the starting sentence of a paragraph.

10. A method for intelligently parsing a scientific paper PDF based on multi-task learning as claimed in claim 4, characterized in that: The step of obtaining the text character features further includes: According to the text character feature, obtain the first character of the text line; The text character feature is input into the first character of the text line in the reading order recognition task channel, the paragraph detection task channel, and the text structure function recognition task channel; The text character features are input into the reference extraction task channel to complete the text character features.

Images