Periodical directory extraction and integration method based on multi-modal large model

Abstract

The invention discloses a multi-modal large model-based journal catalogue extraction and integration method, which comprises the following steps of: firstly, analyzing a PDF (Portable Document Format) by adopting a multi-modal large model, extracting page numbers, textbox coordinates and contents, performing blank character unification and non-core character filtering processing, checking and counting catalogue feature lines through a multi-dimensional rule, and performing multi-dimensional classification on the catalogue feature lines; positioning a directory page range by using a double-threshold judgment mechanism; combined filtering denoising, self-adaptive threshold binarization and morphological post-processing are carried out on the directory page grey-scale map, and a high-quality binarized map is generated; cutting according to the textbox coordinates, dividing lines through a vertical projection method, and obtaining a single-line text through OCR (Optical Character Recognition); and screening candidate page number lines based on page number position and length rules, eliminating mismatching by means of large language model semantic verification, and outputting structured data. And finally, aiming at different typesetting, adopting different merging methods, and integrating into a complete directory entry.

Description

Technical Field

[0001] This invention relates to the field of journal content processing technology, and in particular to a method for extracting and integrating journal catalogs based on a multimodal large model. Background Technology

[0002] Journal PDF documents contain a large amount of high-value academic papers, research reports, technical reviews, and other content. The table of contents not only helps readers quickly locate the target article but is also a crucial prerequisite for automated document processing. However, current journal PDF table of contents parsing faces numerous technical bottlenecks. Existing solutions struggle to accurately extract and structure table of contents entries, and to establish a mapping between table of contents page numbers and actual PDF page numbers.

[0003] Therefore, it is necessary to design a journal catalog extraction and integration method based on a multimodal large model to accurately identify the catalog, filter valid page numbers, and merge catalog entries. Summary of the Invention

[0004] This invention addresses the shortcomings of existing technologies by proposing a journal catalog extraction and integration method based on a multimodal large model, enabling efficient and accurate catalog identification, page number filtering, and catalog entry merging.

[0005] The technical solution to achieve the purpose of this invention is as follows:

[0006] The journal catalog extraction and integration method based on a multimodal large model includes the following specific steps:

[0007] A multimodal large model was used to parse the journal PDF document to obtain element information for each page, including page number. Text box coordinates Text content ; text content The text is processed to remove whitespace and non-core characters, and the page numbering rules are validated. A dual-threshold determination mechanism is used to locate the range of page numbers. ;

[0008] Based on the scope of the table of contents Extract the grayscale image of the table of contents from the journal's PDF document and denote it as... ,in The grayscale image of the directory page is subjected to combined filtering, adaptive threshold binarization, and morphological post-processing to obtain a binarized image of the directory page, denoted as . ;

[0009] Based on the text box coordinates Binarized image of the directory page Cropping is performed to obtain the text box image. The text box image is projected using the vertical projection method. Line breaks are used to obtain the coordinates of a single line of text image. Perform OCR recognition on single-line text and fill the corresponding fields with the recognition results. of middle;

[0010] Based on preset rules, page number lines that match the characteristics of table of contents lines are selected from single-line text. A large language model is introduced to perform semantic understanding on the text content corresponding to the selected page number lines in order to eliminate mismatched entries. At the same time, data cleaning is performed to output structured JSON results.

[0011] Different methods for merging table of contents entries are selected for different layouts to integrate scattered table of contents lines into complete table of contents paragraph entries;

[0012] Furthermore, a multimodal large model is used to parse the journal PDF documents, recording the total number of records in the journal PDF documents. Page, in Extracted element information from the page, including the page number. Text box coordinates Text content ; the extracted first The text content of the page The process involves unifying whitespace characters and filtering non-core characters. Multi-dimensional rules are used to match the processed text content against category page features. A dual-threshold judgment mechanism is then employed, based on page number. Sort the pages in ascending order, and determine the pages whose characteristic lines are greater than or equal to the first threshold as the starting page of the directory. Then, continue traversing backwards to find the directory page feature line, and determine the end page of the directory based on the size relationship between the directory page feature line and the second threshold. ;

[0013] Furthermore, regarding the extracted first... The text content of the page The code uses regular expressions to match all Unicode whitespace characters, including spaces, tabs, and newlines, and replaces them with single ASCII spaces to eliminate whitespace differences caused by different layouts. Additionally, it uses regular expressions to match and filter all characters except letters, numbers, and common symbols. The processed text content is denoted as... ;

[0014] Furthermore, for text content processed by regular expressions... The system counts feature lines on the directory page based on multi-dimensional rules. First, it filters based on formatting rules; feature lines on the directory page must meet the combination of "significant numbers + text content." Then, it analyzes the text content according to the formatting rules. After initial screening, further filtering is performed based on length rules. Lines with character lengths less than the noise length threshold are considered pure page numbers or noise fragments, while lines with character lengths greater than the non-directory length threshold are considered non-directory text. Text lines with character lengths less than the noise length threshold or greater than the non-directory length threshold are excluded. After validation according to the above format and length rules, the number of lines is counted. The number of all directory page feature lines in the page that meet the rules is denoted as . ;

[0015] Furthermore, considering the formatting patterns of journal table of contents, the starting page typically contains complete entries with a relatively large number of entries; subsequent pages mostly contain second- or third-level entries with a relatively smaller number. A dual-threshold determination mechanism is employed to determine the range of table of contents pages through a two-step progressive determination. First, based on the page numbers extracted from each page... Sort the data in ascending order and iterate through it, counting the number of characteristic rows corresponding to each page in the table of contents. Find the first number of rows that meet the characteristics of the directory page. Pages that are greater than or equal to the first threshold are identified as the directory start page, and their corresponding page numbers are denoted as follows: Then from the beginning of the table of contents Initially, traverse each page in ascending order of page number. If the number of directory page characteristic lines for the current page is greater than or equal to the second threshold, determine that the current page is a subsequent page in the directory. If the number of directory page characteristic lines for two consecutive pages is less than the second threshold, stop traversing and determine the last page that satisfies the condition of having a number of directory page characteristic lines greater than or equal to the second threshold as the end page of the directory, and record its corresponding page number as . After the above steps, the final directory page range is obtained. ;

[0016] Furthermore, based on the scope of the directory pages Collect the grayscale image of the corresponding page's directory and record it as... ,in The grayscale image of the directory page is filtered and denoised, then subjected to adaptive threshold binarization and morphological post-processing to obtain a binarized image of the directory page, denoted as . ;

[0017] Furthermore, regarding the grayscale image of the directory page... The image contains various types of interference, such as scanning noise and Gaussian noise. A combined filtering strategy of Gaussian filtering and median filtering is used for layered denoising to obtain the denoised image. It preserves text details while removing noise;

[0018] Based on the grayscale image of the catalog page The image noise intensity is adaptively selected using different Gaussian and median filter kernels for Gaussian and median filtering; specifically, the grayscale image of the directory page is calculated. global grayscale mean Then calculate the grayscale image of the directory page. gray standard deviation As noise intensity;

[0019] Based on the standard deviation of grayscale The size of the Gaussian kernel and the median filter kernel are determined, and the grayscale images of the directory page are processed sequentially according to the selected Gaussian kernel and median filter kernel. Gaussian filtering and median filtering are performed to obtain the denoised directory page diagram. ;

[0020] Furthermore, considering the significant difference in text and background contrast in some grayscale images of the directory pages, adaptive threshold binarization is used to denoise the directory page images and obtain preliminary binarized images. ;

[0021] Denoising the table of contents The image is divided into fixed-size local windows, and the size of the local windows is adaptively adjusted according to the text size in the current denoising catalog page image. For each local window, the average pixel value within that local window is calculated using the aforementioned method for calculating the grayscale mean and grayscale standard deviation. and standard deviation Then, the Sauvola algorithm was used to calculate the pixel threshold within this local window. Then, each pixel within the local window is compared with a pixel threshold. Binarization is performed by comparison; the above method is used to binarize all local windows sequentially, finally obtaining a preliminary binarized image. ;

[0022] Furthermore, to address issues such as broken text strokes, isolated noise residue, background holes, and text edge burrs in the initial binarized image, a combination of morphological dilation and erosion operations was used for precise repair to obtain the binarized image of the directory page. ;

[0023] use The rectangular structural element for the initial binarized image Dilation is performed to fill in the tiny breaks in the text and enhance its continuity; the dilated image is then eroded, and the same rectangular structuring element is used to eliminate burrs and noise at the edges of the text to restore its original outline.

[0024] Connectivity analysis is performed on the image after dilation and erosion, and the area of ​​each background connected region is calculated. Background connected regions with an area of ​​5 pixels or less are considered as hole noise and are filled using a seed filling algorithm to ensure the purity of the background area. At the same time, holes in text regions with an area greater than 5 pixels are preserved to avoid damage to the text structure. After the above morphological post-processing method, the binarized image of the directory page is obtained. ;

[0025] Furthermore, based on the binarized image of the directory page... and text box coordinates Cropping is performed to obtain the text box image. And the text box image is projected using the vertical projection method. Line breaks are used to obtain the coordinates of a single line of text image. Perform OCR recognition on single-line text and fill the corresponding fields with the recognition results. of middle;

[0026] Furthermore, based on the coordinates of the text boxes on each page... Binary image of the corresponding directory page for this page Cropping is performed to obtain the text box image. Then, vertical projection is used to segment the text box image into lines;

[0027] Specifically, for text box images First, calculate its vertical projection curve, and denote the text box image. The width is The height is The vertical projection curve is obtained by counting the total number of black pixels in each row; based on the calculated... Total number of black pixels in a row indexed by row The x-axis represents the total number of black pixels. A vertical projection curve is constructed to reflect the text density of each line, using the ordinate. To eliminate misjudgments caused by local fluctuations in the vertical projection curve, a filter window of size 3 is used to smooth the total number of black pixels in each line, ensuring smoothness while preserving the trend. Valley detection is performed on the smoothed vertical projection curve, and a valley threshold is set as [value missing]. Traversing the smoothed vertical projection curve Filter to meet The consecutive row index intervals are denoted as the trough intervals. ;

[0028] The upper and lower boundaries of a single line of text are determined by separating adjacent trough intervals. For example, for two adjacent trough intervals... and Let the region between these two adjacent trough intervals be called a single line of text, and the upper boundary of this single line of text corresponds to... The lower boundary corresponds to Then, the coordinates of the single-line text image corresponding to the single-line text were determined. ,in This represents the coordinates of the top-left corner of a single-line text area. This indicates the coordinates of the bottom right corner of a single-line text region; OCR recognition technology is used to determine the coordinates of the single-line text image. The function recognizes the single line of text and fills the corresponding text content into the correct field. middle;

[0029] Furthermore, based on the journal table of contents page number rules, page number lines that meet the rules are filtered from single-line text, and the text content of the page number lines is semantically understood through a large language model to exclude mismatched entries and output structured JSON results.

[0030] Specifically, the journal catalog page numbering rules include page number position rules and page number length rules. The page number position rule means that the page number is located at the beginning or end of the text line and there is a clear separator between it and the title text. The page number length rule means that there should be a limit to the character length of the page number, and the page number exceeding the limit is considered an invalid page number. All single-line text in the previous step is filtered through the above two journal catalog page numbering rules, and the text content corresponding to the page number lines that meet the rules is input into the large language model for semantic understanding.

[0031] Furthermore, different table of contents merging methods are selected for different layouts to integrate scattered table of contents lines into complete table of contents paragraph entries, including dynamic distance merging, merging the first line of page number lines downwards, merging the last line of page number lines upwards, and merging according to indentation rules.

[0032] Furthermore, dynamic distance merging first extracts all single-line text information and integrates it into a unified dataset. For each valid table of contents line containing page numbers, an initial merging unit (including index, upper and lower boundaries, and single-line text image coordinates) is constructed. List and text content List); Image coordinates of single-line text in the table of contents in natural reading order. Sorting the list ensures that the processing flow is consistent with the reading logic;

[0033] Then, using the left and right boundaries of the target directory line as a reference, extend 5 pixels to the left and right to form a horizontal effective area; extract candidate single-line text and calculate its vertical distance to the target directory line, set a vertical distance threshold, and sort according to "nearest to farthest + positive distance priority", then remove single-line text that exceeds the threshold; for the filtered candidate single-line text, if it is located between two adjacent directory lines (i.e., it falls within the horizontal effective area of ​​two directory lines), calculate its vertical distance to the two directory lines and assign it to the directory line with the closer distance; if it only falls within the horizontal effective area of ​​one directory line, it is directly assigned to that directory line; at the same time, the image coordinates of the single-line text corresponding to the text line are also calculated. and text content Add to the merged unit corresponding to the directory line, recalculate the overall upper and lower boundaries of the merged unit, and verify the integrity of the information;

[0034] Furthermore, the page number merging process first extracts all single-line text information and constructs a unified dataset to create initial merge entries for valid table of contents lines containing page numbers. It then locates the position of each table of contents line within the full text line list and sorts the initial merge entries according to the natural order of the table of contents lines. Using the table of contents line as the core, it filters single-line text below it that conforms to the coordinate rules and merges them into the corresponding entries. Based on the entry's position in the table of contents (first, middle, or last), it differentiates the attribution logic of surrounding single-line text to ensure the integrity of the merge. Finally, it generates table of contents page number-text paragraph key-value pairs in a unified format.

[0035] Furthermore, the page number merging process first extracts all single-line text information and constructs a unified dataset to create initial merge entries for valid table of contents lines containing page numbers; it then locates the position of each table of contents line in the full text line list and sorts the initial merge entries according to the natural order of the table of contents lines; using the table of contents line as the core, it filters single-line text above it that conforms to the coordinate rules and merges them into the corresponding entries; based on the position of the entry in the table of contents (first, middle, last), it differentiates the belonging logic of the surrounding single-line text to ensure the integrity of the merge; subsequently, it generates table of contents page number-text paragraph key-value pairs in a unified format.

[0036] Furthermore, according to the indentation rules, the page number line information and all single-line text information are first read, and the page is divided into columns based on the page number distribution characteristics in the page number lines. The single-line text image coordinates corresponding to each single-line text are then processed. Assign the text to the corresponding column; then sort the image coordinates of the single-line text in each column according to the reading order from top to bottom, using the image coordinates of the non-right-indented line of text as the merging starting point; traverse downwards from this starting point, and after finding the position corresponding to the page number line, continue merging subsequent right-indented lines that meet the conditions, until a new page number or non-right-indented text is encountered, ending the merging of the current entry; repeat the above entry merging steps until no more can be merged, then traverse the remaining unmerged page number lines and single-line text, treating isolated page number lines as separate valid entries.

[0037] A preferred, improved BERT semantic verification model based on domain knowledge includes the following structure:

[0038] The input layer uses the format CLS + text line content + SEP to standardize the page number lines into a sequence, where CLS refers to the classification task marker and SEP refers to the sequence end marker.

[0039] The embedding layer transforms the input standardized sequence into a high-dimensional semantic vector, including word embedding, position embedding, and domain identifier embedding. The three embeddings are concatenated to obtain the embedding features.

[0040] The domain enhancement encoder layer employs a 12-layer encoder to encode embedded features. Each encoder layer includes a multi-head attention mechanism, a feedforward neural network, layer normalization, and residual connections. In addition, a terminology enhancement module is inserted after the 6th encoder layer to improve the model's understanding of specialized terms.

[0041] A dual-task output layer is designed, featuring a text classification + entity extraction dual-task encoder to achieve simultaneous extraction of directory entries and page numbers or titles;

[0042] Specifically, in the domain enhancement encoder layer, each encoder layer contains a multi-head attention mechanism with 12 attention heads, each with a 64-dimensional dimension. Masked self-attention is used to focus on the current token and the token to its left to capture the semantic relationships between words within the text, conforming to the text reading logic. The feedforward neural network consists of two fully connected layers. The first layer has an output dimension of 768x4 and uses GELU as the activation function. The second layer has an output dimension of 768 and performs a non-linear transformation on the self-attention output of the multi-head attention to enhance representation capabilities. Layer normalization is used to stabilize the training process and accelerate training convergence. Residual connections can effectively alleviate the gradient vanishing problem in deep models. The terminology enhancement module first constructs a dictionary of engineering domain terms and pre-generates 768-dimensional vectors corresponding to the terms. After the output of the 6th encoder layer, the cosine similarity between each token vector in the current sequence and the terminology dictionary vector is calculated. For tokens with a similarity greater than or equal to 0.8, their vectors are weighted and fused with the corresponding term vectors to enhance the semantic expression of the terms.

[0043] Specifically, the dual-task output layer includes a text classification task and an entity extraction task. The input to the output layer of the text classification task is a 768-dimensional semantic vector output by the domain augmentation encoder layer corresponding to CLS. It adopts two fully connected layers. The first layer has an output dimension of 256 dimensions and uses GELU as the activation function. The second layer has an output dimension of 2 dimensions and uses the softmax function for normalization, outputting probability values ​​and obtaining the final judgment result. The input to the output layer of the entity extraction task is a 768-dimensional semantic vector of all tokens output by the domain augmentation encoder layer. It adopts a single fully connected layer with an output dimension of 4 dimensions and uses the softmax function for normalization. Entity labels are assigned to each token, and page numbers and text content are extracted through the label sequence.

[0044] Preferably, the training dataset for the domain knowledge-based BERT semantic verification model is a collection of multilingual journal PDF catalogs. The Adam optimizer is used, with an initial learning rate of 0.001. A linear learning rate decay strategy is adopted, with a decay rate of 10% every 10 training epochs, for a total of 50 training epochs. The loss function is the cross-entropy loss of the two tasks.

[0045] Preferably, a typesetting feature classification model includes the following steps:

[0046] Feature parameters are extracted from the dataset consisting of all single-line texts. The feature parameters corresponding to each single-line text include page number position quantization parameters, indentation quantization parameters, column feature quantization parameters, and line spacing quantization parameters. The four feature parameters are concatenated to obtain the layout features of the single-line text.

[0047] The layout features are input into a lightweight MLP, and the layout types are classified through three hidden layers. Based on the classification results, a suitable method for merging directory entries is selected.

[0048] Specifically, the page number position quantization parameter is obtained by calculating the ratio between the difference between the left boundary of the page number and the left boundary of a single line of text and the total horizontal length of that single line of text; the indentation quantization parameter is obtained by calculating the mean and standard deviation of the left boundaries of all single lines of text on the same page; the column feature quantization parameter is obtained by extracting the coordinates of the left boundaries of all single lines of text on the same page and performing clustering to obtain the optimal number of columns; and the line spacing quantization parameter is the mean and standard deviation of the vertical spacing between all single lines of text on the same page.

[0049] Specifically, after the layout features are input into a lightweight MLP, they pass through three hidden layers to output the final layout type classification result. The first hidden layer has a 64-dimensional output dimension and uses the ReLU activation function; the second hidden layer has a 32-dimensional output dimension, uses the ReLU activation function, and uses L2 regularization to constrain the weights to avoid overfitting; the third hidden layer has a 16-dimensional output dimension and uses the ReLU activation function; finally, an output layer outputs a 4-dimensional result, which is then normalized using the softmax function to obtain the possible probability of each layout type.

[0050] Preferably, the training set of the typesetting feature classification model consists of journal PDF samples containing four typesetting types, using classification cross-entropy loss as the loss function, Adam optimizer, initial learning rate set to 0.001, and training epochs set to 50.

[0051] Compared with existing technologies, this invention employs a multimodal large-scale model to parse PDFs, extracting page numbers, text box coordinates, and content. After unifying whitespace characters and filtering non-core characters, it uses multi-dimensional rule verification to statistically analyze table of contents features, and then uses a dual-threshold judgment mechanism to locate the range of table of contents pages. Subsequently, it performs combined filtering for noise reduction, adaptive threshold binarization, and morphological post-processing on the grayscale image of the table of contents pages to generate a high-quality binarized image. Text boxes are cropped according to their coordinates and divided into lines using vertical projection, and single-line text is obtained through OCR recognition. Candidate page number lines are filtered based on page number position and length rules, and semantic verification using a large language model eliminates mismatches, outputting structured data. Finally, different merging methods are used for different layouts to integrate them into complete table of contents entries. Attached Figure Description

[0052] Figure 1 The flowchart shows a method for extracting and integrating journal catalogs based on a multimodal large model.

[0053] Figure 2 Flowchart for locating the scope of directory pages;

[0054] Figure 3 Here is a flowchart of a single-line text recognition method;

[0055] Figure 4 This is a flowchart of a typesetting feature classification model method. Detailed Implementation

[0056] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0057] Example 1

[0058] like Figure 1 As shown, a specific embodiment of the present invention discloses a method for extracting and integrating journal catalogs based on a multimodal large model, including the following specific steps:

[0059] A multimodal large model was used to parse the journal PDF document to obtain element information for each page, including page number. Text box coordinates Text content ; text content The text is processed to remove whitespace and non-core characters, and the page numbering rules are validated. A dual-threshold determination mechanism is used to locate the range of page numbers. ;

[0060] Based on the scope of the table of contents Extract the grayscale image of the table of contents from the journal's PDF document and denote it as... ,in The grayscale image of the directory page is subjected to combined filtering, adaptive threshold binarization, and morphological post-processing to obtain a binarized image of the directory page, denoted as . ;

[0061] Based on the text box coordinates Binarized image of the directory page Cropping is performed to obtain the text box image. The text box image is projected using the vertical projection method. Line breaks are used to obtain the coordinates of a single line of text image. Perform OCR recognition on single-line text and fill the corresponding fields with the recognition results. of middle;

[0062] Based on preset rules, page number lines that match the characteristics of table of contents lines are selected from single-line text. A large language model is introduced to perform semantic understanding on the text content corresponding to the selected page number lines in order to eliminate mismatched entries. At the same time, data cleaning is performed to output structured JSON results.

[0063] Different methods for merging table of contents entries are selected for different layouts to integrate scattered table of contents lines into complete table of contents paragraph entries;

[0064] Furthermore, such as Figure 2 As shown, a multimodal large model is used to parse the journal PDF document, and the total number of pages in the journal PDF document is recorded. Page, in Extracted element information from the page, including the page number. Text box coordinates Text content ; the extracted first The text content of the page The process involves unifying whitespace characters and filtering non-core characters. Multi-dimensional rules are used to match the processed text content against category page features. A dual-threshold judgment mechanism is then employed, based on page number. Sort the pages in ascending order, and determine the pages whose characteristic lines are greater than or equal to the first threshold as the starting page of the directory. Then, continue traversing backwards to find the directory page feature line. The size relationship between the directory page feature line and the second threshold determines the end page of the directory. ;

[0065] Furthermore, regarding the extracted first... The text content of the page The code uses regular expressions to match all Unicode whitespace characters, including spaces, tabs, and newlines, and replaces them with single ASCII spaces to eliminate whitespace differences caused by different layouts. Additionally, it uses regular expressions to match and filter all characters except letters, numbers, and common symbols. The processed text content is denoted as... ;

[0066] Furthermore, for text content processed by regular expressions... The system counts feature lines on the directory page based on multi-dimensional rules. First, it filters based on formatting rules; feature lines on the directory page must meet the combination of "significant numbers + text content." Then, it analyzes the text content according to the formatting rules. After initial screening, further filtering is performed based on length rules. Lines with character lengths less than the noise length threshold are considered pure page numbers or noise fragments, while lines with character lengths greater than the non-directory length threshold are considered non-directory text. Lines with character lengths less than the noise length threshold or greater than the non-directory length threshold are excluded. For example, the noise length threshold can be set to 4, and the non-directory length threshold can be set to 100. After validation according to the above format and length rules, the number of lines is counted. The number of all directory page feature lines in the page that meet the rules is denoted as . ;

[0067] Furthermore, considering the formatting patterns of journal table of contents, the starting page typically contains complete first-level entries and some second-level entries, with the largest number of entries; subsequent pages mostly contain second- or third-level entries, with a relatively smaller number. A dual-threshold determination mechanism is employed to determine the range of table of contents pages through a two-step progressive determination. First, based on the page numbers extracted from each page... Sort the data in ascending order and iterate through it, counting the number of characteristic rows corresponding to each page in the table of contents. Find the first number of rows that meet the characteristics of the directory page. Pages that are greater than or equal to the first threshold are identified as the directory start page, and their corresponding page numbers are denoted as follows: Then from the beginning of the table of contents Initially, traverse each page in ascending order of page number. If the number of directory page characteristic lines for the current page is greater than or equal to the second threshold, determine that the current page is a subsequent page in the directory. If the number of directory page characteristic lines for two consecutive pages is less than the second threshold, stop traversing and determine the last page that satisfies the condition of having a number of directory page characteristic lines greater than or equal to the second threshold as the end page of the directory, and record its corresponding page number as . After the above steps, the final directory page range is obtained. ;

[0068] Furthermore, based on the scope of the directory pages The grayscale image of the corresponding page's directory is denoted as... ,in The grayscale image of the directory page is filtered and denoised, then subjected to adaptive threshold binarization and morphological post-processing to obtain a binarized image of the directory page, denoted as . ;

[0069] Furthermore, regarding the grayscale image of the directory page... The image contains various types of interference, such as scanning noise and Gaussian noise. A combined filtering strategy of Gaussian filtering and median filtering is used for layered denoising to obtain the denoised image. It preserves text details while removing noise;

[0070] Based on the grayscale image of the catalog page The image noise intensity is adaptively selected using different Gaussian and median filter kernels for Gaussian and median filtering; specifically, the grayscale image of the directory page is calculated. global grayscale mean The calculation formula is as follows:

[0071] ,

[0072] in, This represents the width of the grayscale image for the directory page. The height of the grayscale image of the directory page. Represents the grayscale coordinates of the directory page The grayscale value at that location; then the grayscale image of the directory page is calculated. gray standard deviation The formula for calculating noise intensity is as follows:

[0073] ;

[0074] Then based on the grayscale standard deviation The size determines the size of the Gaussian kernel and the median filter kernel, when the gray standard deviation When the value is less than 20, select a size of Gaussian kernel and size The median filter kernel; when the gray standard deviation When the value is greater than or equal to 20 and less than or equal to 40, select a size of Gaussian kernel and size The median filter kernel; when the gray standard deviation When the value is greater than 40, select a size of Gaussian kernel and size The median filter kernel; based on the selected Gaussian kernel and median filter kernel, the grayscale images of the directory page are processed sequentially. Gaussian filtering and median filtering are performed to obtain the denoised directory page diagram. ;

[0075] Furthermore, considering the significant difference in text and background contrast in some grayscale images of the directory pages, adaptive threshold binarization is used to denoise the directory page images and obtain preliminary binarized images. ;

[0076] Denoising the table of contents The image is divided into fixed-size local windows, and the size of the local windows is adaptively adjusted according to the text size in the current denoising catalog page image. For each local window, the average pixel value within that local window is calculated using the aforementioned method for calculating the grayscale mean and grayscale standard deviation. and standard deviation Then, the Sauvola algorithm was used to calculate the pixel threshold within this local window. The calculation formula is as follows:

[0077] ,

[0078] in, To adjust the coefficient, it adaptively adjusts based on the image contrast; the default setting is 0.4. The value is set to 128, which is half the grayscale range; then each pixel within this local window is compared with the pixel threshold. A comparison is made when a pixel is smaller than a pixel threshold. When a pixel is identified as a text pixel, its size is set to 0; when a pixel is greater than or equal to a pixel threshold... When the pixel is identified as a background pixel, its pixel size is set to 255. Following the above method, all local windows are sequentially binarized to obtain a preliminary binarized image. ;

[0079] Furthermore, to address issues such as broken text strokes, isolated noise residue, background holes, and text edge burrs in the initial binarized image, a combination of morphological dilation and erosion operations was used for precise repair to obtain the binarized image of the directory page. ;

[0080] use The rectangular structural element for the initial binarized image Inflation is used to fill tiny gaps in the text, enhancing its continuity. The specific settings for the rectangular structural element are as follows:

[0081] ;

[0082] Perform an erosion operation on the dilated image, using the same... The rectangular structural element eliminates jagged edges and noise at the edges of the text to restore the initial outline of the text;

[0083] Connectivity analysis is performed on the image after dilation and erosion, and the area of ​​each background connected region is calculated. Background connected regions with an area of ​​5 pixels or less are considered as hole noise and are filled using a seed filling algorithm to ensure the purity of the background area. At the same time, holes in text regions with an area greater than 5 pixels are preserved to avoid damage to the text structure. After the above morphological post-processing method, the binarized image of the directory page is obtained. ;

[0084] Furthermore, such as Figure 3 As shown, based on the binarized image of the directory page and text box coordinates Cropping is performed to obtain the text box image. And the text box image is projected using the vertical projection method. Line breaks are used to obtain the coordinates of a single line of text image. Perform OCR recognition on single-line text and fill the corresponding fields with the recognition results. of middle;

[0085] Furthermore, based on the coordinates of the text boxes on each page... Binary image of the corresponding directory page for this page Cropping is performed to obtain the text box image. Then, vertical projection is used to segment the text box image into lines;

[0086] Specifically, for text box images First, calculate its vertical projection curve, and denote the text box image. The width is The height is The vertical projection curve is obtained by counting the total number of black pixels in each row. For the first row... Total number of black pixels in a row The calculation formula is as follows:

[0087] ,

[0088] in, The text box image is in the first position. Line number The pixel value of the column, if it is black text. If it is a white background, then According to the calculated first Total number of black pixels in a row row index The x-axis represents the total number of black pixels. A vertical projection curve is constructed for the ordinate to reflect the text density of each line; to eliminate misjudgments caused by local fluctuations in the vertical projection curve, a moving average filter is used to smooth the curve, and the calculation formula is as follows:

[0089] ,

[0090] A filter window of size 3 is used to smooth the total number of black pixels in each row, ensuring smoothness while preserving the trend; trough detection is performed on the smoothed vertical projection curve, and a trough threshold is set as . Traversing the smoothed vertical projection curve Filter to meet The consecutive row index intervals are denoted as the trough intervals. ;

[0091] The upper and lower boundaries of a single line of text are determined by separating adjacent trough intervals. For example, for two adjacent trough intervals... and Let the region between these two adjacent trough intervals be called a single line of text, and the upper boundary of this single line of text corresponds to... The lower boundary corresponds to Then, the coordinates of the single-line text image corresponding to the single-line text were determined. ,in This represents the coordinates of the top-left corner of a single-line text area. This indicates the coordinates of the bottom right corner of a single-line text region; OCR recognition technology is used to determine the coordinates of the single-line text image. The function recognizes the single line of text and fills the corresponding text content into the correct field. middle;

[0092] Furthermore, based on the journal table of contents page number rules, page number lines that meet the rules are filtered from single-line text, and the text content of the page number lines is semantically understood through a large language model to exclude mismatched entries and output structured JSON results.

[0093] Specifically, the journal table of contents page numbering rules include page number position rules and page number length rules. The page number position rule specifies that the page number is located at the beginning or end of the text line and is clearly separated from the title text. The page number length rule specifies that the character length of the page number should be limited; pages exceeding the limit are considered invalid. All single-line text from the previous step are filtered using these two journal table of contents page numbering rules. The text content corresponding to the page number lines that meet the rules is input into the large language model for semantic understanding. For example, to effectively exclude rule mismatch entries and ensure compatibility with multilingual filtering rules and Japanese OCR character recognition errors, the prompt word can be set as follows: "1. For each input text line, determine whether it is a journal table of contents entry; 2. If it is a table of contents entry, extract the accurate page number and complete title text, removing irrelevant prefixes or suffixes; 3. If it is not a table of contents entry, mark it as non-table of contents; 4. Supports Chinese, English, German, and Japanese languages, compatible with Japanese OCR recognition errors, German compound word splitting errors, and English abbreviation recognition errors; 5. Outputs a structured JSON result, with fields including: whether it is a table of contents entry, page number, and title."

[0094] Furthermore, different table of contents merging methods are selected for different layouts to integrate scattered table of contents lines into complete table of contents paragraph entries, including dynamic distance merging, merging the first line of page number lines downwards, merging the last line of page number lines upwards, and merging according to indentation rules.

[0095] Furthermore, dynamic distance merging first extracts all single-line text information and integrates it into a unified dataset. For each valid table of contents line containing page numbers, an initial merging unit (including index, upper and lower boundaries, and single-line text image coordinates) is constructed. List and text content List); Image coordinates of single-line text in the table of contents in natural reading order. Sorting the list ensures that the processing flow is consistent with the reading logic;

[0096] Then, using the left and right boundaries of the target directory line as a reference, extend 5 pixels to the left and right to form a horizontal effective area; extract candidate single-line text and calculate its vertical distance to the target directory line, set a vertical distance threshold, and sort according to "nearest to farthest + positive distance priority", then remove single-line text that exceeds the threshold; for the filtered candidate single-line text, if it is located between two adjacent directory lines (i.e., it falls within the horizontal effective area of ​​two directory lines), calculate its vertical distance to the two directory lines and assign it to the directory line with the closer distance; if it only falls within the horizontal effective area of ​​one directory line, it is directly assigned to that directory line; at the same time, the image coordinates of the single-line text corresponding to the text line are also calculated. and text content Add to the merged unit corresponding to the directory line, recalculate the overall upper and lower boundaries of the merged unit, and verify the integrity of the information;

[0097] Furthermore, the page number merging process first extracts all single-line text information and constructs a unified dataset to create initial merge entries for valid table of contents lines containing page numbers. It then locates the position of each table of contents line within the full text line list and sorts the initial merge entries according to the natural order of the table of contents lines. Using the table of contents line as the core, it filters single-line text below it that conforms to the coordinate rules and merges them into the corresponding entries. Based on the entry's position in the table of contents (first, middle, or last), it differentiates the attribution logic of surrounding single-line text to ensure the integrity of the merge. Finally, it generates table of contents page number-text paragraph key-value pairs in a unified format.

[0098] Furthermore, the page number merging process first extracts all single-line text information and constructs a unified dataset to create initial merge entries for valid table of contents lines containing page numbers; it then locates the position of each table of contents line in the full text line list and sorts the initial merge entries according to the natural order of the table of contents lines; using the table of contents line as the core, it filters single-line text above it that conforms to the coordinate rules and merges them into the corresponding entries; based on the position of the entry in the table of contents (first, middle, last), it differentiates the belonging logic of the surrounding single-line text to ensure the integrity of the merge; subsequently, it generates table of contents page number-text paragraph key-value pairs in a unified format.

[0099] Furthermore, according to the indentation rules, the page number line information and all single-line text information are first read, and the page is divided into columns based on the page number distribution characteristics in the page number lines. The single-line text image coordinates corresponding to each single-line text are then processed. Assign the text to the corresponding column; then sort the image coordinates of the single-line text in each column according to the reading order from top to bottom, using the image coordinates of the non-right-indented text in the current line as the merging starting point; traverse downwards from this starting point, and after finding the position corresponding to the page number line, continue to merge subsequent right-indented lines that meet the conditions, until a new page number or non-right-indented text is encountered, at which point the merging of the current entry ends; repeat the above entry merging steps until no more can be merged, then traverse the remaining unmerged page number lines and single-line text, and treat isolated page number lines as separate valid entries.

[0100] Example 2

[0101] This embodiment is an improvement upon the journal catalog extraction and integration method based on a multimodal large model disclosed in Example 1. To reduce redundant descriptions, this embodiment only introduces the improved parts, and the parts that are the same as those in Example 1 are not introduced.

[0102] Example 1 proposes to use a large language model to perform semantic understanding of the text content of page number lines in order to eliminate mismatched entries. However, the large language model is not open source code, making it difficult to optimize the model further. Therefore, a BERT semantic verification model based on domain knowledge is proposed.

[0103] Preferably, a domain-knowledge-based BERT semantic verification model adopts a four-layer architecture consisting of an input layer, an embedding layer, a domain-enhanced encoder layer, and a dual-task output layer, with targeted improvements for directory scenarios. The specific structure is as follows:

[0104] The input layer uses the format CLS + text line content + SEP to standardize the page number lines into a sequence, where CLS refers to the classification task marker and SEP refers to the sequence end marker.

[0105] The embedding layer transforms the input standardized sequence into a high-dimensional semantic vector, including word embedding, position embedding, and domain identifier embedding. The three embeddings are concatenated to obtain the embedding features.

[0106] The domain enhancement encoder layer employs a 12-layer encoder to encode embedded features. Each encoder layer includes a multi-head attention mechanism, a feedforward neural network, layer normalization, and residual connections. In addition, a terminology enhancement module is inserted after the 6th encoder layer to improve the model's understanding of specialized terms.

[0107] A dual-task output layer is designed, featuring a text classification + entity extraction dual-task encoder to achieve simultaneous extraction of directory entries and page numbers or titles;

[0108] Specifically, in the domain enhancement encoder layer, each encoder layer contains a multi-head attention mechanism with 12 attention heads, each with a 64-dimensional dimension. Masked self-attention is used to focus on the current token and the token to its left to capture the semantic relationships between words within the text, conforming to the text reading logic. The feedforward neural network consists of two fully connected layers. The first layer has an output dimension of 768x4 and uses GELU as the activation function. The second layer has an output dimension of 768 and performs a non-linear transformation on the self-attention output of the multi-head attention to enhance representation capabilities. Layer normalization is used to stabilize the training process and accelerate training convergence. Residual connections can effectively alleviate the gradient vanishing problem in deep models. The terminology enhancement module first constructs a domain terminology dictionary and pre-generates 768-dimensional vectors corresponding to the terms. After the output of the 6th encoder layer, the cosine similarity between each token vector in the current sequence and the terminology dictionary vector is calculated. For tokens with a similarity greater than or equal to 0.8, their vectors are weighted and fused with the corresponding term vectors to enhance the semantic expression of the terms.

[0109] Specifically, the dual-task output layer includes a text classification task and an entity extraction task. The input to the output layer of the text classification task is a 768-dimensional semantic vector output by the domain augmentation encoder layer corresponding to CLS. It adopts two fully connected layers. The first layer has an output dimension of 256 dimensions and uses GELU as the activation function. The second layer has an output dimension of 2 dimensions and uses the softmax function for normalization, outputting probability values ​​and obtaining the final judgment result. The input to the output layer of the entity extraction task is a 768-dimensional semantic vector of all tokens output by the domain augmentation encoder layer. It adopts a single fully connected layer with an output dimension of 4 dimensions and uses the softmax function for normalization. Entity labels are assigned to each token, and page numbers and text content are extracted through the label sequence.

[0110] Preferably, the training dataset for the domain knowledge-based BERT semantic verification model is a collection of multilingual journal PDF catalogs. The Adam optimizer is used, with an initial learning rate of 0.001. A linear learning rate decay strategy is adopted, with a decay rate of 10% every 10 training epochs, for a total of 50 training epochs. The loss function is the cross-entropy loss of the two tasks.

[0111] In addition to proposing a BERT semantic verification model based on domain knowledge, considering that the method of merging different directory entries for different layouts in Example 1 is manually selected and lacks flexibility, a layout feature classification model is proposed in Example 2.

[0112] Preferred, such as Figure 4 As shown, a typesetting feature classification model includes the following steps:

[0113] Feature parameters are extracted from the dataset consisting of all single-line texts. The feature parameters corresponding to each single-line text include page number position quantization parameters, indentation quantization parameters, column feature quantization parameters, and line spacing quantization parameters. The four feature parameters are concatenated to obtain the layout features of the single-line text.

[0114] The layout features are input into a lightweight MLP, and the layout types are classified through three hidden layers. Based on the classification results, a suitable method for merging directory entries is selected.

[0115] Specifically, the page number position quantization parameter is obtained by calculating the ratio between the difference between the left boundary of the page number and the left boundary of a single line of text and the total horizontal length of that single line of text; the indentation quantization parameter is obtained by calculating the mean and standard deviation of the left boundaries of all single lines of text on the same page; the column feature quantization parameter is obtained by extracting the coordinates of the left boundaries of all single lines of text on the same page and performing clustering to obtain the optimal number of columns; and the line spacing quantization parameter is the mean and standard deviation of the vertical spacing between all single lines of text on the same page.

[0116] Specifically, after the layout features are input into a lightweight MLP, they pass through three hidden layers to output the final layout type classification result. The first hidden layer has a 64-dimensional output dimension and uses the ReLU activation function; the second hidden layer has a 32-dimensional output dimension, uses the ReLU activation function, and uses L2 regularization to constrain the weights to avoid overfitting; the third hidden layer has a 16-dimensional output dimension and uses the ReLU activation function; finally, an output layer outputs a 4-dimensional result, which is then normalized using the softmax function to obtain the possible probability of each layout type.

[0117] Preferably, the training set of the typesetting feature classification model consists of journal PDF samples containing four typesetting types, and the classification cross-entropy loss is used as the loss function. The Adam optimizer is used, the initial learning rate is set to 0.001, and the number of training epochs is set to 50.

[0118] The beneficial effects of this embodiment are that the domain knowledge-based BERT semantic verification model enhances semantic recognition accuracy and efficiency by incorporating domain terminology, and improves the ability to resist OCR errors and multilingual interference by performing dual tasks in parallel; the typesetting feature classification model realizes automatic matching of merging methods, accurately classifies based on quantitative features, is compatible with diverse typesetting, and lowers the operation threshold. The combination of the two greatly improves the automation rate and accuracy of catalog parsing, and adapts to more journal scenarios.

[0119] This invention discloses a method for extracting and integrating journal catalogues based on a multimodal large-scale model. First, a multimodal large-scale model is used to parse PDFs, extracting page numbers, text box coordinates, and content. After unifying whitespace characters and filtering non-core characters, the catalogue feature lines are statistically verified using multi-dimensional rules, and then a dual-threshold judgment mechanism is used to locate the catalogue page range. Subsequently, the grayscale image of the catalogue page is subjected to combined filtering for noise reduction, adaptive threshold binarization, and morphological post-processing to generate a high-quality binarized image. Text boxes are cropped according to their coordinates and divided into lines using vertical projection, and single-line text is obtained through OCR recognition. Candidate page number lines are filtered based on page number position and length rules, and semantic verification using a large language model is used to eliminate mismatches, outputting structured data. Finally, different merging methods are used for different layouts to integrate them into complete catalogue entries.

[0120] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principle of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for extracting and integrating journal catalogs based on a multimodal large model, characterized in that, Includes the following steps: A multimodal large model is used to parse journal PDF documents, extracting page numbers, text box coordinates, and text content; the text content is processed to remove whitespace and non-core characters, and the table of contents page number rules are validated. A dual threshold judgment mechanism is used to locate the range of table of contents pages. Extract the grayscale image of the directory page, perform combined filtering, adaptive threshold binarization, and morphological post-processing on the grayscale image to generate a binarized image of the directory page; The binarized image is cropped based on the coordinates of the text box, and the coordinates of the single-line text image are obtained by dividing it into lines using the vertical projection method. The single-line text is then subjected to OCR recognition, and the recognition result is filled in. Page number lines that match the characteristics of the directory line are selected based on preset rules, and mismatched entries are excluded through semantic validation, and structured data is output. Choose the appropriate method to merge directory entries for different layouts, and integrate scattered directory behaviors into complete directory paragraph entries.

2. The journal catalog extraction and integration method based on a multimodal large model as described in claim 1, characterized in that, The grayscale image is subjected to combined filtering, adaptive threshold binarization, and morphological post-processing to generate a binarized image of the directory page, including: Calculate the global grayscale mean of the grayscale image of the directory page; then calculate the grayscale standard deviation of the grayscale image of the directory page as the noise intensity; determine the size of the Gaussian kernel and the median filter kernel based on the size of the grayscale standard deviation; and perform Gaussian filtering and median filtering on the grayscale image of the directory page in sequence according to the selected Gaussian kernel and median filter kernel to obtain the denoised directory page image. The denoised catalog page image is divided into fixed-size local windows. For each local window, the mean and standard deviation of the image pixels within the local window are calculated. Then, the Sauvola algorithm is used to calculate the pixel threshold within the local window. Subsequently, each pixel within the local window is compared with the pixel threshold to perform binarization and obtain a preliminary binarized image. use The rectangular structuring element is used to dilate the initial binarized image, and the same rectangular structuring element is used to erode the dilated image. Then, connected component analysis is performed, and the seed filling algorithm is used to fill the binarized image of the directory page.

3. The journal catalog extraction and integration method based on a multimodal large model as described in claim 2, characterized in that, The scope of directory pages is located using a dual-threshold determination mechanism, including: The extracted page numbers are sorted in ascending order and traversed. The number of directory page feature lines for each page is counted. The first page that meets the condition that the number of directory page feature lines is greater than or equal to the first threshold is identified and determined as the directory start page. Then, starting from the directory start page, each page is traversed in ascending order by page number. If the number of directory page feature lines for the current page is greater than or equal to the second threshold, the current page is determined as the directory follow-up page. If the number of directory page feature lines for two consecutive pages is less than the second threshold, the traversal stops and the last page that meets the condition that the number of directory page feature lines is greater than or equal to the second threshold is determined as the directory end page, thus obtaining the final determined range of directory pages.

4. The journal catalog extraction and integration method based on a multimodal large model as described in claim 3, characterized in that, The coordinates of a single line of text image are obtained by dividing the text into lines using the vertical projection method, including: The total number of black pixels in each line of the text box is counted to obtain the vertical projection curve. A filter window of size 3 is used to smooth the total number of black pixels in each line. The smoothed vertical projection curve is then subjected to trough detection. The smoothed vertical projection curve is traversed, and the continuous line index intervals that satisfy the condition that the total number of black pixels is greater than the trough threshold are selected as the trough intervals. For two adjacent trough intervals, the upper boundary of a single line of text corresponds to the right endpoint of the first trough interval +1, and the lower boundary corresponds to the left endpoint of the second trough interval -1.

5. The journal catalog extraction and integration method based on a multimodal large model as described in claim 4, characterized in that, Page number lines that match the characteristics of a table of contents line are filtered based on preset rules, including: The preset rules include page number position rules and page number length rules; the page number position rules refer to the page number being located at the beginning or end of a text line and having a clear separator between it and the title text; the page number length rules refer to the requirement that the character length of the page number should be limited, and page numbers exceeding the limit are considered invalid.

6. The journal catalog extraction and integration method based on a multimodal large model as described in claim 5, characterized in that, Methods for merging directory entries include: Different table of contents merging methods are selected for different layouts to integrate scattered table of contents lines into complete table of contents paragraph entries, including dynamic distance merging, merging the first line of page number lines downwards, merging the last line of page number lines upwards, and merging according to indentation rules.

7. The journal catalog extraction and integration method based on a multimodal large model as described in claim 6, characterized in that, Semantic validation can be further improved. The improved semantic validation model includes the following steps: The input layer uses the CLS + text line content + SEP format to standardize the page number lines into a sequence. The embedding layer transforms the input standardized sequence into a high-dimensional semantic vector, including word embedding, position embedding, and domain identifier embedding. The three embeddings are concatenated to obtain the embedding features. The domain enhancement encoder layer uses a 12-layer encoder to encode the embedded features. Each encoder layer includes a multi-head attention mechanism, a feedforward neural network, layer normalization, and residual connections. A term enhancement module is inserted after the 6th encoder layer. The dual-task output layer is designed with a text classification and entity extraction dual-task encoder to simultaneously determine directory entries and extract page numbers or titles.

8. The journal catalog extraction and integration method based on a multimodal large model as described in claim 7, characterized in that, The terminology enhancement module includes: Construct a dictionary of engineering terms and pre-generate 768-dimensional vectors corresponding to the terms. After the output of the 6th layer encoder, calculate the cosine similarity between each token vector in the current sequence and the term dictionary vector. For tokens with a similarity greater than or equal to 0.8, perform weighted fusion of their vectors with the corresponding term vectors.

9. The journal catalog extraction and integration method based on a multimodal large model as described in claim 8, characterized in that, The method for merging directory entries corresponding to different layouts can be further improved. An improved layout feature classification model includes: Feature parameters are extracted from the dataset consisting of all single-line texts. The feature parameters corresponding to each single-line text include page number position quantization parameters, indentation quantization parameters, column feature quantization parameters, and line spacing quantization parameters. The four feature parameters are concatenated to obtain the layout features of the single-line text. The layout features are input into a lightweight MLP, and the layout types are classified through three hidden layers. Based on the classification results, a suitable method for merging directory entries is selected.

10. The journal catalog extraction and integration method based on a multimodal large model as described in claim 9, characterized in that, Feature parameters, including: The page number position quantization parameter is obtained by calculating the ratio between the difference between the left boundary of the page number and the left boundary of a single line of text and the total horizontal length of that single line of text; the indentation quantization parameter is obtained by calculating the mean and standard deviation of the left boundaries of all single lines of text on the same page; the column feature quantization parameter is obtained by extracting the coordinates of the left boundaries of all single lines of text on the same page and performing clustering to obtain the optimal number of columns; the line spacing quantization parameter is the mean and standard deviation of the vertical spacing between all single lines of text on the same page.

Images