Newspaper complex format structured identification method and system based on multi-modal large model

Abstract

The invention relates to the technical field of image processing, in particular to a newspaper complex format structured identification method and system based on a multi-modal large model, and the method comprises the following steps: preprocessing a newspaper image, carrying out the special fine tuning of the multi-modal large model, extracting typesetting elements, and carrying out the post-processing and optimization. The method has the advantages that the newspaper image quality is optimized through multi-step preprocessing, basic elements such as character blocks, picture blocks and column lines in newspaper typesetting and structural relations such as title levels and image-text surrounding modes are extracted through a multi-mode large model subjected to special fine adjustment, and the newspaper typesetting efficiency is improved by combining rule verification and dynamic optimization of a post-processing module. And outputting high-precision structured typesetting data. According to the method, the identification defects of a traditional technology under complex formats (multi-column mixed arrangement, vertical arrangement of characters and advertisement interpenetration) and noise scenes (yellowing and creases) are overcome, the integrity and accuracy of typesetting element extraction are remarkably improved, and the method can be directly applied to scenes such as newspaper digital archiving, format copying and content secondary editing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a newspaper complex layout structured recognition method and system based on a multi-modal large model. BACKGROUND

[0002] As an important medium for carrying history and information, the digital processing of newspapers needs to accurately preserve the original layout structure (such as column width, title level, and position relationship between text and images).

[0003] The prior art has the following significant defects: (1) traditional OCR technology can only recognize text content and cannot understand the layout logic, for example, merging multi-column text into a single segment output, losing key information such as column spacing, title and body level distinction, etc.; (2) general document analysis models (such as LayoutLM series) are not optimized for newspaper characteristics, and the recognition accuracy for vertical text (such as some ancient newspaper) and cross-page layout (such as cross-page connection of serialized articles) is less than 50%; (3) historical newspaper scans often have yellowing, blurred ink, creases, and other noise, and existing technologies lack targeted preprocessing, resulting in an element extraction error rate of more than 30%; (4) lack of dynamic adaptation mechanism, unable to adjust recognition strategies according to newspaper types (such as daily newspapers, weekly newspapers, and professional journals), for example, for commercial newspapers with more than 40% of advertisements, it is easy to misjudge the advertisement text as body text.

[0004] Chinese patent CN114581237A (a newspaper layout analysis method) uses a traditional deep learning model for block segmentation, but does not introduce the semantic understanding ability of large models, and cannot handle complex scenarios such as "title nested in pictures" and "text surrounding irregular graphics", limiting its practicality. SUMMARY

[0005] The present application aims to provide a newspaper complex layout structured recognition method and system based on a multi-modal large model, which realizes accurate analysis of complex layouts such as multi-column mixed layout, vertical text, and advertisement insertion, to solve the problems raised in the background technology.

[0006] To achieve the above-mentioned purpose, the present application provides the following technical solution: a newspaper complex layout structured recognition method based on a multi-modal large model, comprising the following steps:

[0007] Newspaper image preprocessing: A hierarchical processing flow is designed for the diversity of newspaper images, including noise classification and targeted denoising. Image feature analysis is used to identify speckle noise, crease noise, and ink diffusion types, and non-local mean filtering, morphological operation-based repair algorithm, and adaptive threshold segmentation are used for denoising, respectively. Geometric correction and standardization are performed, including tilt correction by detecting newspaper edge straight lines, with an error control within ±0.3°. Resolution normalization is performed by dynamically adjusting the resolution according to the type of newspaper. Newspapers with cross-page layout are automatically segmented into single-page images by detecting the margin features.

[0008] Special fine-tuning of multi-modal large models and extraction of layout elements: A "pre-trained model + special fine-tuning" architecture is used, and CLIP-ViT-L / 14 is selected as the base model. A fine-tuning dataset covering 1950-2023 diverse newspaper samples is constructed, with annotated content including the bounding box coordinates, type label, and attribute information of each element. Fine-tuning tasks such as layout feature prediction, semantic association modeling, and type adaptive classification are designed. The model output includes a feature vector containing the confidence, coordinates, and type label of each element, with additional output of font, font size, and line spacing for text blocks, and title level and graphic wrap mode for structural relationships.

[0009] Post-processing and optimization: Rule engine verification is performed with a pre-set newspaper layout industry rule library. Outputs that violate the rules are automatically marked as low-confidence results and trigger manual review. The confidence threshold is dynamically adjusted based on image quality scores. For serialized articles with cross-page layout, the text blocks on consecutive pages are associated based on text content similarity to ensure paragraph continuity.

[0010] Preferably, in noise classification and targeted denoising: Noise type identification uses image feature analysis, specifically analyzing gray variance and edge continuity to identify speckle noise, crease noise, and ink diffusion. Non-local mean filtering is used for speckle noise removal, which removes isolated noise points while preserving text edges. A morphological operation-based repair algorithm is used for crease noise removal, which first detects the crease contour and then fills it in through neighborhood pixel interpolation. Adaptive threshold segmentation is used for ink diffusion denoising, which adjusts the binary threshold value based on the local gray mean value.

[0011] Preferably, in the special fine-tuning of the multi-modal large model: the fine-tuning dataset covers diverse newspaper samples from 1950 to 2023, including People's Daily, Reference News, local evening newspapers, and professional journals, with a total sample size of 150,000 pages; the annotated content includes the boundary box coordinates of each element with an accuracy of 1 pixel, type labels including text blocks / picture blocks / column lines / advertising boxes, and attribute information including the font / size / color of text blocks and the caption position of picture blocks; in the layout feature prediction input image block in the fine-tuning task, the output column number, column width, and column spacing are optimized for coordinate prediction accuracy using the L1 loss function; semantic association modeling associates pictures with adjacent text blocks through contrastive learning, and the loss function uses a triplet loss; type adaptive classification dynamically adjusts classification weights for different newspaper types and enhances sensitivity to specific elements through attention mechanisms.

[0012] Preferably, in the layout element extraction process: the model output feature vector includes the confidence, coordinates (x1, y1, x2, y2), and type label of each element; for text blocks, additional font, font size, and line spacing are output; for structural relationships, title levels and picture-text wrapping methods are output.

[0013] Preferably, in the post-processing and optimization mechanism: the rule engine checks the pre-set newspaper layout industry rule library, which includes that the header area must be located in the top 1 / 5 area of the page and the width must be greater than or equal to 1 / 2 of the page width; the font size of the first-level title is 1.5 times the size of the regular font, and it is usually centered or left-aligned; the edge of the advertising area often contains a decorative border and often contains the keywords "advertisement" and "recruitment"; dynamic threshold adjustment adjusts the confidence threshold based on image quality scores, with a high-quality image threshold of 0.8 to prioritize precision; the low-quality image threshold is set to 0.6 to prioritize recall; cross-page consistency verification checks the cross-page layout of serialized articles by associating the text blocks on the previous and next pages based on text content similarity to ensure paragraph continuity.

[0014] A newspaper complex layout structured recognition method based on a multi-modal large model system, including an input module, a preprocessing module, a model module, a post-processing module, a storage module, an output module, and a monitoring module; each module works cooperatively to realize the structured recognition of the complex layout of newspapers;

[0015] The input module supports multiple formats and automatically parses image metadata; the preprocessing module performs noise classification, denoising, geometric correction, and resolution normalization operations; the model module includes pre-trained model loading, fine-tuning parameter configuration, and feature vector output functions; the post-processing module implements rule engine verification, dynamic threshold adjustment, and cross-page consistency verification; the storage module stores structured data, original images, and annotation results in a specific way; the output module provides multiple formats and PDF with editable annotation layers; the monitoring module real-time monitors model-related indicators and triggers incremental training under certain conditions.

[0016] Preferably, the input module: supports multiple format inputs, including JPG, PNG, TIFF and PDF scans; automatically parses image metadata, covering resolution, color mode information, providing basic data support for subsequent processing.

[0017] Preferably, the storage module: uses MySQL to store structured data, specifically element attribute and coordinate information; uses HDFS to store raw images and annotation results; supports indexing by "newspaper name + publication date" for quick retrieval and query of related data.

[0018] Preferably, the output module: provides multiple format outputs, including JSON, XML and editable PDF; the output PDF contains editable annotation layers, supporting user drag adjustment of element boundaries, meeting the user's further editing needs for the layout results.

[0019] Preferably, the monitoring module: calculates the F1 value of the test set every hour to monitor the accuracy of the model in real time; at the same time, it calculates the processing speed to ensure that the average time consumption per page is less than or equal to 1.5 seconds; when the accuracy is lower than 85% for 2 consecutive hours, the incremental training is automatically triggered, 1000 pages of error samples are extracted from the fine-tuning dataset to optimize the performance of the model.

[0020] Compared with the prior art, the beneficial effects of the present application are:

[0021] The newspaper complex layout structured recognition method and system based on multi-modal large model provided by the present application optimizes the quality of newspaper images through multi-step preprocessing, extracts basic elements such as text blocks, picture blocks and column lines in newspaper layout, title levels and graphic wrapping methods, etc. by using a multi-modal large model fine-tuned for special purposes, and combines rule checking and dynamic optimization of the post-processing module to output high-precision structured layout data. The present application solves the recognition defects of traditional technology in complex layout (multi-column mixed layout, vertical text, and advertisement insertion) and noise scenes (yellowing and creases), significantly improves the completeness and accuracy of layout element extraction, and can be directly applied to newspaper digitization archiving, layout reprinting, content secondary editing, etc. The present application has strong practicability and innovation. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 The method flowchart of the present application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical scheme of the present application, and the advantages are clearer and more apparent, the embodiments of the present application are further described in detail below with reference to the drawings. It should be understood that the specific embodiments described herein are part of the embodiments of the present application, rather than all the embodiments, and are used to explain the embodiments of the present application, and are not used to limit the embodiments of the present application, and all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the present application.

[0024] Embodiment one, the present application provides a technical scheme: a newspaper complex layout structured recognition method based on multi-modal large model, comprising the following steps:

[0025] (1) Newspaper image preprocessing technology

[0026] The preprocessing module designs a hierarchical processing flow for the diversity of newspaper images (such as scanning quality, layout type):

[0027] Noise classification and targeted denoising:

[0028] 1) Adopt image feature analysis (gray variance, edge continuity) to identify noise types: speckle noise (common in ancient newspapers), crease noise (common in modern newspaper scans), ink diffusion (ink bleeding in old newspapers);

[0029] 2) For speckle noise, use non-local mean filtering (NL-means) to remove isolated noise points while preserving text edges; for crease noise, use a repair algorithm based on morphological operations (first detect the crease contour, then fill in through neighborhood pixel interpolation); for ink diffusion, use adaptive threshold segmentation (adjust the threshold value for binarization according to the local gray mean value).

[0030] Geometric correction and standardization:

[0031] 1) Tilt correction: by detecting the straight line of the newspaper edge (use Hough transform to extract the edge and calculate the tilt angle), rotate the image to horizontal, with an error of within ±0.3°;

[0032] 2) Resolution normalization: dynamically adjust the resolution according to the newspaper type (unify to 300 dpi for daily newspapers, adjust to 400 dpi for ancient newspapers due to dense text), ensure the details of small font text (such as below 8pt);

[0033] 3) Page segmentation: for newspapers with cross-page layout (such as serialized articles), automatically segment into single-page images by detecting the margin features (usually 1-2 cm of blank space).

[0034] (2) Special fine-tuning of multi-modal large model and extraction of layout elements

[0035] Adopting the architecture of "pre-trained model + special fine-tuning", the model output is optimized for the characteristics of newspaper layout:

[0036] Base model selection: CLIP-ViT-L / 14 is chosen as the base model, which can handle both image visual features and text semantic features due to its cross-modal understanding ability.

[0037] Fine-tuning dataset construction:

[0038] 1) The dataset covers a variety of newspapers from 1950 to 2023: including People's Daily (multi-column text), Reference News (vertical text), local evening newspapers (advertising mixed layout), professional journals (formula and text mixed layout), etc., with a total of 150,000 pages of samples;

[0039] 2) The annotation content includes: the bounding box coordinates of each element (accuracy to 1 pixel), type label (text block / picture block / column line / advertising box), attribute information (font / size / color of text block, picture block description text position).

[0040] Fine-tuning task design:

[0041] 1) Layout feature prediction: input image block, output column number (1-8 columns), column width (5-20 cm), column spacing (0.5-2 cm), use L1 loss function to optimize coordinate prediction accuracy;

[0042] 2) Semantic association modeling: through contrastive learning (Contrastive Learning), associate pictures with adjacent text blocks, such as "picture description below" and "chart corresponding data analysis text", use triplet loss (Triplet Loss) as the loss function;

[0043] 3) Type adaptive classification: for different types of newspapers (such as business newspapers / Party newspapers), dynamically adjust the classification weight (increase the advertising area recognition weight for business newspapers), and use attention mechanism to enhance the sensitivity to specific elements.

[0044] Layout element extraction process:

[0045] 1) Model output feature vector, including confidence (0-1), coordinates (x1, y1, x2, y2), and type label of each element;

[0046] 2) Additional output for text blocks: font (Songti / Black, etc.), font size (8-72pt), line spacing (1-3 times font size);

[0047] 3) Output of structural relationships: title level (first / second / third, based on font size, bold attributes, and position), graphic-text wrapping method (four-way type / tight type / embedded type, based on element boundary overlap calculation).

[0048] (3) Post-processing and optimization mechanism

[0049] Improve result reliability through multi-layer verification:

[0050] Rule engine verification: preset newspaper layout industry rule library, for example:

[0051] 1) The headline area must be located in the top 1 / 5 area of the page, and the width must be greater than or equal to 1 / 2 of the page width;

[0052] 2) The first-level title size is greater than or equal to 1.5 times the size of the regular text, and is usually centered or left-aligned;

[0053] 3) The edge of the advertisement area often contains decorative frames (dotted lines / patterns), and often contains keywords such as "advertisement" and "business invitation";

[0054] 4) Automatically mark the output that violates the rules (such as judging the headline as an advertisement area) as a low confidence result, triggering manual review.

[0055] Dynamic threshold adjustment: adjust the confidence threshold according to the image quality score (0-100 points, calculated based on clarity and noise ratio):

[0056] 1) High-quality image (score ≥ 80): threshold set to 0.8, priority to ensure precision;

[0057] 2) Low-quality image (score < 60): threshold set to 0.6, priority to ensure recall rate;

[0058] Cross-page consistency verification: for serialized article cross-page layout, associate the text blocks on the previous and next pages through text content similarity (cosine similarity ≥ 0.85) to ensure paragraph continuity.

[0059] Example two, on the basis of example one, proposes a newspaper complex layout structure identification system based on multi-modal large model, the system adopts modular design, including:

[0060] Input module: supports multiple formats (JPG / PNG / TIFF / PDF scans), automatically parses image metadata (resolution, color mode);

[0061] Preprocessing module: performs noise classification, denoising, geometric correction, and resolution normalization;

[0062] Model module: includes pre-trained model loading, fine-tuning parameter configuration, and feature vector output;

[0063] Post-processing module: implements rule engine verification, dynamic threshold adjustment, and cross-page consistency verification;

[0064] Storage module: structured data (element attributes, coordinates) is stored by MySQL, raw images and annotation results are stored by HDFS, and index is supported according to "newspaper name + publication date";

[0065] Output module: multi-format output (JSON / XML / editable PDF) is provided, wherein the PDF contains an editable annotation layer (supporting element boundary adjustment by dragging);

[0066] Monitoring module: real-time statistics of model accuracy (F1 value of test set is calculated every hour) and processing speed (single-page average time consumption is less than or equal to 1.5 seconds), when the accuracy is less than 85% for 2 hours in succession, automatic triggering of incremental training (1000 pages of error samples are extracted to join the fine-tuning data set).

[0067] Although the embodiments of the present application have been shown and described, it can be understood by those of ordinary skill in the art that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and spirits of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for structuring recognition of a complex layout of a newspaper based on a multi-modal large model, characterized in that: Comprising the following steps: Newspaper image preprocessing: A hierarchical processing flow is designed for the diversity of newspaper images, including noise classification and targeted denoising. Image feature analysis is used to identify speckle noise, crease noise, and ink diffusion types, and non-local mean filtering, morphological operation-based repair algorithm, and adaptive threshold segmentation are used for denoising respectively. Geometric correction and standardization are performed, including tilt correction by detecting newspaper edge straight lines with an error control within ±0.3°. Resolution normalization is performed by dynamically adjusting the resolution according to the newspaper type. Newspapers with cross-page layout are automatically segmented into single-page images by detecting the margin features. Special fine-tuning of multi-modal large models and extraction of layout elements: A "pre-trained model + special fine-tuning" architecture is used, with CLIP-ViT-L / 14 as the base model. A fine-tuning dataset covering 1950-2023 diverse newspaper samples is constructed, with annotated content including the bounding box coordinates of each element, type label, and attribute information. Fine-tuning tasks are designed for layout feature prediction, semantic association modeling, and type adaptive classification. The model output includes a feature vector containing the confidence, coordinates, and type label of each element. For text blocks, additional outputs include font, font size, and line spacing. For structural relationships, the output includes title level and graphic-text wrapping method. Post-processing and optimization: Rule engine verification is performed with a pre-set newspaper layout industry rule library. Outputs that violate the rules are automatically marked as low-confidence results and trigger manual review. Confidence thresholds are dynamically adjusted based on image quality scores. For cross-page layout of serialized articles, the text blocks are associated based on text content similarity to ensure paragraph continuity.

2. The multi-modal large model-based newspaper complex layout structure identification method according to claim 1, characterized in that: In noise classification and targeted denoising: Noise type identification uses image feature analysis, specifically analyzing gray variance and edge continuity to identify speckle noise, crease noise, and ink diffusion. Non-local mean filtering is used for speckle noise removal, preserving text edges while removing isolated noise points. A morphological operation-based repair algorithm is used for crease noise removal, first detecting the crease contour and then filling it through neighborhood pixel interpolation. Adaptive threshold segmentation is used for ink diffusion denoising, adjusting the binary threshold based on local gray mean.

3. The multi-modal large model-based newspaper complex layout structure identification method according to claim 2, characterized in that: In special fine-tuning of multi-modal large models: The fine-tuning dataset covers 1950-2023 diverse newspaper samples, including People's Daily, Reference News, local evening newspapers, and professional journals, with a total sample size of 150,000 pages. Annotated content includes bounding box coordinate precision to 1 pixel, type labels including text blocks / picture blocks / column lines / advertising frames, and attribute information including text block font / size / color and picture block description text location. In the fine-tuning task, layout feature prediction inputs image blocks and outputs column number, column width, and column spacing, with L1 loss function used to optimize coordinate prediction accuracy. Semantic association modeling associates pictures with adjacent text blocks through contrastive learning, with a triplet loss function. Type adaptive classification dynamically adjusts classification weights for different newspaper types, with an attention mechanism to enhance sensitivity to specific elements.

4. The multi-modal large model-based newspaper complex layout structure identification method according to claim 3, characterized in that: In the layout element extraction process, the model output feature vector contains the confidence, coordinates (x1, y1, x2, y2), and type label of each element. For text blocks, the font, font size, and line spacing are additionally output. For structural relationships, the title level and graphic-text wrapping method are output.

5. The multi-modal large model-based newspaper complex layout structure identification method according to claim 4, characterized in that: In the post-processing and optimization mechanism, the rule engine checks the preset newspaper layout industry rules, including that the headline area must be located in the top 1 / 5 area of the page and have a width ≥ 1 / 2 of the page width. The font size of the first-level headline is ≥ 1.5 times the font size of the regular text, and it is usually centered or left-aligned. The edge of the advertising area often contains a decorative border and often contains the keywords "advertisement" and "business invitation". The dynamic threshold adjustment adjusts the confidence threshold based on the image quality score. For high-quality images, the threshold is set to 0.8, prioritizing precision. For low-quality images, the threshold is set to 0.6, prioritizing recall. Cross-page consistency checking for serialized articles across pages associates the text blocks on the previous and next pages by text content similarity to ensure paragraph continuity.

6. The system of claim 5, wherein the method of structuring recognition of complex newspaper layout based on a multi-modal large model is characterized by: The system includes an input module, a preprocessing module, a model module, a post-processing module, a storage module, an output module, and a monitoring module. These modules work together to achieve structured recognition of complex newspaper layouts. The input module supports multiple formats and automatically parses image metadata. The preprocessing module performs noise classification, denoising, geometric correction, and resolution normalization. The model module includes pre-trained model loading, fine-tuning parameter configuration, and feature vector output functions. The post-processing module implements rule engine verification, dynamic threshold adjustment, and cross-page consistency checking. The storage module stores structured data, raw images, and annotation results in a specific way. The output module provides multiple formats, including JSON, XML, and editable PDF. The PDF output includes editable annotation layers, allowing users to adjust element boundaries by dragging, meeting the further editing needs of users.

7. The system of claim 6, wherein: The monitoring module calculates the accuracy of the model in real time and computes the F1 value of the test set every hour. It also monitors the processing speed to ensure that the average time per page is ≤ 1.5 seconds. When the accuracy is below 85% for two consecutive hours, the system automatically triggers incremental training by extracting 1000 pages from error samples to the fine-tuning dataset to optimize model performance.

8. The system of claim 7, wherein: ​ 9. The system of claim 8, wherein: ​ 10. The system of claim 9, wherein: ​

Images