A method, system, medium, and product for information analysis of graphic and textual materials.

Abstract

This invention discloses a method, system, medium, and product for information parsing of graphic and textual materials, belonging to the field of graphic and textual analysis. The method comprises: acquiring graphic and textual materials, and performing visual recognition and character recognition on the materials to obtain structured text data and plain text data; extracting structured information corresponding to a preset information dimension based on the structured text data or plain text data to obtain a structured information set; performing existence verification and numerical consistency verification on the structured information set based on the graphic and textual materials and a preset multimodal model to obtain structured verification information data; and associating each information item in the structured verification information data with its pixel coordinates or text reference identifier in the graphic and textual materials to generate a structured parsing result with a traceability identifier. By implementing this invention, the problems of low accuracy and difficulty in verifying information extraction from complex-format graphic and textual materials in the prior art can be solved.

Description

Technical Field

[0001] This invention belongs to the field of image and text analysis, and relates to a method, system, medium and product for information analysis of image and text materials. Background Technology

[0002] In fields such as finance, government affairs, and law, a large amount of business information exists in the form of unstructured materials with mixed text and graphics and diverse layouts, such as fund marketing posters, product brochures, or compliance announcements. These materials typically contain tables, multi-column layouts, and a mix of charts and explanatory text, and their informational value highly depends on the combination of visual placement and semantic context. To support automated processing, the key data in such materials needs to be transformed into a machine-readable structured format.

[0003] Currently, mainstream methods typically convert images into plain text using optical character recognition (OCR), and then extract information from them using rule engines or natural language models. However, this process loses the original layout structure in the first step, resulting in the loss of key contextual information such as table row and column relationships, column affiliation, and the association between text and charts. Rule engines struggle to handle flexible expressions, while general-purpose models, lacking layout constraints, are prone to misinterpreting spatial relationships, leading to "misattribution" extraction errors. This fails to ensure that the results truly originate from the original material, resulting in low parsing accuracy and poor reliability, making it difficult to meet the automated application needs of demanding scenarios. Summary of the Invention

[0004] This application provides a method, system, medium, and product for information parsing of graphic and textual materials, which can solve the problems of low accuracy and difficulty in verification of information extraction from complex graphic and textual materials in the prior art.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a method for information parsing of graphic and textual materials, comprising: Obtain text and image materials, and perform visual recognition and character recognition on the text and image materials respectively to obtain structured text data and plain text data; Based on the structured text data or plain text data, extract the structured information corresponding to the preset information dimensions to obtain a set of structured information. Based on the aforementioned graphic and textual materials and a preset multimodal model, the existence verification and numerical consistency verification of the structured information set are performed to obtain structured verification information data. Each information item in the structured verification information data is associated with its pixel coordinates or text reference identifier in the graphic material to generate a structured parsing result with a source tracing identifier.

[0006] Compared with existing technologies, the embodiments of this application have the following beneficial effects: By acquiring graphic and textual materials and performing visual and character recognition on these materials respectively, structured text data and plain text data are obtained, preserving the original graphic and textual layout structure information and accurate character-level text, avoiding structural mismatch or text distortion caused by a single recognition method; based on the structured text data or plain text data, structured information corresponding to the preset information dimensions is extracted, and the on-demand extraction strategy of the optimal text source can be dynamically selected according to the information type, improving the extraction accuracy of different information dimensions; based on the graphic and textual materials and the preset multimodal model, the existence verification and numerical consistency verification of the structured information set are performed, constructing... An internal fact consistency verification mechanism that does not rely on external knowledge filters out false information caused by model illusions. Each verified information item is associated with its pixel coordinates or text reference identifier in the graphic material to generate a structured parsing result with traceability identifier, thus achieving auditability and traceability of the extraction result. The above features work together to solve sub-problems such as structure preservation, text fidelity, on-demand source selection, result credibility, and process traceability. Through a technical closed loop of dual-source identification, source selection extraction, multimodal self-verification, and dual-modal traceability, the problem of low accuracy and unverifiable structured information extraction in complex graphic materials is solved, enabling the automated parsing result to have both high accuracy and high credibility.

[0007] In some embodiments of the first aspect of this application, the step of acquiring graphic and textual materials and performing visual recognition and character recognition on the graphic and textual materials respectively to obtain structured text data and plain text data includes: Obtain text and image materials, and divide the text and image materials into content blocks based on the layout structure to obtain several content block images; Visual recognition and character recognition are performed on each of the content block images to obtain structured text data and plain text data.

[0008] Compared with existing technologies, the above embodiments have the following beneficial effects: by dividing the graphic materials into content blocks based on the layout structure to obtain several content block images, and then performing visual recognition and character recognition on each content block image, the overall parsing task of large-size or long graphic materials is decomposed into multiple logically independent small image parallel processing units. This effectively avoids information truncation or content omission caused by the context length limitation of multimodal large models, while reducing the computational resource consumption of a single model call and improving the system's processing efficiency and robustness for complex graphic materials such as high-resolution marketing posters.

[0009] In some embodiments of the first aspect of this application, the step of dividing the graphic material into content blocks based on layout structure to obtain several content block images includes: Text detection is performed on the aforementioned text and image materials to obtain each text region; Based on each of the text regions, and combining font size, color, direction and semantic boundaries, each first content block is generated by aggregation, and a spatial relationship diagram is constructed according to the relative position and size relationship of each first content block; Based on the spatial relationship diagram and the preset cutting constraints, the decision tree algorithm is used to determine each cutting point, and the graphic material is cut along each cutting point to output several logically independent content block images.

[0010] Compared with existing technologies, the above embodiments have the following beneficial effects: text detection is performed on the graphic materials to obtain each text region, which is then aggregated to generate the first content block. A spatial relationship graph is constructed based on the relative position and size relationship of the content blocks to achieve semantic perception and structural understanding of the graphic content. Furthermore, the decision tree algorithm is combined with cutting constraints to determine the cutting points and cut the graphic materials. Logically independent content block images are generated while preserving the integrity of the information, providing a reliable input unit for subsequent high-precision parallel parsing.

[0011] In some embodiments of the first aspect of this application, the step of performing visual recognition and character recognition on each of the content block images to obtain structured text data and plain text data includes: Visual recognition and character recognition are performed on each of the content block images to obtain the original structured text data and the original plain text data; The original structured text data and the original plain text data are preprocessed to obtain the first structured text data and the plain text data; wherein, the preprocessing includes any one or more combinations of the following: noise filtering, format unification, and synonym and special character replacement; The table data in the first structured text data is filled with empty cells, aligned with rows and columns, and merged into multiple columns to generate a corresponding single-column linearized table. Each data item in the single-column linearized table is then converted into a key-value statement, along with the row header and column header attributes, to obtain the structured text data.

[0012] Compared with existing technologies, the above embodiments have the following beneficial effects: By preprocessing such as noise filtering, format unification, and replacement of synonyms and special characters, the standardization and consistency of text data are improved, and interference noise in subsequent information extraction is reduced; furthermore, empty cell filling, row and column alignment, and multi-column merging are performed on the tables in the preprocessed structured text data to generate single-column linear tables, and the table data items are converted into key-value statements that combine row header and column header attributes, reducing the two-dimensional table structure to a one-dimensional semantically clear linear representation, which greatly reduces the understanding burden of large language models on table content, effectively prevents the mismatch between indicators and entities caused by the complexity of table structure, and significantly improves the extraction accuracy of structure-sensitive information.

[0013] In some embodiments of the first aspect of this application, the step of extracting structured information corresponding to a preset information dimension based on the structured text data or plain text data to obtain a structured information set includes: Identify all core entities in the structured text data, and construct a global semantic context based on each core entity; Based on the global semantic context, the structured information of each core entity under the preset information dimension is extracted from the structured text data or the plain text data respectively, to obtain an initial set of structured information; All numerical values ​​are extracted from the text and image materials to obtain the original numerical value set, and the covered numerical values ​​are extracted from the initial structured information set to obtain the extracted numerical value set. By comparing the original set of values ​​with the extracted set of values, missing values ​​are identified. Based on the global semantic context and the context of the missing values, the large language model is guided to reverse reasoning the corresponding entities, indicators and time information to generate corresponding supplementary information items. The supplementary information items are combined with the initial structured information set to obtain the structured information set.

[0014] Compared with existing technologies, the above embodiments have the following beneficial effects: by identifying all core entities in structured text data and constructing a global semantic context, a comprehensive entity anchor system is established for information extraction; based on this global context, structured information of each entity in a preset dimension is extracted from structured text data or plain text data, realizing precise targeted extraction centered on entities; furthermore, by comparing all values ​​in the graphic and textual materials with the extracted value set, missing values ​​are identified, and based on the global semantic context and the missing value context, the large language model is guided to infer its corresponding entity, indicator, and time information in reverse, actively filling in the numerical omissions caused by task coverage blind spots, thereby ensuring the integrity of the main entity information while also ensuring the comprehensive capture of non-main entity related values, significantly improving the coverage breadth and data integrity of the structured parsing results.

[0015] In some embodiments of the first aspect of this application, the step of performing existence verification and numerical consistency verification on the structured information set based on the graphic materials and a preset multimodal model to obtain structured verification information data includes: For each information item in the structured information set, a closed-form verification problem is constructed, and the closed-form verification problem and the graphic material are input into the multimodal model to output the corresponding existence judgment result. For information items whose existence is affirmative, an open-ended numerical retrieval problem is constructed, and the open-ended numerical retrieval problem and the text and image materials are input into the multimodal model to obtain the corresponding reverse extraction values; The extracted values ​​are compared with the original values ​​in the corresponding information items. If they are completely consistent, the information item is retained as structured verification information data.

[0016] Compared with existing technologies, the above embodiments have the following beneficial effects: By constructing a closed-loop verification problem for each information item in the structured information set and inputting it into the multimodal model with the text and image materials, an existence judgment result is output, thus achieving a preliminary screening of whether the information item truly exists in the original material; furthermore, an open-loop numerical retrieval problem is constructed for the information items whose existence is confirmed and input into the multimodal model to obtain reverse extracted values. By comparing the consistency between the reverse values ​​and the original values, a two-stage numerical accuracy verification mechanism is constructed; this mechanism not only effectively eliminates false information generated by model illusion, but also ensures the absolute accuracy of the retained results through dual verification of numerical reverse retrieval and comparison, greatly improving the reliability of the final structured data.

[0017] In some embodiments of the first aspect of this application, the step of associating each information item in the structured verification information data with its pixel coordinates or text reference identifier in the graphic material to generate a structured parsing result with a source tracing identifier includes: For each information item, if the text and image material is in image format, a location request is constructed based on the corresponding attribute value and context description of the information item, and input into a preset visual language model along with the text and image material to obtain the pixel coordinate range of the information item in the text and image material; if the text and image material is in document format, frequency analysis is performed on the attribute values ​​of each information item: if the attribute value appears uniquely in the text and image material, the attribute value is located in the text and image material through string matching, and a unique reference identifier is injected at the corresponding position of the attribute value; if the attribute value appears repeatedly in the text and image material or the format is ambiguous, a task instruction containing context is constructed based on the attribute value, and input into a preset language model along with the text and image material to obtain the position of the information item in the text and image material and inject a unique reference identifier. Each pixel coordinate range or reference identifier is used as a tracing field and appended to the corresponding information item to generate a structured parsing result with a tracing identifier.

[0018] Compared to existing technologies, the above embodiments have the following beneficial effects: By processing each information item according to the format of the graphic and textual materials: for image format materials, a positioning request is constructed and input into a visual language model to obtain the pixel coordinate range, thus achieving precise spatial positioning of visual content; for document format materials, attribute value frequency analysis is first performed, and unique attribute values ​​are located by string matching and a reference identifier is injected; for repeated or ambiguous attribute values, a task instruction with context is constructed and input into a language model to obtain the position and an identifier is injected; this dual-modal hybrid tracing strategy takes into account both the pixel-level accuracy of image materials and the semantic disambiguation capability of document materials, ensuring that the results extracted from various graphic and textual materials can be accurately and uniquely traced back to the original source, greatly enhancing the verifiability and audit credibility of the structured parsing results.

[0019] Secondly, the present invention also provides an information analysis system for graphic and textual materials, comprising: a data acquisition module, an extraction module, a verification module, and an output module; The data acquisition module is used to acquire graphic and textual materials, and to perform visual recognition and character recognition on the graphic and textual materials to obtain structured text data and plain text data. The extraction module is used to extract structured information corresponding to a preset information dimension based on the structured text data or plain text data, and obtain a set of structured information. The verification module is used to perform existence verification and numerical consistency verification on the structured information set based on the graphic materials and the preset multimodal model, so as to obtain structured verification information data. The output module is used to associate each information item in the structured verification information data with its pixel coordinates or text reference identifier in the graphic material, and generate a structured parsing result with a traceability identifier.

[0020] Compared with existing technologies, the above embodiments of this application have the following beneficial effects: By acquiring graphic and textual materials and performing visual and character recognition on these materials respectively, structured text data and plain text data are obtained, preserving the original graphic and textual layout structure information and accurate character-level text, avoiding structural mismatch or text distortion caused by a single recognition method; based on the structured text data or plain text data, structured information corresponding to the preset information dimensions is extracted, and the on-demand extraction strategy of the optimal text source can be dynamically selected according to the information type, improving the extraction accuracy of different information dimensions; based on the graphic and textual materials and the preset multimodal model, the existence verification and numerical consistency verification of the structured information set are performed, constructing... An internal fact consistency verification mechanism independent of external knowledge is established to filter out false information caused by model illusions. Each verified information item is associated with its pixel coordinates or text reference identifier in the graphic material to generate a structured parsing result with traceability identifier, realizing the auditability and traceability of the extraction result. The above features work together to solve sub-problems such as structure preservation, text fidelity, on-demand source selection, result credibility, and process traceability. Through a technical closed loop of dual-source identification, source selection extraction, multimodal self-verification, and dual-modal traceability, the problem of low accuracy and unverifiable structured information extraction in complex graphic materials is solved, so that the automated parsing result has both high accuracy and high credibility.

[0021] Thirdly, the present invention also provides a computer program product, including a computer program or instructions, characterized in that, when the computer program or instructions are executed, they implement the information parsing method for any type of graphic material of the present invention.

[0022] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the graphic materials information parsing methods of this invention. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating a method for parsing information from graphic materials provided in some embodiments of the present invention.

[0024] Figure 2 This is a schematic diagram of the structure of an information parsing system for graphic materials provided in some embodiments of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Example 1: Please refer to Figure 1 To address the problems of low accuracy and difficulty in verifying information extraction from complex graphic materials in existing technologies, an embodiment of the present invention provides an information parsing method for graphic materials, comprising steps S1 to S4: Step S1: Obtain graphic and textual materials, and perform visual recognition and character recognition on the graphic and textual materials respectively to obtain structured text data and plain text data.

[0027] Furthermore, step S1 can be implemented through the following preferred embodiments, including steps S11-S12, as follows: S11: Obtain graphic and textual materials, and divide the graphic and textual materials into content blocks based on the layout structure to obtain several content block images; S12: Perform visual recognition and character recognition on each of the content block images to obtain structured text data and plain text data.

[0028] In this preferred embodiment, the graphic material is divided into several content block images based on the layout structure. Then, visual recognition and character recognition are performed on each content block image. The overall parsing task of large-size or long graphic materials is decomposed into multiple logically independent small image parallel processing units. This effectively avoids information truncation or content omission caused by the context length limitation of multimodal large models. At the same time, it reduces the computational resource consumption of a single model call and improves the system's processing efficiency and robustness for complex graphic materials such as high-resolution marketing posters.

[0029] Furthermore, step S11 can be implemented through the following preferred embodiments, including steps S111-S113, as follows: S111: Perform text detection on the graphic materials to obtain each text region; S112: Based on each of the text regions, and combining font size, color, direction and semantic boundaries, aggregate to generate each first content block, and construct a spatial relationship diagram according to the relative position and size relationship of each first content block; S113: Based on the spatial relationship diagram and the preset cutting constraints, each cutting point is determined by the decision tree algorithm, and the graphic material is cut along each cutting point to output several logically independent content block images.

[0030] In this preferred embodiment, text detection is performed on the graphic material to obtain each text region, which is then aggregated to generate the first content block. A spatial relationship graph is constructed based on the relative position and size relationship of the content blocks to achieve semantic perception and structural understanding of the graphic content. Furthermore, the decision tree algorithm is used in combination with cutting constraints to determine the cutting points and cut the graphic material. Logically independent content block images are generated while preserving the integrity of the information, providing a reliable input unit for subsequent high-precision parallel parsing.

[0031] Furthermore, step S12 can be implemented through the following preferred embodiments, including steps S121-S123, as follows: S121: Perform visual recognition and character recognition on each of the content block images to obtain the original structured text data and the original plain text data; S122: Preprocess the original structured text data and the original plain text data to obtain the first structured text data and the plain text data; wherein, the preprocessing includes any one or more combinations of the following: noise filtering, format unification, and synonym and special character replacement; S123: Fill empty cells, align rows and columns, and merge multiple columns in the table data of the first structured text data to generate a corresponding single-column linearized table. Then, convert each data item in the single-column linearized table into a key-value statement in combination with the row header and column header attributes to obtain the structured text data.

[0032] In this preferred embodiment, preprocessing such as noise filtering, format unification, and replacement of synonyms and special characters improves the standardization and consistency of text data and reduces interference noise in subsequent information extraction. Furthermore, the tables in the preprocessed structured text data are filled with empty cells, aligned with rows and columns, and merged into single-column linear tables. The table data items are converted into key-value statements that combine row header and column header attributes, reducing the two-dimensional table structure to a one-dimensional, semantically clear linear representation. This greatly reduces the burden on the large language model to understand the table content, effectively prevents the mismatch between indicators and entities caused by the complexity of the table structure, and significantly improves the extraction accuracy of structure-sensitive information.

[0033] Step S2: Based on the structured text data or plain text data, extract the structured information corresponding to the preset information dimension to obtain a set of structured information.

[0034] Furthermore, step S2 can be implemented through the following preferred embodiments, including steps S21-S25, as detailed below: S21: Identify all core entities in the structured text data and construct a global semantic context based on each core entity; S22: Based on the global semantic context, extract the structured information of each core entity under the preset information dimension from the structured text data or the plain text data respectively, and obtain an initial set of structured information; S23: Extract all values ​​from the graphic and textual materials to obtain the original value set, and extract the covered values ​​from the initial structured information set to obtain the extracted value set; S24: Compare the original set of values ​​with the extracted set of values ​​to identify missing values, and based on the global semantic context and the context of the missing values, guide the large language model to reverse reason about the corresponding entities, indicators and time information to generate corresponding supplementary information items. S25: Merge each of the supplementary information items with the initial structured information set to obtain the structured information set.

[0035] In this preferred embodiment, by identifying all core entities in structured text data and constructing a global semantic context, a comprehensive entity anchor system is established for information extraction. Based on this global context, structured information of each entity in a preset dimension is extracted from structured text data or plain text data, achieving precise and targeted extraction centered on entities. Furthermore, by comparing all values ​​in the text and image materials with the extracted value set, missing values ​​are identified. Based on the global semantic context and the missing value context, the large language model is guided to infer the corresponding entity, indicator, and time information in reverse, actively filling in the numerical omissions caused by task coverage blind spots. Thus, while ensuring the integrity of the main entity information, it also ensures the comprehensive capture of non-main entity related values, significantly improving the coverage breadth and data integrity of the structured parsing results.

[0036] Step S3: Based on the graphic materials and the preset multimodal model, perform existence verification and numerical consistency verification on the structured information set to obtain structured verification information data.

[0037] Furthermore, step S3 can be implemented through the following preferred embodiments, including steps S31-S33, as follows: S31: For each information item in the structured information set, construct a closed-form verification problem, and input the closed-form verification problem and the graphic material into the multimodal model, and output the corresponding existence judgment result; S32: For information items whose existence judgment result is affirmative, construct an open numerical retrieval problem, and input the open numerical retrieval problem and the graphic material into the multimodal model to obtain the corresponding reverse extraction value; S33: Compare the extracted values ​​with the original values ​​in the corresponding information items. If they are completely consistent, the information item is retained as structured verification information data.

[0038] In this preferred embodiment, a preliminary screening of whether an information item truly exists in the original material is achieved by constructing a closed-loop verification problem for each information item in the structured information set and inputting it into a multimodal model along with the text and image materials to output an existence judgment result. Further, an open-loop numerical retrieval problem is constructed for the information items whose existence is confirmed and input into the multimodal model to obtain reverse-extracted numerical values. By comparing the consistency between the reverse values ​​and the original values, a two-stage numerical accuracy verification mechanism is constructed. This mechanism not only effectively eliminates false information generated by model illusions but also ensures the absolute accuracy of the retained results through dual verification of reverse numerical retrieval and comparison, significantly improving the reliability of the final structured data.

[0039] Step S4: Associate each information item in the structured verification information data with its pixel coordinates or text reference identifier in the graphic material to generate a structured parsing result with a source identifier.

[0040] Furthermore, step S4 can be implemented through the following preferred embodiments, including steps S41-S42, as follows: S41: For each information item, if the graphic material is in image format, a positioning request is constructed based on the attribute value and context description corresponding to the information item, and input into a preset visual language model along with the graphic material to obtain the pixel coordinate range of the information item in the graphic material; if the graphic material is in document format, frequency analysis is performed on the attribute values ​​of each information item: if the attribute value appears uniquely in the graphic material, the attribute value is located in the graphic material through string matching, and a unique reference identifier is injected at the corresponding position of the attribute value; if the attribute value appears repeatedly in the graphic material or the format is ambiguous, a task instruction containing context is constructed based on the attribute value, and input into a preset language model along with the graphic material to obtain the position of the information item in the graphic material and inject a unique reference identifier. S42: The pixel coordinate range or reference identifier is used as a tracing field and appended to the corresponding information item to generate a structured parsing result with a tracing identifier.

[0041] In this preferred embodiment, each information item is processed according to the format of the graphic and textual material: for image format materials, a positioning request is constructed and input into a visual language model to obtain the pixel coordinate range, thus achieving precise spatial positioning of the visual content; for document format materials, attribute value frequency analysis is first performed, and unique attribute values ​​are located by string matching and a reference identifier is injected; for repeated or ambiguous attribute values, a task instruction with context is constructed and input into a language model to obtain the position and an identifier is injected. This dual-modal hybrid tracing strategy takes into account both the pixel-level accuracy of image materials and the semantic disambiguation capability of document materials, ensuring that the extraction results of various graphic and textual materials can be accurately and uniquely traced back to the original source, greatly enhancing the verifiability and audit credibility of the structured parsing results.

[0042] In practical implementation, to more clearly illustrate the solution of this invention, we will use the graphic and textual promotion of fund products in the financial field as a specific preferred implementation scenario for illustration: Step 1: Input Processing Stage The system supports a wide range of input formats, including but not limited to image files such as PNG and JPG, documents such as Markdown, Word, and PDF, as well as text strings or file paths.

[0043] (1) Document format standardization: For the received non-image document formats (such as Word, PDF, TXT), the system will call a dedicated file conversion interface to directly extract information from the internal structure of the file. Through this step, the system will uniformly convert document formats from various sources into structured Markdown text, providing standardized input for the subsequent information extraction module.

[0044] Word document processing: It parses the document object model of ".docx" files through a dedicated interface, and directly extracts structured information such as text content, tables, and heading levels.

[0045] PDF document processing: Using appropriate parsing libraries, the system prioritizes extracting text stream data and embedded table objects from PDF files. For PDFs generated from scanned images, the system automatically recognizes them and transfers them to subsequent image file processing flows.

[0046] (2) Image file preprocessing: To address industry challenges such as inconsistent image quality and complex backgrounds, the system performs a series of image preprocessing operations before sending the images to VLM, serving as a key component of the solution: Image format standardization: Input images of different formats are converted into a standard processing format and their size is adjusted to suit the optimal processing requirements of the model.

[0047] Image grayscale conversion: Converting a color image to a grayscale image to eliminate the interference of color on structure recognition.

[0048] Quality enhancement and optimization: Image enhancement algorithms are applied to optimize low-quality images with low resolution, insufficient contrast, or noise in order to improve the accuracy of subsequent recognition.

[0049] (3) Intelligent Image Segmentation Based on Visual Layout: After overall image preprocessing and before being sent to the core parsing engine, this invention introduces a key optimization step to achieve precise image segmentation. This aims to solve industry problems such as long processing time, high resource consumption, and easy loss of information (large language models often summarize or truncate long text outputs) caused by large-size marketing posters and long text / image formats, and is adaptable to different types of marketing materials. The complete algorithm flow is as follows: 3.1 Image Preprocessing and Feature Extraction: The Canny edge detection algorithm is used to extract the main edge information in the image, and a deep learning-based text detection model is used to identify all text lines and their bounding boxes in the image; continuous regions in the image are identified through connected component analysis algorithm; and effective background separation and text region enhancement techniques are used to eliminate interference from complex background patterns and decorative elements.

[0050] 3.2 Global Structure Grid Analysis: Based on the distribution density of text regions, a text density heatmap of the image is generated. Hough transform is used to detect straight lines in the image, especially vertical and horizontal dividing lines, identifying obvious column dividing lines, background texture changes, and the spacing between content blocks. Based on the detected dividing lines and density distribution, the overall grid structure of the image is inferred, determining the poster's layout pattern (single-column, double-column, triple-column, table-type, hybrid, etc.), as well as left-right structure, top-bottom structure, and center structure.

[0051] 3.3 Local Content Block Aggregation: DBSCAN clustering is performed on the detected text regions. Adjacent text blocks are aggregated based on spatial distance. The aggregation results are fine-tuned based on the similarity of features such as font size, color, and orientation of adjacent text blocks (e.g., merging titles and body text, charts and explanatory text, etc., into the same content block). Finally, natural language processing techniques are used to analyze the text content, identify semantic boundaries (e.g., paragraph ends, topic transitions, etc.), and merge the clustered text regions into logically complete content blocks.

[0052] 3.4 Structural Relationship Modeling: Using the identified content blocks as nodes, construct a spatial relationship graph. By analyzing the relative positions and size relationships of the content blocks, establish a hierarchical structure.

[0053] 3.5 Intelligent Cutting Point Decision Algorithm: Based on multi-level analysis results, a decision tree algorithm is used to determine the optimal cutting point. Do not cut the text: The cutting line will never pass through the middle of the text line.

[0054] Without disrupting the column: The cutting line will follow the natural boundaries of the column and will not split a complete table or paragraph into two different modules.

[0055] Do not span content: Ensure that each generated small module contains one or more semantically complete units.

[0056] Cutting point generation: Cut the original large image into several logically independent content blocks.

[0057] Overlapping area handling: Preserve appropriate overlapping areas between adjacent blocks to ensure that information is not lost.

[0058] Sequence consistency check: Ensures that the cut image blocks maintain their original logical order.

[0059] This algorithm divides the original large image into several logically independent, smaller image modules, following a top-to-bottom and left-to-right order. This provides the foundation for parallel processing in subsequent dual-engine parsing and is key to the system's high efficiency. Simultaneously, the segmentation algorithm ensures the correctness and integrity of the content order, preventing information fragmentation caused by disordered order after image segmentation.

[0060] (4) Image file structure conversion: All image modules that have undergone intelligent block segmentation will be fed in parallel into an intelligent image-text conversion module driven by a dual-engine hybrid of "VLM + high-precision OCR". In existing technologies, single technical routes often have limitations: VLM performs excellently in understanding the layout structure of tables, charts and other formats, but may have minor text recognition errors when processing large and dense texts; while traditional high-precision OCR is the opposite, it can guarantee extremely high text recognition accuracy, but cannot understand the layout structure.

[0061] Therefore, this invention employs a complementary dual-engine parallel parsing approach. The system simultaneously feeds the same input image into a visual language model and a high-precision OCR engine for parallel processing, generating two independent parsing results. Downstream tasks automatically select the text source: in the subsequent fourth step of information extraction, the system intelligently selects the most suitable text source as input based on the characteristics of different extraction tasks.

[0062] Visual Language Model (VLM) recognizes text: a Markdown document that retains its complete layout structure (such as tables, lists, and heading levels). For structure-sensitive information (such as fund indicators, component ratios, and fund fees), the system uses structured Markdown generated by VLM. Because this type of information highly depends on the row and column correspondences of tables and the accurate association with the context, VLM's structural understanding ability is crucial to ensuring the correctness of numerical relationships.

[0063] Optical Character Recognition (OCR) text recognition: A plain text file containing high-fidelity recognition of all text in the image. For text-sensitive information (such as the full name of a security entity, fund company information, fund manager introductions, risk warnings, and other long explanatory texts), the system uses plain text generated by high-precision OCR. This ensures that factual deviations are not caused by errors in the recognition of a single character when extracting this key descriptive information.

[0064] By employing this dual-engine hybrid and on-demand optimization strategy, this system organically combines the structural understanding capabilities of VLM with the text fidelity capabilities of OCR, laying the most solid foundation for subsequent accurate information extraction.

[0065] (5) Refined processing of specific graphic elements: Table Conversion: For the tables in the image, the system will strictly follow their original row and column relationships to accurately convert them into Markdown table format, fully preserving their structured information.

[0066] Chart Analysis and Conversion: The system can intelligently analyze charts (such as bar charts and line charts). If the chart itself or its annotations indicate specific data points, the system will parse and convert them into a structured table, with the table's dimensions corresponding to the chart's axes and legend. If the chart only has scales without specific values, it will be recognized and output as a text placeholder "[Image]" to avoid misinterpretation of information.

[0067] Title and decorative element recognition: The system identifies titles through visual features (such as unique background colors or borders) to ensure they are not confused with ordinary text content. Simultaneously, it can convert decorative icons in posters (such as circles and stars) into corresponding text symbols (such as "●" and "★"), retaining their semantic function.

[0068] Retention of annotation information: For annotation markers in the text (such as superscript and subscript), the system will retain them in their original positions and enclose them in special brackets (such as "[Note 3]") to ensure that the reference relationship between the main text and the footnotes is not lost.

[0069] (6) Content integrity and formatting rules: Information Fidelity: The system ensures that all text information extracted from images is preserved without addition, omission, or modification. Even if the same or similar content appears in multiple places, it will all be retained to maintain the integrity of the original information.

[0070] Cross-line text aggregation: The system can understand the context and correctly merge the same semantic unit that spans multiple lines (such as the date "2023 year\nfirst half") into "2023 first half".

[0071] Standardized output: The final output of the entire conversion process is strictly limited to pure Markdown format, without any HTML tags, to ensure compatibility and standardization in subsequent processing.

[0072] By executing the above series of complex parsing and transformation logics, this system can transform raw, unstructured text and image materials into a highly structured, information-complete Markdown document that retains the original format, laying a solid foundation for subsequent accurate information extraction.

[0073] Step 2: Document Extraction Preprocessing: While the Visual Language Model (VLM) in Step One boasts powerful conversion capabilities, its output is not without flaws. When processing complex text and images, the model may still exhibit a degree of "illusion," generating seemingly plausible but ultimately inaccurate content. Furthermore, its underlying Optical Character Recognition (OCR) capabilities may misidentify decorative icons, complex background textures, or logos as meaningless text characters, introducing "recognition noise." Therefore, the core purpose of this preprocessing step is to systematically clean, correct, and deeply format the standardized Markdown text output from Step One, eliminating the aforementioned uncertainties and providing a clean and reliable data foundation for subsequent accurate information extraction. This module possesses the following core functions: (1) Text content cleaning and standardization: Noise Removal and Format Standardization: The system automatically cleans up irrelevant content in the recognition results, such as garbled characters, special characters, and introductory or summary text that the model might erroneously output. Simultaneously, it standardizes numerical values ​​and dates. For example, it automatically converts "June 30, 2024" in text to the standard format "20240630".

[0074] Synonym and Special Character Replacement: A flexible, configurable mapping system unifies inconsistent terminology and expressions (such as "yield" and "return"). It also handles HTML escape characters and non-standard newline characters, such as replacing "..." with "...". Replace “” and “\\n” with the standard newline character “\n” to ensure text purity.

[0075] (2) Structured Preprocessing Engine for Complex Tables: This invention includes a powerful, multi-stage table processing engine that corrects structural errors (such as misaligned rows and columns, and misaligned layout) that may be introduced by the upstream VLM during the "image-to-table" process. Through a series of structural transformations, it converts complex two-dimensional tables into a one-dimensional linear format that is easily processed by the subsequent information extraction model (LLM). This preprocessing step greatly reduces the understanding burden of LLM and avoids the "mismatch" problem (i.e., incorrect matching of values ​​and indicators) caused by incorrect alignment of rows and columns in the model. It is a key step in ensuring the accuracy of the final information extraction.

[0076] Structural integrity restoration: This is the first stage, which aims to repair table structure defects caused by OCR recognition errors or non-standard original layout.

[0077] Empty cell filling: Automatically detects empty cells such as "| |" and fills them with a uniform placeholder (such as "---"). This prevents data column misalignment due to missing values.

[0078] Row and column alignment: Count all rows in the table and find the row with the most columns as the baseline. Then, for all rows with fewer than 10 columns, automatically add cells at the end until the number of columns matches the baseline row. This ensures the table's complete and aligned structure, which is the basis for subsequent processing, such as when the table recognized by the input VLM model is incomplete. ; Output (after repair): .

[0079] Layout Conversion: For side-by-side tables (two or more columns) designed to save space, the system intelligently identifies and converts them into standard single-column tables. Identification Logic: Conversion is triggered by detecting duplicate column name patterns in the header row (e.g., "| Name| Code| Name| Code|"). Conversion Logic: The system appends the data rows from the right column sequentially below the data rows from the left column, using the left column's header as the new unified header. For example, inputting a two-column table: ; Output (single-column table): .

[0080] Structure Transpose: For special tables where the first column is time or date (time series tables), the system will automatically transpose, swapping rows and columns. Recognition Logic: The system checks the values ​​in the first column of the table using regular expressions. If more than 50% of the cell content conforms to a date or year format (e.g., "YYYY-MM-DD", "YYYY year"), it is determined to be a time series table. Conversion Logic: The original first column (time) is converted into a new header row, and the original header row is converted into a new first column (indicator name). For example, inputting a time series table: Output (after transpose): .

[0081] Semantic linearization: After completing all the above structured processing, the well-organized two-dimensional table is completely transformed into a series of one-dimensional, LLM-friendly key-value statements. Transformation rules: The system iterates through each data cell (excluding the header) of the table, concatenating its row header, column header, and cell value into a complete natural language description. Concatenation format: "{row header} - {column header} is {cell value}".

[0082] Special Logic: The system also has built-in optimization for concatenating specific keywords. For example, if the header of a line is "performance benchmark", the system will automatically associate it with the header of the previous line (usually the fund name) to generate a more complete description such as "the performance benchmark of YiXda Anxin Return Bond A - 3.5% in 2023".

[0083] Final result: Through the above series of transformations, the complex table in the original image is finally transformed into a text with a clear structure and explicit semantics, which greatly reduces the difficulty and error rate of subsequent LLM information extraction.

[0084] Step 3: Global Context Construction and Core Entity Recognition The system first performs a full scan of the preprocessed document to identify all core financial entities (such as funds, indices, and stocks), establishing a global context for subsequent refined extraction. This process employs a hybrid strategy of "large-scale model initial screening + rule refinement," and incorporates multi-level precise extraction and decision-making logic. Entity Attribute Extraction: The system is configured to extract key attributes for each entity, such as its code (usually a 6-digit number or alphanumeric combination), abbreviation, and performance benchmark. For the performance benchmark, the system will standardize its output format, prioritizing the use of the full name followed by the code.

[0085] Advertising Subject Decision Algorithm: To focus on the most important information, the system incorporates a decision algorithm to determine the unique subject of advertising materials. This algorithm prioritizes products explicitly stated in the text as being about to be released or being heavily promoted. If no explicit marker is found, it uses entity attributes (whether it contains a stock code, name length comparison, etc.) to ensure that exactly one subject is identified.

[0086] Multi-threaded parallel extraction and consensus mechanism: To improve robustness, the system sends extraction requests to the Large Language Model (LLM) in parallel. By sorting and aggregating the returned results, the system prioritizes the most comprehensive and richest results as the basis for subsequent processing, effectively reducing the risk of illusions or omissions that may occur in a single model call.

[0087] Deep post-processing and data refinement: The model's raw output undergoes a rigorous post-processing and cleaning process to ensure data accuracy and consistency.

[0088] Noise filtering: The system will automatically filter out redundant information related to the sample in the model output, entity names that are too short or obviously irrelevant (such as fund manager names being misidentified as fund names).

[0089] Attribute cleaning: The extracted core attributes such as "code", "abbreviation", and "benchmark" are validated and cleaned. For example, attributes with empty or meaningless content are removed, performance benchmarks with non-standard formats are corrected, and it is ensured that the "abbreviation" is an alias created by the model based on the full name of the entity.

[0090] Related Completion: The system can understand the relationships between entities. For example, if both "EasyX Fund" and its abbreviation "XX Fund" are extracted, the system will automatically supplement the latter with information such as the former's code and benchmark, ensuring the completeness of the information.

[0091] Share handling: The system can intelligently identify different share classes of the same fund (such as Class A and Class C). Even if the model fails to extract the complete share class, as long as it appears in the original text, the system will automatically fill in the missing share class entries and verify whether the codes of different share classes are incorrectly identified as the same.

[0092] Two-stage code recognition and completion: To maximize code coverage and accuracy, the system employs a two-stage strategy: Text extraction: Prioritize extracting the security codes that appear alongside the entity name directly from the original text.

[0093] External search completion: For entities whose securities codes are not explicitly identified in the original text, the system will automatically call external search engines to perform a query and store the retrieved codes as "supplementary codes", thereby enhancing the information dimension.

[0094] Step 4: Parallelized Multidimensional Information Extraction After constructing the global context in step three, the system launches multiple dedicated information extraction units in parallel. Each unit is an independent "domain expert," responsible for a specific information dimension, and selects the most suitable text source (high-fidelity OCR text or VLM structured text) generated in step one as the extraction basis based on the characteristics of the information type. This "divide and conquer" strategy allows each model to focus on a smaller, more specific task, thereby significantly improving the accuracy and depth of extraction. The following is a detailed description of each core extraction unit: (1) Extraction of fund manager information: Focus on extracting detailed information of all current and former fund managers of the main fund, such as educational background, work experience, investment philosophy and other lengthy descriptions. The value of information is reflected in the complete semantic expression. Errors in a single word may lead to factual deviations. High-fidelity OCR text is used for extraction.

[0095] Extraction logic: The model is guided to distinguish between two types of core information: 1) Static attributes, such as the fund manager's degree and work experience; 2) Dynamic (time-sensitive) attributes, such as years of experience, investment experience, and fund size under management. The model is explicitly required to associate dynamic attributes with the data cutoff date mentioned in the documentation.

[0096] Logical consistency check: The system will check whether the extracted years data is logically consistent (e.g., "Years of work experience ≥ Investment research experience ≥ Investment experience") and remove contradictory information that does not conform to logic.

[0097] Appointment status determination: By analyzing the "term of office" field (such as "to date" or the specific date of departure), the system can automatically determine and distinguish between "current" and "former" fund managers.

[0098] Anti-confusion mechanism: In response to the rich and diverse nature of poster marketing language, the system has built-in rules to prevent the model from confusing similar concepts. For example, if the model extracts a description of "work experience" that ends with "...fund manager of products" and does not contain job-related words, the system will correct the indicator type to "funds under management".

[0099] (2) Extraction of fund company information: Extract relevant information of the fund company to which the fund entity belongs, such as the company's performance description, investment period, business scope and other explanatory text, and use high-fidelity OCR text for extraction.

[0100] Extraction logic: Similar to the extraction by fund managers, the model is guided to distinguish between the static attributes (such as the nature of the company) and dynamic attributes (such as the length of time since its establishment, the number of years of investment in various businesses, and the awards received).

[0101] Logical consistency check: A strict age-based logical check is also performed ("Company establishment time ≥ Company investment time ≥ Specific business investment time").

[0102] Anti-illusion verification: To prevent the model from generating false information, the system will verify whether the extracted descriptive content (such as "company nature") actually exists in the original text.

[0103] Preventing entity confusion: By performing logical verification with the fund manager's personal investment years, the system will prevent the model from misjudging the fund manager's personal investment years as the fund company's investment years.

[0104] (3) Extraction of fund product descriptions: Extract product description information of all funds (not limited to the promotional subject) that appear in the text, such as fund type, risk level, investment scope, investment objectives, etc., but they are often presented in the form of tables, so high-fidelity OCR text and VLM structured text are used for extraction, and finally the descriptions of different aspects are merged to generate the final result.

[0105] Extraction Logic: The model's extraction scope will be dynamically adjusted based on the identified fund type. For example, when a REITs fund is identified, REITs-specific information such as "infrastructure projects" and "profit distribution methods" will be automatically extracted.

[0106] Precise categorization: Through numerous heuristic rules, it prevents the incorrect extraction of information from different categories. For example, it ensures that the extracted result for "investment ratio" must contain numbers and not be a calculation formula (such as the composition of performance benchmarks); it ensures that "investment objectives" is a complete, descriptive paragraph, rather than a marketing statement.

[0107] Source text verification: For key phrases such as risk level, the system will verify whether they actually exist in the original text, effectively filtering out the illusionary outputs that the model creates out of thin air.

[0108] (4) Extraction of fund risk warning information: Extract various risk warnings and explanatory texts of the publicity subject, such as risk statements, disclaimers, investor notices and other legal texts. The content is mainly long paragraphs and detailed descriptions, and the accuracy of text recognition is extremely important. High-fidelity OCR text is used for extraction.

[0109] Extraction Logic: The model is guided to extract structured risk warnings, such as "risk-return characteristics," "general risk descriptions of the fund," "specific risk descriptions of the fund," and various warnings regarding performance, rankings, awards, and ratings. The design of the prompts in the large language model input emphasizes preserving complete text passages and avoiding information truncation.

[0110] Keyword verification: The post-processing rules will verify whether the extracted content contains keywords that match the indicator type (such as "risk", "performance", "rating"), to ensure that the correspondence between the content and the indicator is accurate.

[0111] Content completeness and filtering: The rules ensure that complete sentences or paragraphs are extracted, rather than fragmented information. At the same time, some potentially misleading outputs from the model are filtered out. For example, for "risk-return characteristics," the rules check whether the content is too short or contains overly generalized prompts such as "investment requires caution," to ensure that specific product feature descriptions are extracted.

[0112] (5) Extraction of fund expense information: Extract the expense structure of all funds mentioned in the text. The content is presented in a structured form such as tables, charts, and lists. Strict requirements are placed on spatial location and row-column correspondence. It is necessary to understand the layout structure to extract accurately. VLM structured text is used for extraction.

[0113] Extraction Logic: The model is specifically trained to identify and extract different types of fees, including subscription fees, purchase fees, redemption fees, management fees, custody fees, and sales service fees. The prompts in the large language model input include logic for handling tiered fee rates (such as rates that vary based on the subscription share) and differentiating between different share classes (Class A / Class C) fee differences.

[0114] Text preprocessing: Before being fed into the model, the context is enhanced by synonym replacement (such as replacing "this fund" with the specific name of the main fund), which helps the model more accurately associate fee information with the correct fund entity.

[0115] Results cleaning: The post-processing stage filters out redundant information and cost-irrelevant content from the model output to ensure the purity of the final results.

[0116] (6) Extraction of component proportion information: Extract the component proportion information such as asset allocation, industry distribution, and holdings of all entities (including funds and indices) using VLM structured text, for the same reason as the extraction of fund fee information.

[0117] Extraction Logic: The model is guided to extract distribution information containing the time dimension, such as "the distribution of the top ten stocks as of the end of a certain quarter". The design of the prompt words in the input of the large language model requires the model to output a list of components in a structured manner, and to integrate information such as component names, codes, weights (proportions) according to the original text content.

[0118] Indicator Classification and Filtering: Post-processing rules will classify the extracted indicators based on keywords (such as "proportion", "distribution", "weight") and strictly filter out indicators that do not belong to the component proportion category, such as "rate of return" and "growth rate", to prevent confusion with other modules.

[0119] Structured Merging: A key post-processing step is to merge discrete holdings information. For example, if the model extracts multiple entries such as "the weight of stock XX in the top ten holdings is Y%", the post-processing program will aggregate them into a single entry such as "the top ten holdings are: XX(Y%), AA(B%), etc.", forming a structured list.

[0120] Format standardization: Ensure that the output component list has a consistent format. For example, automatically add parentheses to weight numbers that are missing parentheses to form a standard format for "component (weight)".

[0121] (7) Extraction of fund data indicators: Extract various performance indicators and data indicators of all entities (funds, indices and their performance benchmarks). VLM structured text is used for extraction, and the principle is the same as that for extracting fund fee information. By using the table structure, row and column relationships and spatial location information in the original layout, the correspondence between data can be accurately understood, avoiding the problem of "misattribution" caused by loss of structure.

[0122] Deep Table Understanding: The model is trained to deeply understand the table after it has been linearized in step two, accurately linking the three dimensions of "entity", "metric" and "time" for each data point.

[0123] Time dimension differentiation: The model is explicitly required to differentiate between absolute time (such as "2023 return") and relative time (such as "recent 6-month return"), and automatically associate the latter with the data cutoff date of the context.

[0124] Indicator name generalization: For tables without explicit headers (e.g., only year and fund name), the model is guided to extract its values ​​as the "yield" indicator by default, which has good generalization ability.

[0125] Multi-round consensus and supplementation: The system will perform multiple extractions in parallel, select the most detailed result as the basis, and supplement it with unique and non-repeating indicators from other results to achieve "drawing on the strengths of all".

[0126] Advanced Deduplication: In the post-processing stage of step six, the system performs complex semantic and numerical deduplication. For example, it can identify that "China X Dividend Index" and "EasyX China X Dividend ETF Performance Benchmark" are numerically duplicated, and decide based on the reliability rules extracted from the security code, retaining the more accurate one.

[0127] Indicator normalization: unifies different expressions such as "2023" and "23 years" into "2023 years", and automatically completes or corrects the indicator name to "rate of return" or "ranking" according to the format of the numerical value ("x%" or "n / m").

[0128] (8) Mechanism for Ensuring the Indicator Completeness Based on Differential Analysis: To ensure that all key indicators mentioned in the marketing materials are fully extracted, this system introduces a mechanism for ensuring the indicator completeness based on differential analysis. This mechanism proactively identifies and supplements potentially missing indicator information through two stages: "broad-spectrum pre-extraction" and "differential supplementation." The specific implementation is as follows: Broad-spectrum pre-extraction: In step three, the system has already identified the core promotional entities in the material (the full name of the main fund, and its alternative names and abbreviations). In this step, the system passes the list of core promotional entities as prompts to the Large Language Model (LLM), instructing it to scan the VLM structured text content multiple times to extract all other mentioned entities besides the core entities (such as reference indices for comparison, macroeconomic data, other non-core funds, etc.) and their corresponding data. For example, Country A's core CPI rose 3.1% year-on-year in February, and LME spot gold closed at $2984.48 / ounce. Although these are not core entities in fund marketing and usually involve fewer entities and data, they are still an important part of the material and need to be extracted completely as supplementary information.

[0129] Differential Analysis and Targeted Supplementation: The system first summarizes the formal extraction results of all parallel sub-modules (such as the performance indicators and component proportions mentioned in step four) and extracts a "list of extracted values". All values ​​appearing in the original document are compared with the "list of extracted values" to identify "missing values" that exist in the original text but were not captured by any extraction module. For each "missing value", the system initiates a targeted supplementation process. This process utilizes few-shot learning technology, using some successfully extracted "time-entity-indicator-value" quadruples as high-quality examples, such as "As of November 15, 2023, the cumulative return of XinX Sheng (300502) over the past 10 years is 1587.37%" and "The dividend yield of the CSI 300 All Share Index over the past 12 months as of February 29, 2024 is 2.29%", to guide LLM in finding the meaning of the 1.6% value in marketing materials. After receiving a "missing value", LLM will locate the value in the original text and, based on its context, infer the time (Object) and metric (Target) to which it belongs.

[0130] For example, if the original text describes "This fund's year-to-date return is 5.4%, while the benchmark return for the same period is 3.1%", but the main extraction module only successfully extracts "5.4%" for "This Fund", the differential analysis mechanism will detect that "3.1%" is a missing value. Subsequently, the targeted supplementation process will be initiated. Based on the context, the LLM will associate "3.1%" with the entity "Benchmark" and the indicator "Return for the Same Period", thereby generating a complete data record and ensuring the integrity of the information.

[0131] Through this mechanism, the system can effectively complete the discrete and unstructured data points in the document, solving the problem of information omission caused by incomplete rule or model coverage in traditional extraction methods.

[0132] Step 5: Two-stage self-verification based on a multimodal large model: To ensure the accuracy of the extracted results, this invention introduces a closed-loop verification mechanism based on a multimodal large model and self-verification against the original text and image materials. This mechanism verifies each information point initially extracted from the promotional materials through a two-stage questioning process, thereby achieving high-precision internal factual consistency verification without introducing external dependencies.

[0133] Existence verification: This confirms whether the initially extracted complete information (including time, object, indicators, and values) truly exists in the original image. The system constructs a closed-ended yes / no question for each initially extracted structured information point. The system submits this question, along with the original marketing material image, to another multimodal large model (different from the initial large model, equivalent to cross-verification).

[0134] Question format: "In [time], is the [metric] of [object] [numerical]?"

[0135] Example: "Was the yield of YiXida Anxin Return Bond A 3.5% in 2023?"

[0136] Judgment: The model is required to answer "yes" or "no" directly. Only when the model explicitly answers "yes" can the information point pass the first stage of verification and proceed to the second stage of review. If the model answers "no" or cannot determine, the information point is initially marked as "questionable".

[0137] Numerical reverse retrieval: Building upon the confirmation of information existence in the first stage, the model is further required to actively retrieve the corresponding numerical values ​​from the image to prevent "confirmation bias" or insensitivity to numerical values ​​in the first stage. Based on the information points verified in the first stage, the system constructs an open-ended fill-in-the-blank question. The system then submits this question along with the original image to the multimodal large model.

[0138] Question format: "At [time], what is the [metric] of [object]?"

[0139] Example: "What will be the yield of YiXida Anxin Return Bond A in 2023?"

[0140] Comparison and Final Judgment: The system rigorously compares the numerical results returned by the model with the numerical values ​​initially extracted in step four. Only when the numerical values ​​returned by the model are completely consistent with the initially extracted numerical values ​​is the information point finally confirmed as "verification passed". If the numerical values ​​returned by the model are inconsistent, or the corresponding numerical values ​​cannot be found, the information point is finally marked as "verification failed".

[0141] Through a two-stage verification process of "first determining existence, then reverse retrieval," this invention leverages the visual understanding and reasoning capabilities of the multimodal large model to perform high-precision self-verification of the extracted results. This method not only eliminates dependence on external data sources, reducing system complexity and maintenance costs, but more importantly, it ensures that all final output data originates directly from the original marketing materials, guaranteeing the originality and authenticity of the information.

[0142] Step Six: Post-processing of extracted results: This stage employs a hybrid strategy of "large model as the main component and heuristic rules as a supplement," which deeply cleans, deduplicates, merges, and refines the results initially extracted from each module in step four, transforming them from the model's raw output into high-quality, reliable structured data.

[0143] (1) Entity normalization and format conversion: This step aims to unify the diverse original extraction results into a standard format, laying the foundation for subsequent processing.

[0144] Entity Alias ​​Mapping: The system maintains a dynamically constructed mapping dictionary of fund aliases and full names. During processing, it can automatically and accurately map various abbreviations and colloquial names (e.g., "GaoX Bond C") to their official full names (e.g., "YiX Da GaoXing Bond C"), achieving entity consistency.

[0145] Code completion and verification: For fund entities that are identified but lack code, the system will first search for a match in the existing context. If no match is found, an external search interface will be called to query and complete the code, and the reliability of the code will be marked (for example, the code extracted from the original text has high reliability, while the reliability of the code obtained through search is lower).

[0146] Standardization of Index Keys: The system strictly standardizes the "index keys" that uniquely identify each information point. This includes: invoking date processing tools to unify time expressions, replacing non-standard index names through a thesaurus, and removing meaningless conjunctions such as "of" and "is". A special example is that the system will unify an expression like "Performance Comparison Benchmark for 2024" to "Performance Comparison Benchmark 2024" through pre-set rules, ensuring that index keys with the same semantics have exactly the same string representation, providing guarantee for subsequent precise duplicate removal.

[0147] (2) Semantic-level Duplicate Removal Based on Large Models: Semantic Similarity Judgment: The system submits a batch of preliminarily extracted index descriptions to the LLM for judgment. For example, the LLM can accurately identify that two expressions, "Return Rate of the Performance Comparison Benchmark of aa Fund in 2024" and "Return Rate of the 2024 Performance Comparison Benchmark of aa Fund", although their word orders are different, have exactly the same core semantics and should be regarded as duplicate items.

[0148] Information Retention Strategy: Within the identified duplicate sets, the system will adopt heuristic rules (e.g., retaining the item with the longest text description) to determine which entry should be retained, thus ensuring that the information finally stored in the database is the most complete and rich.

[0149] Numerical Cross-Verification: To prevent the LLM from having biases in semantic judgment, the system adds a key "safety lock". If two indicators determined by the model to be duplicates have significantly different attached numerical values (e.g., one is 5.40% and the other is 6.80%), the system will overturn the model's judgment and determine that they are not truly duplicate items. This mechanism greatly reduces the risk of losing valid information due to incorrect merging.

[0150] (3) Structured Information Merging and Aggregation: This embodiment includes a powerful heuristic merging engine for aggregating multiple discrete but related indicators into a more rich and information-dense structured indicator.

[0151] Identifying Similar Types of Mergable Indicators: The system merges these discrete indicators into a new list-type indicator by finding indicator clusters with the same "subject" (e.g., "YiX Da Zhongzheng 1000"), the same "suffix" (e.g., "weight") but different "middle parts". For example, for the input discrete indicators: The weight of ShanX Coking Coal in YiX Da Zhongzheng 1000 is 1.34%; The weight of YanX Energy in YiX Da Zhongzheng 1000 is 1.08%; The weight of HengX Coal Power in YiX Da Zhongzheng 1000 is 1.31%; Output (Aggregated Indicator): The weighted stocks in the YiXda CSI 1000 Index are: Shanxi Coking Coal (1.34%), Yanzhou Energy (1.08%), and Hengxin Coal & Electricity (1.31%).

[0152] This feature is extremely effective for processing scenarios such as holdings lists and industry distributions, as it can automatically aggregate previously scattered data points into structured list information.

[0153] (4) Global Refinement Based on Large Model: After all indicators have undergone preliminary standardization, deduplication, and merging, the system will call LLM again to perform a global "review" and refinement of all results in the current batch: Correction and Alignment: Corrects potential typos and strictly aligns all indicator names to the system's predefined official indicator classification system.

[0154] Standardized format: Ensure that all fund entities are presented in the standard format of "Full Fund Name (Fund Code)".

[0155] Invalid entry removal: Removes all invalid data entries whose descriptions are "None", "Not Applicable", or "Empty".

[0156] (5) Heuristic rule verification throughout the entire process: In addition to relying on large models for macro-level semantic processing, the system also embeds a large number of domain-knowledge-based heuristic rules in each processing stage as the last line of defense for data quality: Logical consistency check: The system performs strict logical checks. For example, when processing fund company information, the logical relationship must be satisfied: "Company establishment time ≥ Investment years ≥ Investment years of specific business (such as fixed income, equity)". Any data point that does not meet this logical chain will be considered an extraction error and discarded.

[0157] Context Existence Verification: To effectively combat the "illusion" problem of the model, the system verifies whether the extracted key information (such as fund code, risk level "R4", etc.) actually exists in the original input text. If an extracted value cannot be found in the original text, it will be judged as invalid and filtered out.

[0158] Deduplication based on content inclusion: This rule is used to handle information redundancy. If the "subject" (e.g., "EasyX Fund"), "indicator" (e.g., "Shenzhen Level 1 Industry"), and "value" (e.g., "17") of indicator A all appear in the "indicator" of indicator B (e.g., "industry coverage"), then indicator A is considered redundant information and is removed. For example: Indicator A: The number of first-level industries for YiXda xx Fund is 17; Indicator B: The industry coverage of YiXda xx Fund is 17 first-level industries and 35 second-level industries.

[0159] Judgment: Indicator A has been deleted.

[0160] Deduplication based on entity and indicator length: This rule is triggered when two indicators have exactly the same value.

[0161] Rule 1 (Substring as Subject): If the subject of indicator A is a substring of the subject of indicator B (e.g., subject A: "China X Dividend", subject B: "China X Dividend Value Index"), then retain indicator A with the shorter subject and delete B. This rule is based on the assumption that "the more concise the description, the more core the information."

[0162] Rule 2 (Same Body Length): If two indicators have the same body length, retain the one with the longer "indicator description" because it may contain richer contextual information.

[0163] Deduplication based on a combination of numerical and date data: When two indicators have exactly the same numerical value (such as all percentages) and the same date, they are considered duplicates. In this case, the system will decide which one to keep based on secondary rules such as entity type (e.g., prioritizing "index" over "fund") or entity name length.

[0164] Step 7: Result Tracing: (1) Image Source Tracing: For materials parsed from images, the system employs a coordinate mapping method based on a Visual Language Model (VLM). The implementation details are as follows: Input construction: For each information point that needs to be located, the system will construct a structured request containing a unique identifier (id), the value or text to be located (value), and its context description (context).

[0165] VLM Call: This request is submitted to VLM along with the original image. Contextual description information helps VLM accurately disambiguate when multiple identical values ​​or text exist in the image.

[0166] Coordinate output: VLM returns a dictionary that corresponds one-to-one with the input ID, with each value representing the precise pixel coordinate range ([x1, y1, x2, y2]) of the information point on the original image, thus achieving high-precision visual positioning.

[0167] (2) Document tracing: For information extracted from documents (including Markdown documents converted from images), the system adopts a hybrid strategy of "direct matching + model assistance" for tracing, the details of which are as follows: Unique numerical direct matching: The system first performs frequency analysis on all values ​​to be located. For values ​​that appear only once in the document, the system uses various string matching strategies (such as exact match, suffix matching for improved robustness, and word segmentation matching) to quickly and directly locate them, and then injects a unique reference identifier (such as {#id}) after them.

[0168] Model-assisted complex localization: For values ​​that appear multiple times in a document or are difficult to match directly due to formatting issues, the system initiates a model-assisted localization process. Input Construction: The system constructs a structured list of tasks for all values ​​that failed to be located in the first stage, along with their context descriptions, and submits it along with the complete Markdown text to a large language model (LLM).

[0169] Instruction constraints for injecting citation identifiers: The system imposes a series of strict instruction constraints on the output behavior of the LLM, requiring it to inject the corresponding citation identifiers at precise locations in the original text after understanding the semantics of the context. These constraints include: Formatting Preservation: The complete Markdown document after injection must be returned, and no indentation, symbols, spaces, or line breaks in the original text can be changed.

[0170] Uniqueness of the identifier: Each reference identifier can only appear once in the output document.

[0171] Location accuracy: The identifier must be placed in the most semantically appropriate position based on the context information.

[0172] Allow multiple citations: Allows the same position in the original text to be injected with multiple different citation markers as needed.

[0173] Post-processing and verification of results: After receiving the document returned by LLM, the system will perform final verification and correction, such as automatically checking and completing the format of the reference identifier, and for the few duplicate values ​​that may still be missed by the model, adopting the strategy of defaulting to the first occurrence position to ensure that each extracted item has traceability information.

[0174] Step 8: Report Generation and Output: The system aggregates all validated and post-processed structured data, along with the precise source information (image coordinates or document reference identifiers) obtained in step seven, into a comprehensive, high-quality JSON data structure. This structured object contains multiple keys, primarily including: Raw extracted text: The original Markdown text output from step one, without any modifications.

[0175] Text with source tracing identifiers: After document source tracing in step seven, a large amount of Markdown text with {#id} reference identifiers was injected.

[0176] Information content list: A list of all information points extracted from the document. Each entry contains an ID, the object of the information content, the index of the information content, the value of the information content, and the source coordinates of the information content in the image.

[0177] Image and text analysis time: Total time spent on the entire processing flow.

[0178] Ultimately, by generating such a detailed and well-structured report, the system not only completed an in-depth analysis of the fund's graphic marketing materials, but also provided a complete and traceable record of the processing.

[0179] In summary, compared with the prior art, the above embodiments of this application have the following beneficial effects: by acquiring graphic and textual materials and performing visual recognition and character recognition on the graphic and textual materials respectively, structured text data and plain text data are obtained, preserving the original graphic and textual layout structure information and accurate character-level text, avoiding structural mismatch or text distortion caused by a single recognition method; based on the structured text data or plain text data, structured information corresponding to the preset information dimensions is extracted, and the on-demand extraction strategy of the optimal text source can be dynamically selected according to the information type, improving the extraction accuracy of different information dimensions; based on the graphic and textual materials and the preset multimodal model, the existence verification and numerical consistency verification of the structured information set are performed. This paper constructs an internal fact consistency verification mechanism that does not rely on external knowledge to filter out false information caused by model illusions. It associates each verified information item with its pixel coordinates or text reference identifiers in the graphic materials to generate structured parsing results with traceability identifiers, thereby achieving auditability and traceability of the extraction results. The above features work together to solve sub-problems such as structure preservation, text fidelity, on-demand source selection, result credibility, and process traceability. Through a technical closed loop of dual-source identification, source selection extraction, multimodal self-verification, and dual-modal traceability, it solves the problem of low accuracy and unverifiable structured information extraction in complex graphic materials, enabling automated parsing results to have both high accuracy and high credibility.

[0180] Example 2: Please refer to Figure 2 Based on the same inventive concept, the present invention discloses an information parsing system for graphic materials, comprising: a data acquisition module M1, an extraction module M2, a verification module M3, and an output module M4; The data acquisition module M1 is used to acquire graphic and textual materials, and to perform visual recognition and character recognition on the graphic and textual materials to obtain structured text data and plain text data.

[0181] Furthermore, the data acquisition module M1 includes: a block unit and a dual identification unit; The segmentation unit is used to acquire graphic and textual materials and perform content segmentation of the graphic and textual materials based on the layout structure to obtain several content block images. The dual recognition unit is used to perform visual recognition and character recognition on each of the content block images to obtain structured text data and plain text data.

[0182] By dividing the text and image materials into content blocks based on the layout structure, several content block images are obtained. Then, visual recognition and character recognition are performed on each content block image. The overall parsing task of large-size or long text and image materials is decomposed into multiple logically independent small image parallel processing units. This effectively avoids information truncation or content omission caused by the context length limitation of multimodal large models. At the same time, it reduces the computational resource consumption of a single model call and improves the system's processing efficiency and robustness for complex text and image materials such as high-resolution marketing posters.

[0183] Furthermore, the segmentation unit includes: a text detection subunit, an aggregation subunit, and a segmentation subunit; The text detection subunit is used to perform text detection on the graphic material to obtain each text region; The aggregation subunit is used to aggregate and generate each first content block based on each text region, combined with font size, color, direction and semantic boundary, and to construct a spatial relationship diagram according to the relative position and size relationship of each first content block; The cutting subunit is used to determine each cutting point based on the spatial relationship diagram and preset cutting constraints using a decision tree algorithm, and to cut the graphic material along each cutting point to output several logically independent content block images.

[0184] In this preferred embodiment, text detection is performed on the graphic material to obtain each text region, which is then aggregated to generate the first content block. A spatial relationship graph is constructed based on the relative position and size relationship of the content blocks to achieve semantic perception and structural understanding of the graphic content. Furthermore, the decision tree algorithm is used in combination with cutting constraints to determine the cutting points and cut the graphic material. Logically independent content block images are generated while preserving the integrity of the information, providing a reliable input unit for subsequent high-precision parallel parsing.

[0185] Furthermore, the dual identification unit includes: an initial identification subunit, a preprocessing subunit, and a table processing subunit; The initial recognition subunit is used to perform visual recognition and character recognition on each of the content block images to obtain the original structured text data and the original plain text data. The preprocessing subunit is used to preprocess the original structured text data and the original plain text data to obtain the first structured text data and the plain text data; wherein, the preprocessing includes any one or more combinations of the following: noise filtering, format unification, and synonym and special character replacement; The table processing subunit is used to fill empty cells, align rows and columns, and merge multiple columns in the table data of the first structured text data to generate a corresponding single-column linearized table. Each data item in the single-column linearized table is then converted into a key-value statement in combination with the row header and column header attributes to obtain the structured text data.

[0186] In this preferred embodiment, preprocessing such as noise filtering, format unification, and replacement of synonyms and special characters improves the standardization and consistency of text data and reduces interference noise in subsequent information extraction. Furthermore, the tables in the preprocessed structured text data are filled with empty cells, aligned with rows and columns, and merged into single-column linear tables. The table data items are converted into key-value statements that combine row header and column header attributes, reducing the two-dimensional table structure to a one-dimensional, semantically clear linear representation. This greatly reduces the burden on the large language model to understand the table content, effectively prevents the mismatch between indicators and entities caused by the complexity of the table structure, and significantly improves the extraction accuracy of structure-sensitive information.

[0187] The extraction module M2 is used to extract structured information corresponding to a preset information dimension based on the structured text data or plain text data, and obtain a set of structured information.

[0188] Furthermore, the extraction module M2 includes: an entity context construction unit, an initial extraction unit, a numerical extraction unit, a reverse calculation unit, and a supplementary unit; The entity context construction unit is used to identify all core entities in the structured text data and construct a global semantic context based on each core entity. The initial extraction unit is used to extract the structured information of each core entity under a preset information dimension from the structured text data or the plain text data based on the global semantic context, so as to obtain an initial set of structured information. The numerical extraction unit is used to extract all numerical values ​​from the graphic material to obtain the original numerical value set, and to extract the covered numerical values ​​from the initial structured information set to obtain the extracted numerical value set. The reverse reasoning unit is used to compare the original set of values ​​with the extracted set of values, identify missing values, and guide the large language model to reverse reason the corresponding entity, indicator and time information based on the global semantic context and the context of the missing values, and generate corresponding supplementary information items. The supplementary unit is used to merge each of the supplementary information items with the initial structured information set to obtain the structured information set.

[0189] In this preferred embodiment, by identifying all core entities in structured text data and constructing a global semantic context, a comprehensive entity anchor system is established for information extraction. Based on this global context, structured information of each entity in a preset dimension is extracted from structured text data or plain text data, achieving precise and targeted extraction centered on entities. Furthermore, by comparing all values ​​in the text and image materials with the extracted value set, missing values ​​are identified. Based on the global semantic context and the missing value context, the large language model is guided to infer the corresponding entity, indicator, and time information in reverse, actively filling in the numerical omissions caused by task coverage blind spots. Thus, while ensuring the integrity of the main entity information, it also ensures the comprehensive capture of non-main entity related values, significantly improving the coverage breadth and data integrity of the structured parsing results.

[0190] The verification module M3 is used to perform existence verification and numerical consistency verification on the structured information set based on the graphic materials and the preset multimodal model, so as to obtain structured verification information data.

[0191] Furthermore, the verification module M3 includes: an existence judgment unit, a numerical reverse extraction unit, and a numerical verification unit; The existence judgment unit is used to construct a closed-form verification problem for each information item in the structured information set, and input the closed-form verification problem and the graphic material into the multimodal model, and output the corresponding existence judgment result. The numerical reverse extraction unit is used to construct an open numerical retrieval problem for information items whose existence judgment result is affirmative, and input the open numerical retrieval problem and the graphic material into the multimodal model to obtain the corresponding reverse extraction value; The numerical verification unit is used to compare each of the reverse-extracted numerical values ​​with the original numerical values ​​in the corresponding information item. If they are completely consistent, the information item is retained as structured verification information data.

[0192] In this preferred embodiment, a preliminary screening of whether an information item truly exists in the original material is achieved by constructing a closed-loop verification problem for each information item in the structured information set and inputting it into a multimodal model along with the text and image materials to output an existence judgment result. Further, an open-loop numerical retrieval problem is constructed for the information items whose existence is confirmed and input into the multimodal model to obtain reverse-extracted numerical values. By comparing the consistency between the reverse values ​​and the original values, a two-stage numerical accuracy verification mechanism is constructed. This mechanism not only effectively eliminates false information generated by model illusions but also ensures the absolute accuracy of the retained results through dual verification of reverse numerical retrieval and comparison, significantly improving the reliability of the final structured data.

[0193] The output module M4 is used to associate each information item in the structured verification information data with its pixel coordinates or text reference identifier in the graphic material, and generate a structured parsing result with a traceability identifier.

[0194] Furthermore, the output module M4 includes: a tracing unit and an association unit; The tracing unit is configured to, for each information item, construct a location request based on the corresponding attribute value and context description of the information item if the graphic material is in image format, and input the request and the graphic material into a preset visual language model to obtain the pixel coordinate range of the information item in the graphic material; if the graphic material is in document format, perform frequency analysis on the attribute values ​​of each information item: if the attribute value appears uniquely in the graphic material, locate the attribute value in the graphic material through string matching, and inject a unique reference identifier at the corresponding position of the attribute value; if the attribute value appears repeatedly or has an ambiguous format in the graphic material, construct a task instruction containing context based on the attribute value, input the request and the graphic material into a preset language model to obtain the position of the information item in the graphic material and inject a unique reference identifier. The association unit is used to attach the pixel coordinate range or reference identifier as a tracing field to the corresponding information item to generate a structured parsing result with a tracing identifier.

[0195] In this preferred embodiment, each information item is processed according to the format of the graphic and textual material: for image format materials, a positioning request is constructed and input into a visual language model to obtain the pixel coordinate range, thus achieving precise spatial positioning of the visual content; for document format materials, attribute value frequency analysis is first performed, and unique attribute values ​​are located by string matching and a reference identifier is injected; for repeated or ambiguous attribute values, a task instruction with context is constructed and input into a language model to obtain the position and an identifier is injected. This dual-modal hybrid tracing strategy takes into account both the pixel-level accuracy of image materials and the semantic disambiguation capability of document materials, ensuring that the extraction results of various graphic and textual materials can be accurately and uniquely traced back to the original source, greatly enhancing the verifiability and audit credibility of the structured parsing results.

[0196] In summary, compared with existing technologies, the embodiments of this application have the following beneficial effects: by acquiring graphic and textual materials and performing visual and character recognition on these materials respectively, structured text data and plain text data are obtained, preserving the original graphic and textual layout structure information and accurate character-level text, avoiding structural mismatch or text distortion caused by a single recognition method; based on the structured text data or plain text data, structured information corresponding to the preset information dimensions is extracted, and the on-demand extraction strategy of the optimal text source can be dynamically selected according to the information type, improving the extraction accuracy of different information dimensions; based on the graphic and textual materials and the preset multimodal model, the existence verification and numerical consistency verification of the structured information set are performed. An internal fact consistency verification mechanism independent of external knowledge is constructed to filter out false information caused by model illusions. Each verified information item is associated with its pixel coordinates or text reference identifier in the graphic material to generate a structured parsing result with traceability identifier, realizing the auditability and traceability of the extraction result. The above features work together to solve sub-problems such as structure preservation, text fidelity, on-demand source selection, result credibility, and process traceability. Through a technical closed loop of dual-source identification, source selection extraction, multimodal self-verification, and dual-modal traceability, the problem of low accuracy and unverifiable structured information extraction in complex graphic materials is solved, so that the automated parsing result has both high accuracy and high credibility.

[0197] Example 3: This invention also provides a computer program product, including a computer program or instructions, capable of running on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to execute any of the graphic material information parsing methods of this invention.

[0198] Example 4: This invention also provides a computer-readable storage medium storing at least one executable instruction that, when executed on a graphic material information parsing system, causes the graphic material information parsing system to perform one of the graphic material information parsing methods described in any of the above method embodiments.

[0199] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. Similarly, for the purpose of simplification and aiding understanding of one or more aspects of the invention, in the above description of exemplary embodiments of this application, various features of the embodiments are sometimes grouped together in a single embodiment, figure, or description thereof. The claims, which follow the detailed description, are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0200] Those skilled in the art will understand that the modules in the system of the embodiments can be adaptively changed and placed in one or more systems different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components, except that at least some of such features and / or processes or units are mutually exclusive.

Claims

1. A method for information analysis of graphic and textual materials, characterized in that, include: Obtain text and image materials, and perform visual recognition and character recognition on the text and image materials respectively to obtain structured text data and plain text data; Based on the structured text data or plain text data, extract the structured information corresponding to the preset information dimensions to obtain a set of structured information. Based on the aforementioned graphic and textual materials and a preset multimodal model, the existence verification and numerical consistency verification of the structured information set are performed to obtain structured verification information data. Each information item in the structured verification information data is associated with its pixel coordinates or text reference identifier in the graphic material to generate a structured parsing result with a source tracing identifier.

2. The information parsing method for graphic and textual materials as described in claim 1, characterized in that, The process of acquiring text and image materials, and performing visual recognition and character recognition on the text and image materials to obtain structured text data and plain text data includes: Obtain text and image materials, and divide the text and image materials into content blocks based on the layout structure to obtain several content block images; Visual recognition and character recognition are performed on each of the content block images to obtain structured text data and plain text data.

3. The information parsing method for graphic and textual materials as described in claim 2, characterized in that, The step of dividing the graphic material into content blocks based on layout structure to obtain several content block images includes: Text detection is performed on the aforementioned text and image materials to obtain each text region; Based on each of the text regions, and combining font size, color, direction and semantic boundaries, each first content block is generated by aggregation, and a spatial relationship diagram is constructed according to the relative position and size relationship of each first content block; Based on the spatial relationship diagram and the preset cutting constraints, the decision tree algorithm is used to determine each cutting point, and the graphic material is cut along each cutting point to output several logically independent content block images.

4. The method for information parsing of graphic and textual materials as described in claim 2, characterized in that, The step of performing visual recognition and character recognition on each of the content block images to obtain structured text data and plain text data includes: Visual recognition and character recognition are performed on each of the content block images to obtain the original structured text data and the original plain text data; The original structured text data and the original plain text data are preprocessed to obtain the first structured text data and the plain text data; wherein, the preprocessing includes any one or more combinations of the following: noise filtering, format unification, and synonym and special character replacement; The table data in the first structured text data is filled with empty cells, aligned with rows and columns, and merged into multiple columns to generate a corresponding single-column linearized table. Each data item in the single-column linearized table is then converted into a key-value statement, along with the row header and column header attributes, to obtain the structured text data.

5. The method for information parsing of graphic and textual materials as described in claim 1, characterized in that, The step of extracting structured information corresponding to a preset information dimension based on the structured text data or plain text data to obtain a set of structured information includes: Identify all core entities in the structured text data, and construct a global semantic context based on each core entity; Based on the global semantic context, the structured information of each core entity under the preset information dimension is extracted from the structured text data or the plain text data respectively, to obtain an initial set of structured information; All numerical values ​​are extracted from the text and image materials to obtain the original numerical value set, and the covered numerical values ​​are extracted from the initial structured information set to obtain the extracted numerical value set. By comparing the original set of values ​​with the extracted set of values, missing values ​​are identified. Based on the global semantic context and the context of the missing values, the large language model is guided to reverse reasoning the corresponding entities, indicators and time information to generate corresponding supplementary information items. The supplementary information items are combined with the initial structured information set to obtain the structured information set.

6. The method for information parsing of graphic and textual materials as described in claim 1, characterized in that, Based on the textual and graphic materials and a preset multimodal model, the structured information set is subjected to existence verification and numerical consistency verification to obtain structured verification information data, including: For each information item in the structured information set, a closed-form verification problem is constructed, and the closed-form verification problem and the graphic material are input into the multimodal model to output the corresponding existence judgment result. For information items whose existence is affirmative, an open-ended numerical retrieval problem is constructed, and the open-ended numerical retrieval problem and the text and image materials are input into the multimodal model to obtain the corresponding reverse extraction values; The extracted values ​​are compared with the original values ​​in the corresponding information items. If they are completely consistent, the information item is retained as structured verification information data.

7. The method for information parsing of graphic and textual materials as described in claim 1, characterized in that, The step of associating each information item in the structured verification information data with its pixel coordinates or text reference identifier in the graphic material to generate a structured parsing result with a source tracing identifier includes: For each information item, if the text and image material is in image format, a location request is constructed based on the corresponding attribute value and context description of the information item, and input into a preset visual language model along with the text and image material to obtain the pixel coordinate range of the information item in the text and image material; if the text and image material is in document format, frequency analysis is performed on the attribute values ​​of each information item: if the attribute value appears uniquely in the text and image material, the attribute value is located in the text and image material through string matching, and a unique reference identifier is injected at the corresponding position of the attribute value; if the attribute value appears repeatedly in the text and image material or the format is ambiguous, a task instruction containing context is constructed based on the attribute value, and input into a preset language model along with the text and image material to obtain the position of the information item in the text and image material and inject a unique reference identifier. Each pixel coordinate range or reference identifier is used as a tracing field and appended to the corresponding information item to generate a structured parsing result with a tracing identifier.

8. An information analysis system for graphic and textual materials, characterized in that, include: Data acquisition module, extraction module, verification module, and output module; The data acquisition module is used to acquire graphic and textual materials, and to perform visual recognition and character recognition on the graphic and textual materials to obtain structured text data and plain text data. The extraction module is used to extract structured information corresponding to a preset information dimension based on the structured text data or plain text data, and obtain a set of structured information. The verification module is used to perform existence verification and numerical consistency verification on the structured information set based on the graphic materials and the preset multimodal model, so as to obtain structured verification information data. The output module is used to associate each information item in the structured verification information data with its pixel coordinates or text reference identifier in the graphic material, and generate a structured parsing result with a traceability identifier.

9. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed, they implement the information parsing method for graphic materials as described in any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements an information parsing method for graphic materials as described in any one of claims 1-7.

Images