A document image-text integrated intelligent understanding and processing method and system

Abstract

This application provides a method and system for integrated intelligent understanding and processing of text and images in documents, relating to the field of artificial intelligence technology. It includes: separating text and image elements and establishing contextual relationships; calling a graphics vectorization engine to perform intelligent vector conversion processing on rasterized illustrations; combining text semantic analysis to perform hierarchical classification, attribute assignment, and structural reconstruction of vector primitives, generating vector entity objects with complete attributes; inputting text elements and vector entity objects into a pre-trained multimodal large language model to generate a unified knowledge representation including vectors, topology, attributes, and document context, supporting integrated text and image applications such as question answering, editing, summarizing, and reporting. This invention solves the problems of semantic loss and difficulty in fine-grained parsing caused by vector-to-rasterized illustrations in documents such as papers and reports, significantly improving the overall understanding depth and interpretability of AI for complex text and image documents, and providing high-performance semantic understanding capabilities for document-based training and intelligent processing.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and system for intelligent understanding and processing of documents and images. Background Technology

[0002] As a crucial carrier of scientific papers, research reports, engineering designs, construction implementation, and equipment management and maintenance decisions, text and image documents typically consist of lengthy text descriptions combined with various types of raster-formatted illustrations. With the deepening of enterprise digital transformation and the development of artificial intelligence technology, knowledge management, content retrieval (RAG), and automated office environments are placing increasingly higher demands on the intelligent processing and understanding of documents. Among the massive amounts of text and image documents, illustrations encompass flowcharts, development design diagrams, process and instrumentation diagrams (P&ID), electrical system diagrams, equipment structure diagrams, CAD design drawings, and GIS thematic maps. Original illustrations are usually generated or stored in vector graphics format. However, during document creation and archiving, these vector graphics are often embedded as rasterized images in document formats such as PDF, CAJ, Word, and WPS. This results in the loss of the original graphic geometric structure, symbol system, semantics, and spatial relationships, significantly increasing the difficulty of subsequent intelligent parsing and semantic understanding. Meanwhile, high-quality text and image documents are important materials for training large models. However, due to the lack of original illustration information and insufficient contextual connections in the content, the model cannot fully utilize the materials during training, and may even cause the model to misunderstand some content. Therefore, it is urgent to solve the problem of intelligent understanding and information restoration of original illustration information.

[0003] Existing intelligent document processing technologies mainly rely on OCR text recognition, layout-based text extraction, rule-template-based flowchart parsing, and engineering drawing symbol recognition models for a few specific fields. However, these methods generally have the following limitations:

[0004] (1) Text and illustrations are usually processed separately, lacking a unified semantic modeling mechanism for text and graphics, which makes it difficult to associate illustration descriptions with text semantics and form a complete knowledge semantic network;

[0005] (2) The recognition ability of multiple types of illustrations is insufficient. Different types of graphics have significant differences in symbol system, structural expression, connection logic and spatial layout. Traditional methods are difficult to achieve adaptive and generalized processing.

[0006] (3) Most existing methods can only identify local elements, but cannot reconstruct the topological spatial relationships and attribute assignments within the graphics, such as upstream-downstream relationships in the process flow, equipment connection networks, electrical bus topology or GIS spatial associations, adjacency and inclusion relationships, etc., which makes it difficult to carry out subsequent reasoning, review or engineering change analysis.

[0007] In recent years, new technologies such as multimodal large models and visual Transformers have provided a new methodological foundation for collaborative text-image understanding. However, text-image document scenarios are characterized by highly specialized knowledge systems, significant industry differences in symbol systems, inconsistent quality of raster graphics, and tight coupling between structural semantics and textual semantics, making it difficult to directly apply existing multimodal models to achieve effective results. In particular, after rasterization, illustrations often suffer from issues such as broken symbols, noise interference, and occlusion / overlap in symbol morphology, text annotations, and connecting lines, making it difficult to accurately perform symbol recognition and topology construction. Furthermore, titles, numbering, text citations, legend definitions, and contextual knowledge in text-image documents have strong structured relationships, requiring text-image understanding models to not only possess visual perception and textual semantic modeling capabilities but also cross-modal alignment and semantically consistent reasoning capabilities.

[0008] Current technologies in the field of intelligent document processing still face the following core technological bottlenecks:

[0009] (1) There is a lack of a unified processing and structured analysis framework for multiple types of illustrations, making it impossible to obtain the inherent geometry, attributes and spatial relationships of illustrations, and making it difficult to achieve a computable expression of graphic semantics;

[0010] (2) There is a lack of a graph-text fusion model that can simultaneously process text semantics and graph topology, which makes it impossible to automatically reconstruct knowledge structure and is not conducive to understanding document content in large model training scenarios.

[0011] (3) Lacking the ability to verify consistency between text and graphics, extract knowledge and answer questions based on unified semantic expression, it is difficult to support the intelligent needs of documents in the stages of review, modification, operation and maintenance and delivery.

[0012] (4) Traditional document management methods rely on human professionals to check and explain each page, resulting in prominent pain points such as low information retrieval efficiency, high error risk and difficulty in knowledge inheritance. Summary of the Invention

[0013] In view of the above problems, this application proposes a document-text integrated intelligent understanding and processing method and system to overcome the shortcomings of the prior art.

[0014] In a first aspect, embodiments of this application provide a document-text integrated intelligent understanding and processing method, including:

[0015] Obtain the document to be processed, and generate a document object model based on the document to be processed;

[0016] The document object model is used to separate the elements of the document content to be processed, extracting text elements and image elements. Based on the layout structure, figure title, figure legend and text reference relationship, the initial semantic association between the image element and its corresponding context text is established.

[0017] The general-purpose graphics vectorization engine based on a deep learning model is invoked to perform intelligent raster-to-vector processing on the image elements. Through deep visual feature extraction, the original appearance of the vector graphics is identified and restored to obtain a basic vector primitive set containing points, lines, surfaces, volumes, text and line types, and legend symbols.

[0018] Based on the initial semantic association, semantic analysis is performed on the text elements to extract attribute description information, and combined with the geometric features of the basic vector primitive set, the basic vector primitive set is subjected to element layering and semantic classification.

[0019] Based on the attribute description information, attributes are assigned to the relevant graphic elements after classification, and semantically unsemantic geometric primitives are reconstructed and semantically restored, and reconstructed into vector entity objects with complete attribute information.

[0020] The text elements and the vector entity objects of the structured reconstruction and semantic restoration are input into a pre-trained multimodal large language model. The reasoning ability of the multimodal large language model is used to perform cross-modal representation alignment and consistency verification, and generate a unified knowledge representation that includes graph topology, entity attributes and document context.

[0021] Based on the unified knowledge representation, the content of the document to be processed is automatically, precisely, and interpretably understood.

[0022] Optionally, the document object model is used to separate the elements of the document content to be processed, extracting text elements and image elements. Based on the layout structure, figure titles, legends, and text references, an initial semantic association is established between the image elements and their corresponding contextual text, including:

[0023] The document to be processed is parsed for layout structure, and the corresponding text content is extracted using optical character recognition method to form the text element;

[0024] The corresponding illustration regions are extracted using document layout region detection or image segmentation methods to form the image elements;

[0025] Based on the document layout position, figure title number, text citation mark and semantic similarity in the document to be processed, a correspondence is established between the text paragraphs and the illustrations, thereby obtaining the initial semantic association between the image elements and their corresponding context text.

[0026] Optionally, a general-purpose graphics vectorization engine based on a deep learning model is invoked to perform intelligent raster-to-vector conversion processing on the image elements, extracting and restoring the original appearance of the illustration vector graphics, including:

[0027] A visual encoder is constructed using a convolutional neural network or a visual Transformer architecture to extract high-dimensional visual features from the image elements;

[0028] After extracting high-dimensional visual features, image elements are processed by a target detection and instance segmentation network to accurately identify and segment legend symbols, endpoints of connecting lines, intersections, and arrow directions from background noise.

[0029] Using a geometric primitive prediction model, the segmented pixel regions are transformed into parameterized vector representations;

[0030] Locate the legend region in the image, extract the legend by vectorization, obtain the geometric shape of the legend symbol and its corresponding text interpretation, and establish a legend feature library;

[0031] The optical character recognition model is used to extract text annotations within an image and record their coordinate positions.

[0032] The basic vector primitive set is formed by combining the legend symbols, endpoints of connecting lines, intersections and arrow points, as well as the parameterized vector representation, the geometric shape of the legend symbols and their corresponding textual interpretations, and textual annotations recording their coordinate positions. This basic vector primitive set includes points, lines, surfaces, volumes, text and line types, and legend symbols.

[0033] Optionally, based on the initial semantic association, semantic analysis is performed on the text elements to extract attribute description information, and combined with the geometric features of the basic vector primitive set, element layering and semantic classification are performed on the basic vector primitive set, including:

[0034] Natural language processing algorithms are used to perform entity recognition and relation extraction on text paragraphs, figure titles and captions associated with image elements, and key information including entity name, specification parameters, material properties and functional description is extracted as the attribute description information;

[0035] A topological analysis is performed on the geometric lines in the basic vector element set. Based on the line type characteristics and connection relationships, the outline lines, leader lines, dimension lines and center lines are distinguished, and the geometric elements are separated into layers, that is, the elements are layered.

[0036] The entity keywords in the extracted attribute description information are mapped and matched with the legend symbols or geometric shapes in the basic vector primitive set, thereby classifying semantically meaningless geometric figures into different semantic categories, i.e., performing semantic classification.

[0037] Optionally, based on the attribute description information, attributes are assigned to the classified related graphic elements, and semantically non-semantic geometric primitives are structurally reconstructed and semantically restored, including:

[0038] Based on the legend feature library, matching geometric shapes are retrieved from the basic vector primitive set, and corresponding semantic tags are assigned to the matching geometric primitives according to the text interpretation in the legend.

[0039] Based on the text annotations and their coordinate positions in the image, find the geometric primitives that are closest in space or connected by leaders, and assign the text annotation content as attribute information to the geometric primitive.

[0040] Based on the connection relationship description in the attribute description information, the occluded or broken vector lines are identified, and the connectivity is repaired based on the geometric extension and fitting algorithm to construct a complete topological connection network.

[0041] Geometric primitives endowed with semantic labels, attribute information, and topological relationships are encapsulated into independent vector entity objects to perform structured reconstruction and semantic restoration of semantically meaningless geometric primitives.

[0042] Optionally, the text elements and the structured reconstruction and semantic restoration vector entity objects are jointly input into a pre-trained multimodal large language model. Utilizing the reasoning capabilities of the multimodal large language model, cross-modal representation alignment and consistency verification are performed to generate a unified knowledge representation that includes graph topology, entity attributes, and document context, including:

[0043] The vector entity object is serialized into a structured description language or code format and used as visual modal input;

[0044] The text elements and extracted attribute description information are used as text modal input;

[0045] The system constructs prompting instructions to guide the multimodal large language model to verify and correct entity names, connection relationships, and attribute values ​​in the visual modal input based on the text modal input.

[0046] By utilizing the reasoning capabilities of the multimodal large language model, the implicit functional logic or causal relationships between vector entity objects can be derived.

[0047] The output is a graph structure data containing entity nodes, attribute edges, and relation edges, which serves as the unified knowledge representation.

[0048] Optionally, the pre-trained multimodal large language model is an encoder-decoder model based on the Transformer architecture, whose input modalities include: text embedding vectors and vector image embedding vectors;

[0049] During the fine-tuning training of the multimodal large language model, a consistency constraint is introduced to ensure logical consistency between the structured reconstruction and semantic restoration and the factual statements in the text elements. The specific process includes:

[0050] Perform word segmentation and context encoding on the text elements in the sample to generate the text embedding vector;

[0051] Graph structure encoding is performed on the structured reconstruction and semantic restoration of the samples, and the graph embedding vector is generated using GCN;

[0052] The text embedding vector and the graph embedding vector are input into the encoder-decoder model based on the Transformer architecture, and semantic alignment is performed through cross-modal attention in conjunction with the initial semantic association.

[0053] The aligned joint semantics are fed into the decoder to achieve the textual representation of graphical content and the structured completion of textual content with generative objectives, and a fused multimodal semantic representation is constructed using a unified representation template.

[0054] A consistency constraint term is introduced to verify the consistency of symbolic meaning, topological logic, spatial relationships, and process logic of the generated multimodal semantic representation. After the verification is passed, the model parameters are updated to complete the training.

[0055] Optionally, based on the unified knowledge representation, the content of the document to be processed is subjected to automated, refined, and interpretable understanding processing, including:

[0056] The document to be processed is subjected to at least one application, which includes: performing semantic-level retrieval of document illustrations based on the unified knowledge representation and querying attributes through natural language; performing parametric editing or style transfer of document illustrations based on the unified knowledge representation and synchronously updating associated attribute data; and generating intelligent question-and-answer or summary reports containing image-text comparisons for document content based on the unified knowledge representation.

[0057] Secondly, embodiments of this application provide a document-text integrated intelligent understanding and processing system, including:

[0058] The document parsing module is used to obtain the document to be processed and generate a document object model based on the document to be processed.

[0059] The element extraction and association module is used to separate the elements of the document content to be processed using the document object model, extract text elements and image elements, and establish the initial semantic association between the image elements and their corresponding context text based on the layout structure, figure title, figure legend and text reference relationship;

[0060] The vectorization processing module is used to call a general graphics vectorization engine to perform intelligent raster-to-vector processing on the image elements. Through deep visual feature extraction, it identifies and restores the original appearance of vector graphics to obtain a basic vector primitive set containing points, lines, surfaces, volumes, text and line types, and legend symbols.

[0061] The attribute analysis and classification module is used to perform semantic analysis on the text elements based on the initial semantic association, extract attribute description information, and combine the geometric features of the basic vector primitive set to perform element layering and semantic classification on the basic vector primitive set.

[0062] The entity reconstruction module is used to assign attributes to the relevant graphic elements after classification based on the attribute description information, and to perform structured reconstruction and semantic restoration of semantically meaningless geometric primitives, reconstructing them into vector entity objects with complete attribute information.

[0063] The large model reasoning module is used to input the text elements and the structured reconstruction and semantic restoration vector entity objects into a pre-trained multimodal large language model. By utilizing the reasoning ability of the multimodal large language model, cross-modal representation alignment and consistency verification are performed to generate a unified knowledge representation that includes graph topology, entity attributes and document context.

[0064] The application processing module is used to perform automated, refined, and interpretable understanding processing of the content of the document to be processed based on the unified knowledge representation.

[0065] The document-text integrated intelligent understanding and processing method proposed in this application achieves automatic parsing, knowledge reconstruction, and semantic-level reasoning of text and image content in documents through document structured parsing, graphic semantic restoration, multimodal semantic fusion modeling, and intelligent reasoning applications. By constructing a multi-stage collaborative processing system of "document structured preprocessing—illustration extraction—vectorization—attribute analysis—object reconstruction—large model understanding—intelligent understanding application," a high-precision, multimodal semantic understanding solution for engineering scenarios is formed.

[0066] In this application, a document object model is generated based on the parsing of the document to be processed, which is equivalent to document structure preprocessing; the document object model is used to separate the elements of the document content to be processed, extracting text elements and illustration image elements, and establishing an initial semantic association between the illustration image elements and their corresponding context text, which is equivalent to illustration extraction; intelligent raster-to-vector processing is performed on the image elements, and the original appearance of the vector graphics is identified and restored through deep visual feature extraction to obtain a basic vector primitive set, which is equivalent to vectorization.

[0067] Semantic analysis is performed on text elements to extract attribute description information. Combined with the geometric features of the basic vector primitive set, the basic vector primitive set is layered and semantically classified, which is equivalent to attribute analysis. Attributes are assigned to the classified related graphic elements, and semantically non-semantic geometric primitives are restructured and semantically restored to reconstruct vector entity objects with complete attribute information, which is equivalent to object reconstruction.

[0068] By leveraging the reasoning capabilities of a multimodal large language model, cross-modal representation alignment and consistency verification are performed to generate a unified knowledge representation that includes graph topology, entity attributes, and document context, which is equivalent to large model understanding. Based on the unified knowledge representation, the content of the document to be processed is automatically, refined, and interpretable, which is equivalent to intelligent understanding application.

[0069] This application enables integrated intelligent understanding of text and rasterized illustrations, achieving collaborative parsing, structured extraction, and unified knowledge representation. It overcomes key technological bottlenecks in existing automated document understanding, improving document information processing efficiency and intelligence. It effectively supports the intelligent needs of document review, modification, maintenance, and delivery. It solves the problems of traditional document management methods that rely on manual professionals to check and interpret each page, resulting in low information retrieval efficiency, high error risk, and difficulty in knowledge transfer.

[0070] This application proposes a cross-modal, reasonable document semantic fusion model, which achieves collaborative understanding and unified knowledge representation of illustrations and text content. It overcomes the bottleneck of existing document processing technologies in modeling graphic semantics and cross-modal associations, and solves the problems of loss of graphic semantics and difficulty in fine-grained parsing caused by vector-to-raster conversion in documents such as papers and reports. It significantly improves the overall understanding depth and interpretability of AI for complex graphic documents, provides high-performance semantic understanding capabilities for document-based training and intelligent processing, and significantly enhances the intelligent processing capability, understanding depth and traceability of document content. It has broad application value and practicality in engineering design, drawing review and verification, safety production management and operation and maintenance knowledge management. Attached Figure Description

[0071] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0072] Figure 1 This is a flowchart of a document-text integrated intelligent understanding and processing method proposed in an embodiment of this application.

[0073] Figure 2 This is a block diagram of a document-text integrated intelligent understanding and processing system proposed in an embodiment of this application. Detailed Implementation

[0074] The embodiments of this application will now be described in detail. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0075] This application proposes a document-text integrated intelligent understanding and processing method, referring to... Figure 1 The flowchart shown includes:

[0076] Step 101: Obtain the document to be processed and generate a document object model based on the document to be processed.

[0077] To achieve intelligent understanding and processing of documents and images, it is first necessary to obtain the document to be processed and generate a document object model based on the obtained document.

[0078] Step 102: Use the document object model to separate the elements of the document content to be processed, extract text elements and image elements, and establish the initial semantic association between the image elements and their corresponding context text based on the layout structure, figure titles, legends and text references.

[0079] Once the document object model is obtained, it can be used to separate the document content into elements, extracting text and image elements. Simultaneously, based on the layout structure, captions, legends, and text references, an initial semantic association is established between the image elements and their corresponding contextual text.

[0080] Steps 101 and 102 above can be understood as performing document structuring preprocessing and illustration extraction on the document to be processed. Through layout structure analysis, text recognition, and illustration region extraction, a document object model is established to distinguish between text segments and illustration elements, resulting in text elements and image elements. Based on the layout structure (e.g., layout block positions), illustration titles (e.g., illustration label), legends (e.g., legend descriptions), and text citation relationships (e.g., text citation statements), an initial correspondence between the illustration and text is constructed. These two steps ensure that the subsequent semantic analysis of the illustration can disambiguate with contextual information and complete the semantic expression, providing a structural foundation for the overall text-image fusion.

[0081] In one embodiment of this application, a preferred method for establishing an initial semantic association between an image element and its corresponding context text includes:

[0082] The document to be processed is parsed for its layout structure. Optical character recognition is used to extract the corresponding text content to form text elements. Then, document layout area detection or image segmentation is used to extract the corresponding illustration areas to form image elements. Based on the document layout position, figure title number, text citation mark and semantic similarity in the document to be processed, a correspondence is established between text paragraphs and illustrations, thereby obtaining the initial semantic association between image elements and their corresponding context text.

[0083] Step 103: Call the general-purpose graphics vectorization engine based on the deep learning model to perform intelligent raster-to-vector processing on the image elements. Through deep visual feature extraction, identify and restore the original appearance of the vector graphics to obtain a basic vector primitive set containing points, lines, surfaces, volumes, text and line types, and legend symbols.

[0084] After completing the first two steps and obtaining the image elements, you can call the general-purpose graphics vectorization engine based on a deep learning model to perform intelligent raster-to-vector processing on the image elements. Through deep visual feature extraction, it can identify and restore the original appearance of vector graphics, and obtain a basic vector primitive set containing points, lines, surfaces, volumes, text and line types, and legend symbols, which is equivalent to realizing vectorization.

[0085] In one embodiment of this application, preferably, the method for performing intelligent raster-to-vector conversion on image elements to extract and restore the original appearance of vector graphics includes:

[0086] First, a visual encoder is constructed using a convolutional neural network (CNN) or visual Transformer architecture to extract high-dimensional visual features from image elements. Then, the image elements after high-dimensional visual feature extraction are processed by an object detection and instance segmentation network to accurately identify and segment legend symbols, endpoints of connecting lines, intersections, and arrow directions from complex background noise.

[0087] Then, using a geometric primitive prediction model, the segmented pixel regions are transformed into parameterized vector representations; the legend regions in the image are located, the legends are vectorized and extracted to obtain the geometric shape of the legend symbols and their corresponding textual interpretations, and a legend feature library is established; the optical character recognition model is used to extract text annotations in the image and record their coordinate positions.

[0088] Finally, the identified and obtained elements—legendary symbols, endpoints of connecting lines, intersections and arrow directions, as well as parameterized vector representations, geometric shapes of legendary symbols and their corresponding textual interpretations, and textual annotations recording their coordinate positions—can be understood as identified and parameterized points, lines, surfaces, linear elements, legendary symbols, and textual annotations. These are combined to form a basic vector primitive set, which includes points, lines, surfaces, volumes, text and line types, and legendary symbols.

[0089] Step 104: Based on the initial semantic association, perform semantic analysis on the text elements, extract attribute description information, and combine the geometric features of the basic vector primitive set to perform element layering and semantic classification on the basic vector primitive set.

[0090] After obtaining the basic vector primitive set, semantic analysis is performed on the text elements based on the initial semantic association to extract attribute description information. Then, combined with the geometric features of the basic vector primitive set, the basic vector primitive set is layered and semantically classified.

[0091] In one embodiment of this application, a preferred method for element layering and semantic classification of the basic vector primitive set includes:

[0092] First, natural language processing algorithms are used to perform entity recognition and relationship extraction on text paragraphs, figure titles and captions associated with image elements, and key information including entity name, specification parameters, material properties and functional descriptions is extracted as attribute description information.

[0093] Next, a topological analysis is performed on the geometric lines in the basic vector primitive set. Based on line type characteristics and connection relationships, contour lines, leader lines, dimension lines, and center lines are distinguished, and the geometric elements are separated into layers; this is called element layering. The entity keywords in the extracted attribute description information are mapped and matched with the legend symbols or geometric shapes in the basic vector primitive set, thereby classifying semantically insensitive geometric figures into different semantic categories; this is called semantic classification.

[0094] Step 105: Based on the attribute description information, assign attributes to the relevant graphic elements after classification, and reconstruct and restore the semantics of the semantically null geometric primitives into vector entity objects with complete attribute information.

[0095] After performing element layering and semantic classification on the basic vector primitive set through the aforementioned steps, attributes are assigned to the classified related graphic elements based on attribute description information. This process restructures and restores the semantics of semantically nuanced geometric primitives, reconstructing them into vector entity objects with complete attribute information.

[0096] In one embodiment of this application, a preferred method for structural reconstruction and semantic restoration of semantically null geometric primitives includes:

[0097] Based on the aforementioned feature library of illustrations, matching geometric shapes are retrieved from the basic vector primitive set. Semantic labels are assigned to the matched geometric primitives according to the textual interpretations in the illustrations. Based on the textual annotations and their coordinate positions within the image, geometric primitives with the closest spatial distance or connected by leading lines are identified, and the textual annotation content is assigned as attribute information to these geometric primitives. According to the connection relationship descriptions in the attribute descriptions, occluded or broken vector lines are identified, and connectivity is repaired using geometric extension and fitting algorithms to construct a complete topological connection network. Finally, the geometric primitives assigned semantic labels, attribute information, and topological relationships are encapsulated as independent vector entity objects to perform structured reconstruction and semantic restoration of semantically nuanced geometric primitives.

[0098] Step 106: Input the text elements and the vector entity objects of structured reconstruction and semantic restoration into the pre-trained multimodal large language model. Utilize the reasoning ability of the multimodal large language model to perform cross-modal representation alignment and consistency verification, and generate a unified knowledge representation that includes graph topology, entity attributes and document context.

[0099] After structural reconstruction and semantic restoration of semantically non-semantic geometric primitives, text elements and vector entity objects that have undergone structural reconstruction and semantic restoration are input into a pre-trained multimodal large language model. The reasoning ability of the multimodal large language model is used to perform cross-modal representation alignment and consistency verification, generating a unified knowledge representation that includes graph topology, entity attributes and document context.

[0100] In one embodiment of this application, a preferred method for generating a unified knowledge representation that includes graph topology, entity attributes, and document context includes:

[0101] Vector entity objects are serialized into a structured description language or code format and used as visual modal input; text elements and extracted attribute description information are used as text modal input; prompt engineering instructions are constructed to guide the multimodal large language model to verify and correct entity names, connection relationships and attribute values ​​in the visual modal input based on the text modal input.

[0102] By leveraging the reasoning capabilities of a multimodal large language model, the implicit functional logic or causal relationships between vector entity objects are derived; finally, a graph structure data containing entity nodes, attribute edges, and relation edges is output as a unified knowledge representation.

[0103] The multimodal large language model used in the above process is an encoder-decoder model based on the Transformer architecture. Its input modalities include: text embedding vectors and graph embedding vectors of structured image semantics.

[0104] In one embodiment of this application, preferably, during the training process of the multimodal large language model for optimization training, a consistency constraint term can be introduced to ensure logical consistency between the structured reconstruction and semantic restoration of image elements and the statements of facts in text elements. The specific process includes:

[0105] Text elements are segmented and context-encoded to generate text embedding vectors; image elements are reconstructed and semantically restored using graph structure encoding, and graph embedding vectors are generated using GCN.

[0106] After obtaining the text embedding vector and graph embedding vector, the text embedding vector and graph embedding vector are input into the encoder-decoder model based on the Transformer architecture, that is, input to the multimodal large language model, and then combined with the initial semantic association, semantic alignment is performed through cross-modal attention.

[0107] The aligned joint semantics are fed into the decoder to achieve the textual representation of graphical content and the structured completion of textual content with a generative objective. A unified representation template is then used to construct the fused multimodal semantic representation. Subsequently, a consistency constraint term is introduced to verify the consistency of symbolic meaning, topological logic, spatial relationships, and process logic of the generated multimodal semantic representation. After the verification passes, the model parameters are updated to complete the training.

[0108] Step 106 employs a multimodal large model based on the Transformer architecture to jointly model document text (i.e., text embedding vectors) embeddings and image structured semantic graph (i.e., graph embedding vectors) embeddings. Through cross-modal semantic consistency constraints and generative reasoning mechanisms, a unified knowledge representation space is constructed. This enables semantic completion, concept regularization, and logical consistency reasoning between graphical entities and text descriptions, thereby significantly improving the interpretability of implicit relationships in document content.

[0109] Step 107: Based on the unified knowledge representation, perform automated, refined, and interpretable understanding of the content of the document to be processed.

[0110] After generating a unified knowledge representation, the content of the document to be processed can be automatically, refined, and interpreted based on this unified knowledge representation. Preferably, the automated, refined, and interpretable processing of the document content includes:

[0111] The document to be processed shall be subjected to at least one application, which includes: semantic-level retrieval of document illustrations based on unified knowledge representation, querying attributes through natural language; parametric editing or style transfer of document illustrations based on unified knowledge representation, and synchronously updating associated attribute data; and generating intelligent question-answering or summary reports containing text-image comparisons based on unified knowledge representation. Only one of these applications may be performed, or multiple or all of them may be performed simultaneously, depending on the specific needs.

[0112] Based on the above-mentioned document-text integrated intelligent understanding and processing method, this application also proposes a document-text integrated intelligent understanding and processing system, referring to... Figure 2 The block diagram shown includes:

[0113] The document parsing module 210 is used to obtain the document to be processed and generate a document object model based on the document to be processed.

[0114] The element extraction and association module 220 is used to separate the elements of the document content to be processed using the document object model, extract text elements and image elements, and establish an initial semantic association between the image elements and their corresponding context text based on the layout structure, figure title, figure legend and text reference relationship;

[0115] The vectorization processing module 230 is used to call a general graphics vectorization engine to perform intelligent raster-to-vector processing on the image elements. Through deep visual feature extraction, it identifies and restores the original appearance of vector graphics to obtain a basic vector primitive set containing points, lines, surfaces, volumes, text, line types, and legend symbols.

[0116] The attribute analysis and classification module 240 is used to perform semantic analysis on the text elements based on the initial semantic association, extract attribute description information, and combine the geometric features of the basic vector primitive set to perform element layering and semantic classification on the basic vector primitive set.

[0117] The entity reconstruction module 250 is used to assign attributes to the classified related graphic elements based on the attribute description information, and to perform structured reconstruction and semantic restoration of semantically meaningless geometric primitives, reconstructing them into vector entity objects with complete attribute information.

[0118] The large model reasoning module 260 is used to input the text elements and the vector entity objects of the structured reconstruction and semantic restoration into the pre-trained multimodal large language model, and use the reasoning ability of the multimodal large language model to perform cross-modal representation alignment and consistency verification, and generate a unified knowledge representation that includes graph topology, entity attributes and document context.

[0119] The application processing module 270 is used to perform automated, refined, and interpretable understanding processing of the content of the document to be processed based on the unified knowledge representation.

[0120] Optionally, the element extraction and association module 220 is specifically used for:

[0121] The document to be processed is parsed for layout structure, and the corresponding text content is extracted using optical character recognition method to form the text element;

[0122] The corresponding illustration regions are extracted using document layout region detection or image segmentation methods to form the image elements;

[0123] Based on the document layout position, figure title number, text citation mark and semantic similarity in the document to be processed, a correspondence is established between the text paragraphs and the illustrations, thereby obtaining the initial semantic association between the image elements and their corresponding context text.

[0124] Optionally, the vectorization processing module 230 is specifically used for:

[0125] A visual encoder is constructed using a convolutional neural network or a visual Transformer architecture to extract high-dimensional visual features from the image elements;

[0126] After extracting high-dimensional visual features, image elements are processed by a target detection and instance segmentation network to accurately identify and segment legend symbols, endpoints of connecting lines, intersections, and arrow directions from background noise.

[0127] Using a geometric primitive prediction model, the segmented pixel regions are transformed into parameterized vector representations;

[0128] Locate the legend region in the image, extract the legend by vectorization, obtain the geometric shape of the legend symbol and its corresponding text interpretation, and establish a legend feature library;

[0129] The optical character recognition model is used to extract text annotations within an image and record their coordinate positions.

[0130] The basic vector primitive set is formed by combining the legend symbols, endpoints of connecting lines, intersections and arrow points, as well as the parameterized vector representation, the geometric shape of the legend symbols and their corresponding textual interpretations, and textual annotations recording their coordinate positions. This basic vector primitive set includes points, lines, surfaces, volumes, text and line types, and legend symbols.

[0131] Optionally, the attribute analysis and classification module 240 is specifically used for:

[0132] Natural language processing algorithms are used to perform entity recognition and relation extraction on text paragraphs, figure titles and captions associated with image elements, and key information including entity name, specification parameters, material properties and functional description is extracted as the attribute description information;

[0133] A topological analysis is performed on the geometric lines in the basic vector element set. Based on the line type characteristics and connection relationships, the outline lines, leader lines, dimension lines and center lines are distinguished, and the geometric elements are separated into layers, that is, the elements are layered.

[0134] The entity keywords in the extracted attribute description information are mapped and matched with the legend symbols or geometric shapes in the basic vector primitive set, thereby classifying semantically meaningless geometric figures into different semantic categories, i.e., performing semantic classification.

[0135] Optionally, the entity reconstruction module 250 is specifically used for:

[0136] Based on the legend feature library, matching geometric shapes are retrieved from the basic vector primitive set, and corresponding semantic tags are assigned to the matching geometric primitives according to the text interpretation in the legend.

[0137] Based on the text annotations and their coordinate positions in the image, find the geometric primitives that are closest in space or connected by leaders, and assign the text annotation content as attribute information to the geometric primitive.

[0138] Based on the connection relationship description in the attribute description information, the occluded or broken vector lines are identified, and the connectivity is repaired based on the geometric extension and fitting algorithm to construct a complete topological connection network.

[0139] Geometric primitives endowed with semantic labels, attribute information, and topological relationships are encapsulated into independent vector entity objects to perform structured reconstruction and semantic restoration of semantically meaningless geometric primitives.

[0140] Optionally, the large model inference module 260 is specifically used for:

[0141] The vector entity object is serialized into a structured description language or code format and used as visual modal input;

[0142] The text elements and extracted attribute description information are used as text modal input;

[0143] The system constructs prompting instructions to guide the multimodal large language model to verify and correct entity names, connection relationships, and attribute values ​​in the visual modal input based on the text modal input.

[0144] By utilizing the reasoning capabilities of the multimodal large language model, the implicit functional logic or causal relationships between vector entity objects can be derived.

[0145] The output is a graph structure data containing entity nodes, attribute edges, and relation edges, which serves as the unified knowledge representation.

[0146] The pre-trained multimodal large language model is an encoder-decoder model based on the Transformer architecture, and its input modalities include: text embedding vectors and vector image embedding vectors.

[0147] During the fine-tuning training of the multimodal large language model, a consistency constraint is introduced to ensure logical consistency between the structured reconstruction and semantic restoration and the factual statements in the text elements. The specific process includes:

[0148] Perform word segmentation and context encoding on the text elements in the sample to generate the text embedding vector;

[0149] Graph structure encoding is performed on the structured reconstruction and semantic restoration of the samples, and the graph embedding vector is generated using GCN;

[0150] The text embedding vector and the graph embedding vector are input into the encoder-decoder model based on the Transformer architecture, and semantic alignment is performed through cross-modal attention in conjunction with the initial semantic association.

[0151] The aligned joint semantics are fed into the decoder to achieve the textual representation of graphical content and the structured completion of textual content with generative objectives, and a fused multimodal semantic representation is constructed using a unified representation template.

[0152] A consistency constraint term is introduced to verify the consistency of symbolic meaning, topological logic, spatial relationship and process logic of the multimodal semantic expression of the generated output, and the model parameters are updated after the verification is passed to complete the training.

[0153] The document to be processed is subjected to at least one application, which includes: performing semantic-level retrieval of document illustrations based on the unified knowledge representation and querying attributes through natural language; performing parametric editing or style transfer of document illustrations based on the unified knowledge representation and synchronously updating associated attribute data; and generating intelligent question-and-answer or summary reports containing image-text comparisons for document content based on the unified knowledge representation.

[0154] In summary, the document-text integrated intelligent understanding and processing method proposed in this application achieves automatic parsing, knowledge reconstruction, and semantic-level reasoning of text and image content in documents through document structured parsing, graphic semantic restoration, multimodal semantic fusion modeling, and intelligent reasoning applications. By constructing a multi-stage collaborative processing system of "document structured preprocessing—illustration extraction—vectorization—attribute analysis—object reconstruction—large model understanding—intelligent understanding application," a high-precision, multimodal semantic understanding solution for engineering scenarios is formed.

[0155] In this application, a document object model is generated based on the parsing of the document to be processed, which is equivalent to document structure preprocessing; the document object model is used to separate the elements of the document content to be processed, extract text elements and image elements, and establish the initial semantic association between the image elements and their corresponding context text, which is equivalent to illustration extraction; intelligent raster to vector processing is performed on the image elements, and the original appearance of the vector graphics is identified and restored through deep visual feature extraction to obtain a basic vector primitive set, which is equivalent to vectorization.

[0156] Semantic analysis is performed on text elements to extract attribute description information. Combined with the geometric features of the basic vector primitive set, the basic vector primitive set is layered and semantically classified, which is equivalent to attribute analysis. Attributes are assigned to the classified related graphic elements, and semantically non-semantic geometric primitives are restructured and semantically restored to reconstruct vector entity objects with complete attribute information, which is equivalent to object reconstruction.

[0157] By leveraging the reasoning capabilities of a multimodal large language model, cross-modal representation alignment and consistency verification are performed to generate a unified knowledge representation that includes graph topology, entity attributes, and document context, which is equivalent to large model understanding. Based on the unified knowledge representation, the content of the document to be processed is automatically, refined, and interpretable, which is equivalent to intelligent understanding application.

[0158] This application enables integrated intelligent understanding of text and rasterized illustrations, achieving collaborative parsing, structured extraction, and unified knowledge representation. It overcomes key technological bottlenecks in existing automated document understanding, improving document information processing efficiency and intelligence. It effectively supports the intelligent needs of document review, modification, maintenance, and delivery. It solves the problems of traditional document management methods that rely on manual professionals to check and interpret each page, resulting in low information retrieval efficiency, high error risk, and difficulty in knowledge transfer.

[0159] This application proposes a cross-modal, reasonable document semantic fusion model, which achieves collaborative understanding and unified knowledge representation of illustrations and text content. It overcomes the bottleneck of existing document processing technologies in modeling graphic semantics and cross-modal associations, and solves the problems of loss of graphic semantics and difficulty in fine-grained parsing caused by vector-to-raster conversion in documents such as papers and reports. It significantly improves the overall understanding depth and interpretability of AI for complex graphic documents, provides high-performance semantic understanding capabilities for document-based training and intelligent processing, and significantly enhances the intelligent processing capability, understanding depth and traceability of document content. It has broad application value and practicality in engineering design, drawing review and verification, safety production management and operation and maintenance knowledge management.

[0160] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0161] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0162] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims. All of these forms are within the protection scope of this application.

Claims

1. A document-text integrated intelligent understanding and processing method, characterized in that, include: Obtain the document to be processed, and generate a document object model based on the document to be processed; The document object model is used to separate the elements of the document to be processed, extracting text elements and image elements. Based on the layout structure, figure titles, legends, and text citations, an initial semantic association is established between the image elements and their corresponding context text. Specifically, this includes: performing layout structure parsing on the document to be processed, using optical character recognition (OCR) to extract the corresponding text content to form the text elements; using document layout region detection or image segmentation methods to extract the corresponding illustration regions to form the image elements; and establishing a correspondence between text paragraphs and illustrations based on the document layout position, figure title number, text citation markers, and semantic similarity in the document to be processed, thereby obtaining the initial semantic association between the image elements and their corresponding context text. The general-purpose graphics vectorization engine based on a deep learning model is invoked to perform intelligent raster-to-vector processing on the image elements. Through deep visual feature extraction, the original appearance of the vector graphics is identified and restored to obtain a basic vector primitive set containing points, lines, surfaces, volumes, text and line types, and legend symbols. Based on the initial semantic association, semantic analysis is performed on the text elements to extract attribute description information, and combined with the geometric features of the basic vector primitive set, the basic vector primitive set is subjected to element layering and semantic classification. Based on the attribute description information, attributes are assigned to the relevant graphic elements after classification, and semantically unsemantic geometric primitives are reconstructed and semantically restored, and reconstructed into vector entity objects with complete attribute information. The text elements and the vector entity objects of the structured reconstruction and semantic restoration are input into a pre-trained multimodal large language model. The reasoning ability of the multimodal large language model is used to perform cross-modal representation alignment and consistency verification, and generate a unified knowledge representation that includes graph topology, entity attributes and document context. Based on the unified knowledge representation, the content of the document to be processed is automatically, precisely, and interpretably understood.

2. The document-text integrated intelligent understanding and processing method according to claim 1, characterized in that, The system invokes a general-purpose graphics vectorization engine based on a deep learning model to perform intelligent raster-to-vector conversion on the image elements, extracting and restoring the original appearance of the illustration vector graphics, including: A visual encoder is constructed using a convolutional neural network or a visual Transformer architecture to extract high-dimensional visual features from the image elements; After extracting high-dimensional visual features, image elements are processed by a target detection and instance segmentation network to accurately identify and segment legend symbols, endpoints of connecting lines, intersections, and arrow directions from background noise. Using a geometric primitive prediction model, the segmented pixel regions are transformed into parameterized vector representations; Locate the legend region in the image, extract the legend by vectorization, obtain the geometric shape of the legend symbol and its corresponding text interpretation, and establish a legend feature library; The optical character recognition model is used to extract text annotations within an image and record their coordinate positions. The basic vector primitive set is formed by combining the legend symbols, endpoints of connecting lines, intersections and arrow points, as well as the parameterized vector representation, the geometric shape of the legend symbols and their corresponding textual interpretations, and textual annotations recording their coordinate positions. This basic vector primitive set includes points, lines, surfaces, volumes, text and line types, and legend symbols.

3. The document-text integrated intelligent understanding and processing method according to claim 2, characterized in that, Based on the initial semantic association, semantic analysis is performed on the text elements to extract attribute description information. Then, combined with the geometric features of the basic vector primitive set, element stratification and semantic classification are performed on the basic vector primitive set, including: Natural language processing algorithms are used to perform entity recognition and relation extraction on text paragraphs, figure titles and captions associated with image elements, and key information including entity name, specification parameters, material properties and functional description is extracted as the attribute description information; A topological analysis is performed on the geometric lines in the basic vector element set. Based on the line type characteristics and connection relationships, the outline lines, leader lines, dimension lines and center lines are distinguished, and the geometric elements are separated into layers, that is, the elements are layered. The entity keywords in the extracted attribute description information are mapped and matched with the legend symbols or geometric shapes in the basic vector primitive set, thereby classifying semantically meaningless geometric figures into different semantic categories, i.e., performing semantic classification.

4. The document-text integrated intelligent understanding and processing method according to claim 3, characterized in that, Based on the attribute description information, attributes are assigned to the classified related graphic elements, and semantically meaningless geometric primitives are structurally reconstructed and semantically restored, including: Based on the legend feature library, matching geometric shapes are retrieved from the basic vector primitive set, and corresponding semantic tags are assigned to the matching geometric primitives according to the text interpretation in the legend. Based on the text annotations and their coordinate positions in the image, find the geometric primitives that are closest in space or connected by leaders, and assign the text annotation content as attribute information to the geometric primitive. Based on the connection relationship description in the attribute description information, the occluded or broken vector lines are identified, and the connectivity is repaired based on the geometric extension and fitting algorithm to construct a complete topological connection network. Geometric primitives endowed with semantic labels, attribute information, and topological relationships are encapsulated into independent vector entity objects to perform structured reconstruction and semantic restoration of semantically meaningless geometric primitives.

5. The document-text integrated intelligent understanding and processing method according to claim 1, characterized in that, The text elements and the structured reconstruction and semantic restoration vector entity objects are jointly input into a pre-trained multimodal large language model. Utilizing the reasoning capabilities of the multimodal large language model, cross-modal representation alignment and consistency verification are performed to generate a unified knowledge representation that includes graph topology, entity attributes, and document context, including: The vector entity object is serialized into a structured description language or code format and used as visual modal input; The text elements and extracted attribute description information are used as text modal input; The system constructs prompting instructions to guide the multimodal large language model to verify and correct entity names, connection relationships, and attribute values ​​in the visual modal input based on the text modal input. By utilizing the reasoning capabilities of the multimodal large language model, the implicit functional logic or causal relationships between vector entity objects can be derived. The output is a graph structure data containing entity nodes, attribute edges, and relation edges, which serves as the unified knowledge representation.

6. The document-text integrated intelligent understanding and processing method according to claim 5, characterized in that, The pre-trained multimodal large language model is an encoder-decoder model based on the Transformer architecture, whose input modalities include: text embedding vectors and vector image embedding vectors. During the fine-tuning training of the multimodal large language model, a consistency constraint is introduced to ensure logical consistency between the structured reconstruction and semantic restoration and the factual statements in the text elements. The specific process includes: Perform word segmentation and context encoding on the text elements in the sample to generate the text embedding vector; Graph structure encoding is performed on the structured reconstruction and semantic restoration of the samples, and the graph embedding vector is generated using GCN; The text embedding vector and the graph embedding vector are input into the encoder-decoder model based on the Transformer architecture, and semantic alignment is performed through cross-modal attention in conjunction with the initial semantic association. The aligned joint semantics are fed into the decoder to achieve the textual representation of graphical content and the structured completion of textual content with generative objectives, and a fused multimodal semantic representation is constructed using a unified representation template. A consistency constraint term is introduced to verify the consistency of symbolic meaning, topological logic, spatial relationships, and process logic of the generated multimodal semantic representation. After the verification is passed, the model parameters are updated to complete the training.

7. The document-text integrated intelligent understanding and processing method according to claim 1, characterized in that, Based on the unified knowledge representation, the content of the document to be processed is automatically, precisely, and interpretably understood, including: The document to be processed is subjected to at least one application, which includes: performing semantic-level retrieval of document illustrations based on the unified knowledge representation and querying attributes through natural language; performing parametric editing or style transfer of document illustrations based on the unified knowledge representation and synchronously updating associated attribute data; and generating intelligent question-and-answer or summary reports containing image-text comparisons for document content based on the unified knowledge representation.

8. A document-text integrated intelligent understanding and processing system, characterized in that, include: The document parsing module is used to obtain the document to be processed and generate a document object model based on the document to be processed. The element extraction and association module is used to separate elements from the document content to be processed using the document object model, extract text elements and image elements, and establish an initial semantic association between the image elements and their corresponding context text based on the layout structure, figure titles, legends, and text references. Specifically, it is used to: perform layout structure parsing on the document to be processed, extract the corresponding text content using optical character recognition methods to form the text elements; and extract the corresponding illustration regions using document layout region detection or image segmentation methods to form the image elements. Based on the document layout position, figure title number, text citation mark and semantic similarity in the document to be processed, a correspondence is established between the text paragraphs and the illustrations, thereby obtaining the initial semantic association between the image elements and their corresponding context text. The vectorization processing module is used to call a general graphics vectorization engine to perform intelligent raster-to-vector processing on the image elements. Through deep visual feature extraction, it identifies and restores the original appearance of vector graphics to obtain a basic vector primitive set containing points, lines, surfaces, volumes, text and line types, and legend symbols. The attribute analysis and classification module is used to perform semantic analysis on the text elements based on the initial semantic association, extract attribute description information, and combine the geometric features of the basic vector primitive set to perform element layering and semantic classification on the basic vector primitive set. The entity reconstruction module is used to assign attributes to the relevant graphic elements after classification based on the attribute description information, and to perform structured reconstruction and semantic restoration of semantically meaningless geometric primitives, reconstructing them into vector entity objects with complete attribute information. The large model reasoning module is used to input the text elements and the structured reconstruction and semantic restoration vector entity objects into a pre-trained multimodal large language model. By utilizing the reasoning ability of the multimodal large language model, cross-modal representation alignment and consistency verification are performed to generate a unified knowledge representation that includes graph topology, entity attributes and document context. The application processing module is used to perform automated, refined, and interpretable understanding processing of the content of the document to be processed based on the unified knowledge representation.

Images