Method and system for generating official document key abstract based on multi-modal feature extraction

Abstract

The invention provides an official document key abstract generation method and system based on multi-modal feature extraction, and relates to the technical field of multi-modal artificial intelligence generation, and the method comprises the steps: firstly obtaining text modal data, image modal data and table modal data of a to-be-processed official document, then carrying out the hierarchical semantic analysis processing of the text modal data, and obtaining a to-be-processed official document key abstract; the method comprises the following steps: generating a text semantic feature set, performing visual element extraction processing on image modal data, generating an image feature set, performing structured analysis processing on table modal data, generating a table feature set, and performing cross-modal alignment processing on the text semantic feature set, the image feature set and the table feature set. According to the method, the dynamic association feature sets among the text semantics, the image elements and the table elements are determined according to the text semantics, the image elements and the table elements, the three feature sets are subjected to multi-modal fusion processing according to the feature sets, the target abstract content of the to-be-processed official document is generated, complementarity of multi-modal information in the official document is fully utilized, and the generated abstract is more complete, accurate and targeted.

Description

Technical Field

[0001] The present invention relates to the field of multimodal artificial intelligence generation technology, and in particular to a method and system for generating key summaries of official documents based on multimodal feature extraction. Background Art

[0002] In today's era of rapid digital information development, official documents, as crucial documents for conveying important information, regulating administrative affairs, and guiding work, are increasingly diverse in content and presentation. Traditional methods for generating key summaries for official documents primarily focus on the textual content of the document, processing only the text portion. These methods typically employ techniques such as keyword extraction and sentence compression to filter out key information from the text to generate summaries.

[0003] However, with the increasing pace of office automation and informatization, modern official documents, in addition to extensive text descriptions, often use non-textual formats such as images and tables to present data, demonstrate results, or illustrate issues. Images can intuitively display scenarios, processes, trends, and other information, while tables can clearly present data comparisons, classification statistics, and other content. The information contained in these images and tables is crucial for understanding the core points and complete content of official documents.

[0004] Existing document summary generation technology focuses solely on textual content, completely ignoring the rich information contained in images and tables within documents. This results in incomplete summaries that fail to accurately reflect the document's overall message and key points. Furthermore, because it fails to consider the potential inherent connections between text, images, and tables, summaries deviate from the document's actual message. This inability to provide users with complete and accurate information references makes it difficult to meet the practical needs of quickly and accurately understanding documents. Summary of the Invention

[0005] In view of the above-mentioned problems, in combination with the first aspect of the present invention, an embodiment of the present invention provides a method for generating a key summary of an official document based on multimodal feature extraction, the method comprising: Acquire a multimodal data set of the document to be processed, wherein the multimodal data set includes text modal data, image modal data, and table modal data; Performing hierarchical semantic parsing on the text modal data to generate a text semantic feature set of the document to be processed; Performing visual element extraction processing on the image modality data to generate an image feature set of the document to be processed; Performing structured parsing on the form modal data to generate a form feature set of the document to be processed; Performing cross-modal alignment processing on the text semantic feature set, the image feature set, and the table feature set to determine a dynamic association feature set between text semantics, image elements, and table elements; Based on the dynamic association feature set, multimodal fusion processing is performed on the text semantic feature set, the image feature set and the table feature set to generate the target summary content of the document to be processed.

[0006] On the other hand, an embodiment of the present invention also provides a system for generating key summaries of official documents based on multimodal feature extraction, including a processor and a machine-readable storage medium, wherein the machine-readable storage medium is connected to the processor, the machine-readable storage medium is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the machine-readable storage medium to implement the above method.

[0007] Based on the above aspects, by comprehensively acquiring a multimodal data set containing text modal data, image modal data, and table modal data in the official documents to be processed, a special processing method is adopted for data of different modalities. The text modal data is subjected to layered semantic parsing processing to deeply analyze the semantic structure of the text and accurately extract the semantic features of the text; the image modal data is subjected to visual element extraction processing to effectively capture the key visual information in the image; the table modal data is subjected to structured parsing processing to clearly sort out the data relationships and structural features in the table. Through cross-modal alignment processing, the dynamic correlation feature set between text semantics, image elements and table elements is determined, and the deep fusion and mutual correlation between different modal information is achieved. Based on this dynamic correlation feature set, multimodal fusion processing is performed to organically combine the features of text, image and table. The generated target summary content can comprehensively and accurately reflect the core points and complete information of the official document. Compared with traditional summary generation methods based only on text, it breaks the limitations of single-modal processing, fully utilizes the complementarity of multimodal information in official documents, and can explore the potential connections hidden between different modal data. The generated summary is more complete, accurate, and targeted, greatly improving the quality and practicality of official document summaries, and providing users with a more efficient and high-quality way to obtain information. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a schematic diagram of the execution flow of the method for generating key summaries of official documents based on multimodal feature extraction provided by an embodiment of the present invention.

[0009] Figure 2 Schematic diagram of exemplary hardware and software components of a system for generating key summaries of official documents based on multimodal feature extraction provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0010] The present invention will be described in detail below with reference to the accompanying drawings. Figure 1 This is a flow chart of a method for generating a key summary of an official document based on multimodal feature extraction provided by an embodiment of the present invention. The method for generating a key summary of an official document based on multimodal feature extraction is introduced in detail below.

[0011] Step S110: Acquire a multimodal data set of the document to be processed, wherein the multimodal data set includes text modal data, image modal data, and table modal data.

[0012] In this embodiment, the documents to be processed may exist in different storage media and file systems and be presented in multiple formats, most commonly PDF, DOCX, OFD, etc. For documents in different formats, corresponding technical means are needed to obtain the multimodal data sets therein.

[0013] Specifically, if the official document is in PDF format, data can be extracted with the help of a PDF parsing library. Taking Python as an example, the PyPDF2 library can read the text content of PDF files. The specific implementation process is to read the PDF file page by page, extract the text information of each page, and form text modal data. For image modal data, since the images in PDF files exist in the form of specific objects, some dedicated tools or extension libraries, such as pdf2image, can be used to convert the image objects in the PDF into common image formats, such as JPEG or PNG. For table modal data, some advanced PDF parsing tools, such as Tabula-py, can recognize the structure of PDF tables and extract the data in the table row by row and column by column by analyzing the row and column information of the table.

[0014] For example, consider a PDF document about a large-scale project plan, which includes a detailed text description of the project, a visual chart of the project's progress, and a table of project resource allocation. Using the PyPDF2 library, you can retrieve textual information about project goals and implementation steps. Using pdf2image, you can extract images like bar charts and pie charts of the project's progress. And with Tabula-py, you can extract specific data from the project's resource allocation table, including the allocation of human and material resources.

[0015] If the document is in DOCX format, the python-docx library is an excellent choice. This library can directly read the text content of DOCX files. It can read each paragraph according to the document's paragraph structure, collecting the text information of each paragraph to form text modal data. For image modal data, images in DOCX files are often present as embedded objects. The python-docx library can recognize these objects, extract the images, and save them in common image formats. For table modal data, the python-docx library can recognize the row and column structure of the table and extract the data by traversing each row and column of the table.

[0016] Assuming that this project planning document exists in DOCX format, the python-docx library can be used to obtain the text description of the project background, strategic planning, etc. in the document; extract images such as project architecture diagrams and process diagrams from the document; and extract data such as project budget tables and time schedule tables from the document, thereby forming a complete multimodal data set.

[0017] If the document is in OFD format, the text content in the OFD file can be read through a permission-verified API interface, and this text information can be combined to form text modal data. For image modal data, images in the OFD file can be extracted through a permission-verified API interface. For table modal data, the table can also be identified through a permission-verified API interface and the data in the table can be extracted row by row and column by column.

[0018] When acquiring multimodal data sets, it is necessary to ensure the integrity and accuracy of the data. For some official documents that may be encrypted or have permission restrictions, decryption or permission verification operations must be performed first. For example, if the document is password-protected, the correct password must be obtained to decrypt it before data extraction can continue. In addition, for data storage, an appropriate storage method must be adopted. Text data can be stored as plain text files to facilitate subsequent text processing; image data can be stored as common image format files to facilitate image analysis; and tabular data can be stored as CSV files or Excel files to facilitate data processing and analysis.

[0019] Step S120: performing hierarchical semantic parsing on the text modal data to generate a text semantic feature set of the document to be processed.

[0020] In this embodiment, the hierarchical semantic parsing process is to deeply mine the semantic information in the text modality data. The process includes multiple sub-steps, which are described in detail below: Step S121: performing paragraph division processing on the text modal data to obtain a plurality of independent text paragraph units.

[0021] Text modality data is typically a continuous stream of text. To facilitate subsequent analysis and processing, it needs to be divided into multiple independent text paragraphs. Paragraph segmentation can be based on a variety of rules, including line breaks, punctuation marks, and specific paragraph separators.

[0022] In practice, you can write a paragraph division function. This function accepts text modal data as input, iterates through the characters in the text, and then determines the paragraph division rules based on the preset paragraph division rules. For example, when a line break is detected and the text before and after the line break meets the predefined semantic conditions (such as different topics before and after the text), it is divided into different paragraphs. For special paragraph separators such as "---" and "###", the text is also divided at these points when these symbols are detected.

[0023] For example, a document about a large-scale project plan might contain different topic paragraphs, such as a project overview, project objectives, and a project implementation plan. By detecting line breaks and specific paragraph-starting keywords (such as "project overview" and "project objectives"), the text can be divided into multiple independent paragraph units. Each paragraph unit has relatively independent semantic content, facilitating subsequent in-depth analysis.

[0024] Step S122: calling a pre-trained semantic parsing model to perform context dependency analysis on each of the independent text paragraph units, and extracting the core semantic vector and modifier association features of the independent text paragraph unit.

[0025] Pre-trained semantic parsing models can be language models based on deep learning, such as the BERT (Bidirectional Encoder Representations from Transformers) or GPT (Generative Pretrained Transformer) series of models. These models are pre-trained on large-scale text data and can learn the semantic and grammatical information of the language.

[0026] When processing each individual text paragraph, it is input into a pre-trained semantic parsing model. The semantic parsing model encodes each word in the paragraph, converting it into a high-dimensional vector representation. Specifically, the semantic parsing model first performs word segmentation on the input paragraph, breaking it down into individual words or subwords. These word segments are then processed by a multi-layer neural network within the semantic parsing model, generating a corresponding vector for each word segmentation.

[0027] By aggregating and processing these word vectors, the semantic parsing model can extract the core semantic vector of the paragraph. A common aggregation method is average pooling, which averages the vectors of all words in a paragraph to obtain a vector representing the paragraph. Another method is max pooling, which takes the maximum value along each dimension to obtain a new vector. More complex methods can also be used, such as pooling based on an attention mechanism, which assigns different weights to words based on their importance in the paragraph and then performs a weighted average.

[0028] At the same time, the semantic parsing model also performs contextual dependency analysis to identify the associations between modifiers and the modified objects in a paragraph. This is achieved through the semantic parsing model's attention mechanism. The attention mechanism focuses on the associations between different words in a paragraph, identifies modifiers and modified objects, and represents this association as a feature vector. For example, in the sentence "The key task of this phase is to complete the preliminary design of the project," "this phase's" modifies "key task," and "project's" modifies "preliminary design." The semantic parsing model identifies these modification relationships and extracts the corresponding modifier association features.

[0029] Step S123: performing topic clustering processing on the core semantic vector to generate topic distribution weights of the independent text paragraph units.

[0030] The purpose of topic clustering is to group together core semantic vectors with similar themes, thereby determining the topic distribution weight of each independent text paragraph unit. Clustering algorithms such as K-Means or DBSCAN (Density-Based Spatial Clustering of Applications with Noise) can be used.

[0031] The K-Means algorithm is a distance-based clustering algorithm that iteratively divides data points into K clusters, minimizing the sum of the distances from each cluster's center to the data points within each cluster. When clustering core semantic vectors, the number of clusters, K, must first be determined. An appropriate K value can be selected using the elbow method or the silhouette coefficient method.

[0032] The elbow method is implemented by calculating the sum of squared errors (SSE) within clusters for different K values. SSE is the sum of the squares of the distances from each data point to the center of its cluster. As K increases, SSE decreases, but the rate of decrease slows after reaching a certain value. The appropriate K value is chosen as the point at which the SSE decreases more slowly.

[0033] The silhouette coefficient method selects an appropriate K value by calculating the silhouette coefficient for each data point. The silhouette coefficient measures how closely a data point fits within its cluster and how closely it separates from other clusters. For each data point, the average distance a to other data points in the same cluster and the average distance b to the nearest data point in the other cluster are calculated. Silhouette coefficient s = (b a ) / max (a, b ). The silhouette coefficient ranges from -1 to 1, with values ​​closer to 1 indicating a better fit between the data point and its cluster. The K value with the largest silhouette coefficient is chosen as the appropriate number of clusters.

[0034] Assuming the K-Means algorithm is chosen for clustering, taking the core semantic vectors of independent text paragraph units as input, the algorithm will divide these vectors into K clusters. Each cluster represents a topic, and by counting the number of times each paragraph unit belongs to each cluster, we can obtain the topic distribution weight for that paragraph unit. For example, a paragraph unit may belong to topic A 70% of the time and topic B 30% of the time across multiple iterations. In this case, its topic distribution weight is (0.7, 0.3).

[0035] Step S124: performing syntax tree traversal processing on the modifier-related features to determine the logical connective features and the modification and limitation relationships in the independent text paragraph units.

[0036] Syntax tree traversal processing is a method based on grammatical analysis, which can represent the sentences in a paragraph as a syntax tree and determine the logical connective features and modification and restriction relationships by traversing the syntax tree.

[0037] You can use natural language processing tools, such as Stanford CoreNLP or SpaCy, to generate a grammatical tree for a paragraph. These tools can perform operations such as part-of-speech tagging and syntactic analysis on the sentence and convert it into a grammatical tree.

[0038] Taking Stanford CoreNLP as an example, it first performs word segmentation on the input sentence, breaking it down into individual words. It then performs part-of-speech tagging on each word to determine its part of speech (such as noun, verb, or adjective). Next, it performs syntactic analysis and constructs a syntax tree based on the parts of speech and grammatical rules. Each node in the syntax tree represents a word or phrase, and the edges between nodes indicate the grammatical relationships between them.

[0039] After obtaining the syntax tree, the tree is traversed using either a depth-first or breadth-first traversal. During the traversal, logical connectives such as "but," "therefore," and "and" are identified, and their position and function in the sentence are recorded. Furthermore, the modifying and limiting relationships between modifiers and their modified objects are determined, such as attributive modification and adverbial modification.

[0040] For example, in the sentence "Although the market environment is complex, we still developed an effective strategy," a syntax tree traversal identifies "although...but..." as a logical conjunction, "although the market environment is complex" as a concession clause that modifies the entire sentence, and "effective" as an attributive modifier that modifies "strategy."

[0041] Step S125: constructing a semantic weight graph of the independent text paragraph unit according to the topic distribution weight, the logical connective features and the modification and limitation relationship.

[0042] A semantic weight graph is a graph structure that represents the relationships and importance of semantic elements within an independent text paragraph. Nodes in the graph represent semantic elements, such as core semantic vectors and modifier association features. Edges represent the relationships between them, and edge weights indicate the strength of the relationships.

[0043] First, we consider topic distribution weights, logical connective features, and modifier-limiter relationships as node attributes. For topic distribution weights, each node can be assigned a weight based on the topic to which it belongs. For logical connective features, edges between related nodes are assigned different weights based on the type and function of the logical connective. For example, "but" represents a transitional relationship, and its corresponding edge weight can be set to a larger value to indicate the importance of this relationship. For modifier-limiter relationships, edges between related nodes are assigned corresponding weights based on the closeness between the modifier and the modified object.

[0044] Then, a graph is constructed based on these attributes and relationships. Graph theory algorithms, such as the Dijkstra algorithm or the Floyd-Warshall algorithm, can be used to calculate the shortest paths and connectivity between nodes. By analyzing the semantic weight graph, the importance and interrelationships of the various semantic elements within a paragraph unit can be determined, providing a basis for subsequent weighted aggregation of semantic vectors.

[0045] Step S126: performing dynamic weighted aggregation on the core semantic vectors based on the semantic weight map to generate an enhanced semantic vector for the independent text paragraph unit.

[0046] After obtaining the semantic weight graph, the core semantic vectors are dynamically weighted and aggregated using the weight information of its nodes and edges. Specifically, for each core semantic vector, its weight in the aggregation process is determined based on its node weight in the semantic weight graph and the edge weights with other nodes.

[0047] For example, if a core semantic vector corresponds to a node with a high weight in the semantic weight graph and has large edge weights with other important nodes, then this core semantic vector will be given a higher weight during the aggregation process. We can use a weighted summation approach for aggregation: multiply each core semantic vector by its corresponding weight, then add all the weighted vectors together to obtain a new vector, the enhanced semantic vector.

[0048] Through dynamic weighted aggregation, we can highlight the important semantic information in the paragraph unit, improve the expressive power of the semantic vector, and provide more accurate semantic representation for subsequent summary generation.

[0049] Step S127: merging the enhanced semantic vectors of the independent text paragraph units to obtain a text semantic feature set of the document to be processed.

[0050] After obtaining the enhanced semantic vectors of each independent text paragraph unit, they are merged to form a text semantic feature set of the document to be processed. The merging method can be a simple splicing, that is, connecting the enhanced semantic vectors in the order of the paragraphs.

[0051] For example, suppose an official document has three paragraph units, with their enhanced semantic vectors being vectors A, B, and C. Then, the concatenated vector sequence (vector A, vector B, vector C) is the document's text semantic feature set. This text semantic feature set encompasses the document's overall semantic information.

[0052] Step S130: performing visual element extraction processing on the image modality data to generate an image feature set of the document to be processed.

[0053] Image modality data usually contains rich visual information. In order to make better use of this information, visual element extraction processing is required. The specific steps are as follows: Step S131: performing region segmentation processing on the image modality data to divide the image modality data into a table region, an illustration region, and an annotation text region.

[0054] Region segmentation can be performed using a variety of methods, such as threshold-based segmentation, edge detection-based segmentation, or deep learning-based segmentation.

[0055] Threshold-based segmentation methods use a threshold value based on the image's grayscale or color values ​​to classify the pixels into two categories: pixels above the threshold and pixels below the threshold. By adjusting the threshold, different image regions can be segmented. For example, for an image containing a table, an appropriate threshold can be set based on the grayscale difference between the table lines and the background to segment the table area.

[0056] Edge detection-based segmentation methods divide regions by detecting edges in an image. Common edge detection operators include the Sobel operator and the Canny operator. These operators detect edge pixels in an image and then connect these edge pixels to form closed regions, thereby achieving region segmentation. For example, for an illustration, edge detection can be used to find the illustration's boundaries and separate it from the background.

[0057] Deep learning-based segmentation methods use pretrained image segmentation models, such as U-Net and Mask R-CNN. These models are trained on large-scale image datasets and can learn the features and boundaries of different objects in an image. When an image is input to these image segmentation models, they output segmentation results for each region in the image.

[0058] Taking an image of an official document containing a project progress chart and related illustrations as an example, a deep learning-based segmentation method was used to accurately segment the table area, illustration area, and annotated text area within the image. For the table area, the segmentation results clearly show the table boundaries and cell locations; for the illustration area, the illustration is separated from the background; and for the annotated text area, the text portion is extracted from the image.

[0059] Step S132: calling a pre-trained visual feature extraction model to perform row and column structure analysis on the table area, and extracting the cell layout features and header association relationships of the table area.

[0060] Pre-trained visual feature extraction models can be based on convolutional neural networks (CNNs), such as ResNet and VGG. These models are pre-trained on large-scale image datasets and can learn the visual features of images.

[0061] The segmented table area image is fed into a pre-trained visual feature extraction model, which first extracts features from the image to generate a series of feature maps. The feature maps are then analyzed and processed to parse the row and column structure of the table.

[0062] Methods such as the Hough transform can be used to detect table lines and determine the positions of table rows and columns. Cell boundaries and content can be identified by analyzing features such as grayscale and color values. Furthermore, feature analysis of the table header area can be used to determine the relationship between the header and the table content.

[0063] For example, in a project resource allocation table, a visual feature extraction model can accurately identify the rows and columns of the table, determining the location and content of each cell. It can also identify field names in the table header, such as "Resource Type," "Quantity," and "Assignment Department," and establish a relationship between the header and the table data.

[0064] Step S133: performing object detection processing on the illustration area to identify the categories of graphic objects in the illustration area and the spatial position features between objects.

[0065] Object detection processing can use deep learning-based object detection models such as Faster R-CNN, YOLO (You Only Look Once), etc. These models can detect different objects in images and determine their categories and locations.

[0066] The inset image is fed into the object detection model. It first extracts features from the image and then, through a series of classification and regression operations, identifies the object categories and determines their bounding boxes. Furthermore, by analyzing the position and size of the bounding boxes, it determines spatial features between objects, such as their distance and relative position.

[0067] For example, for the illustration area of ​​a project architecture diagram, the object detection model can identify different graphic objects in the diagram, such as rectangles representing departments and circles representing projects, and determine the spatial position relationships between them, such as which department is associated with which project, and their distance and relative positions.

[0068] Step S134: performing optical character recognition processing on the annotated text area to convert the annotated text area into structured text data.

[0069] Optical Character Recognition (OCR) processing is the process of converting text in an image into editable text data. The specific steps are as follows: Step S1341: performing binarization processing on the annotated text area to enhance the contrast between the characters and the background of the annotated text area.

[0070] Binarization is the process of converting the pixel values ​​in an image into only two values ​​(usually 0 and 255) to enhance the contrast between the characters and the background. Binarization can be performed using global thresholding or adaptive thresholding.

[0071] The global thresholding method sets a fixed threshold, setting pixels with values ​​greater than the threshold to 255 and pixels with values ​​less than the threshold to 0. This method is suitable for images with uniform lighting. The adaptive thresholding method dynamically determines the threshold based on the characteristics of the local area of ​​the image and is more effective for images with uneven lighting. Through binarization, the characters in the annotated text area become clearer and the background more uniform, providing a better foundation for subsequent character segmentation and recognition.

[0072] Step S1342: calling a pre-trained character segmentation model to perform line segmentation and character segmentation processing on the binarized annotated text area to obtain a single character image block.

[0073] Pre-trained character segmentation models can be based on deep learning architectures, such as convolutional neural networks combined with recurrent neural networks (CRNNs). By learning from large amounts of text image data, these models can accurately identify line and character boundaries within text. The binarized image of the annotated text region is fed into the character segmentation model. The model first performs a line segmentation operation, analyzing the vertical projection distribution of pixels in the image to identify regions with low pixel values. These regions often correspond to the blank spaces between text lines, thereby determining the boundaries of the text lines.

[0074] After completing line segmentation, the character segmentation model performs character segmentation on each line of text. It analyzes the horizontal pixel projection of each line of text, combines the character's shape and structural features, identifies the start and end positions of each character, and segments the text line into individual character image blocks. During the segmentation process, the character segmentation model also handles cases where characters are stuck together. By analyzing the character's context and features, it attempts to separate the stuck characters to ensure accurate individual character image blocks.

[0075] Step S1343: performing multi-scale feature extraction processing on the single character image block to generate local texture features and global shape features of the character image block.

[0076] Multi-scale feature extraction aims to capture the feature information of character image blocks from different scales and angles, including local texture features and global shape features. Different feature extraction methods can be used to achieve this goal.

[0077] The Local Binary Pattern (LBP) algorithm can be used to extract local texture features. This algorithm compares the grayscale values ​​of a central pixel with those of its neighboring pixels and encodes the result as a binary number, thereby describing local texture information. Applying the LBP algorithm to each individual character image block with varying neighborhood radii and sampling points yields local texture features at multiple scales. These features reflect the internal texture details of the character, such as stroke thickness and texture density.

[0078] Principal component analysis (PCA) can be used to extract global shape features. First, individual character image patches are converted into one-dimensional vectors, and the covariance matrix of these vectors is calculated. Eigenvalue decomposition of the covariance matrix is ​​performed to obtain eigenvectors and eigenvalues. Eigenvectors with larger eigenvalues ​​are selected as principal components, and the character image patches are projected onto these principal components to obtain global shape features. Global shape features reflect the overall shape and contour information of characters and help distinguish characters of different shapes.

[0079] Step S1344: calling a character recognition model to perform character classification processing based on the local texture features and the global shape features to determine the character category corresponding to the character image block.

[0080] The character recognition model can be a classification model based on deep learning, such as a convolutional neural network (CNN). This model is trained on a large amount of character image data to learn the characteristic patterns and classification rules of different characters.

[0081] The local texture features and global shape features of a single character image patch are concatenated to form a comprehensive feature vector. This feature vector is then fed into the character recognition model, which processes and analyzes the features through internal convolutional, pooling, and fully connected layers. Convolutional layers extract local patterns of features, pooling layers reduce the dimensionality of features, and fully connected layers map features to different character categories.

[0082] The character recognition model outputs a probability score for each character category and selects the category with the highest probability score as the character category corresponding to the character image patch. During training, a cross-entropy loss function is used to measure the difference between the model's predictions and the true labels. Backpropagation is then used to update the model's parameters to improve classification accuracy.

[0083] Step S1345: performing context correction processing on the recognized character category, and correcting the misrecognized characters based on the semantic relevance of adjacent characters.

[0084] Contextual correction uses the semantic relationships between characters to correct misrecognized characters. In text, adjacent characters often have certain semantic and logical relationships, such as forming words, phrases, or sentences. By analyzing the contextual information of adjacent characters, we can determine whether the recognition result of a particular character is reasonable and make corrections accordingly.

[0085] A language model can be built by learning from a large amount of text data to understand the probability distribution of different character combinations. Recognized character sequences are fed into the language model to calculate a probability score for each character sequence. If the recognition result for a particular character results in a low probability score for the entire sequence, the character is considered likely misrecognized.

[0086] Based on information about neighboring characters and knowledge from the language model, the system attempts to replace the character with other possible characters and recalculates the probability score of the character sequence. The character sequence with the highest probability score is selected as the corrected result. For example, if the recognized character sequence is "wrok," the language model suggests that "work" has a higher probability score, so "k" is corrected to "r."

[0087] Step S1346: Combining the corrected characters into a complete sentence according to the segmentation order to generate structured text data of the annotated text area.

[0088] After character recognition and correction, the corrected characters are combined according to the previous line segmentation and character segmentation to form a complete sentence. First, the characters in the same line are connected sequentially to form a text line. Then, the different text lines are arranged in order from top to bottom to form a complete text paragraph.

[0089] During the combination process, the text can also be formatted based on its formatting information, such as punctuation marks and spaces, to make it more consistent with natural language expression habits. The resulting structured text data can be easily analyzed and processed later.

[0090] Step S135: performing feature encoding processing on the cell layout features, the header association relationship, the graphic object category, the spatial position features and the structured text data to generate an image feature set of the image modality data.

[0091] Feature encoding converts different types of feature information into a unified vector representation for subsequent analysis and processing. For cell layout features, an encoding method can be used to convert information such as the cell's position and size into a vector. For example, the coordinates of the cell's top-left and bottom-right corners can be normalized and then combined into a vector.

[0092] For table header associations, a graph encoding approach can be used to represent the association between the table header and its contents as a graph structure, which is then converted into a vector. By calculating the node and edge features of the graph and aggregating these features, a vector representing the table header association can be obtained.

[0093] For the graphic object category and spatial position features, the category information of the graphic object can be one-hot encoded, the spatial position features can be normalized, and then the two can be spliced ​​together to form a vector representing the graphic object.

[0094] For structured text data, word embedding techniques can be used to convert the text into vectors. For example, you can use pre-trained word vector models such as Word2Vec or GloVe to convert each word in the text into a vector, and then aggregate these word vectors into a vector representing the text through average pooling or other methods.

[0095] Finally, the vectors obtained above are concatenated to form an image feature set of image modal data, which contains the feature information of various visual elements in the image.

[0096] Step S140: performing structured parsing processing on the table modal data to generate a table feature set of the document to be processed.

[0097] Tabular modal data usually has certain structural and semantic information. Structured parsing processing aims to deeply mine this information and generate a tabular feature set. The specific steps are as follows: Step S141: performing row and column structure analysis on the table modal data to extract the cell layout features and table header association relationships of the table.

[0098] Parsing the row and column structure can be achieved by analyzing the delimiters and formatting information of the table data. First, the table row and column delimiters are determined, such as commas and tabs. Then, the table data is read row by row, and each row of data is divided into cells according to the delimiters.

[0099] By counting the number of cells in each row, the number of columns in a table can be determined. Furthermore, by analyzing the position and content of cells, cell layout features such as cell merging and cell size can be extracted. For table header associations, the first row of data is typically considered the header. By analyzing the correspondence between the header and the table content, a header association can be established.

[0100] For example, in a sales data table, the header contains fields such as "Product Name", "Sales Volume", and "Sales Amount". By analyzing the table content, we can determine the product name, sales volume, and sales amount corresponding to each data row, thereby establishing an association between the header and the table content.

[0101] Step S142: Identify the merged cell area in the table and establish a hierarchical relationship map between the cells.

[0102] Identifying merged cell areas can be achieved by analyzing the coordinate information and content of the cells. If adjacent cells have the same content and are continuous in coordinates, these cells are considered merged cells.

[0103] After identifying the merged cell area, a hierarchical relationship map between cells is established. This map can be represented as a graph structure, where nodes represent cells and edges represent the hierarchical relationship between cells. For a merged cell, it is treated as a parent node and the contained child cells as child nodes, establishing a parent-child relationship.

[0104] The hierarchical relationship diagram can clearly represent the nesting and association relationships between cells in the table, providing important information for subsequent table analysis and processing.

[0105] Step S143: Perform type recognition processing on the table data to distinguish between numeric cells and text cells.

[0106] Type identification can be achieved by analyzing the cell contents. For numeric cells, their contents typically only contain numbers and possibly symbols such as decimal points and positive and negative signs. Regular expressions can be used to match these patterns to determine whether the cell is a numeric type.

[0107] For text cells, their contents contain non-numeric characters such as letters, Chinese characters, and punctuation marks. By checking whether the cell content contains these non-numeric characters, you can determine whether the cell is a text type.

[0108] The type recognition process can also handle special cases, such as cells containing mixed numbers and text. Based on specific business needs, these cells can be classified as numeric or text, or further processed.

[0109] Step S144: extract the field names of the header row and the numerical distribution characteristics of the data rows, and construct a semantic structure tree of the table.

[0110] To extract the field names from the table header row, read the first row of data in the table and use the contents of each cell as the field name. These field names reflect the semantic information of the table data, such as "Product Name," "Sales Volume," and "Sales Amount."

[0111] For the numerical distribution characteristics of data rows, you can perform statistical analysis on the data in numerical cells and extract some characteristic indicators, such as maximum value, minimum value, average value, standard deviation, etc. These characteristic indicators reflect the distribution and statistical characteristics of the data.

[0112] Based on the field names in the header row and the numerical distribution characteristics of the data rows, a semantic structure tree for the table is constructed. The root node of this tree represents the entire table, and each child node represents a field. The attributes of the child node contain the numerical distribution characteristics of the field. This semantic structure tree clearly represents the semantic structure and data characteristics of the table.

[0113] Step S145: vectorize and encode the cell layout features, hierarchical relationship graph, and semantic structure tree to generate the table feature set.

[0114] Vectorized encoding converts different types of table feature information into a unified vector representation. For cell layout features, an encoding method can be used to convert information such as cell position and size into vectors. For example, the coordinates of the top-left and bottom-right corners of a cell can be normalized and then combined into a vector.

[0115] For hierarchical relationship graphs, graph embedding techniques can be used to convert the graph structure into a vector. For example, graph neural network models such as GraphSAGE or GCN can be used to extract and encode the graph structure to obtain a vector representing the hierarchical relationship graph.

[0116] For semantic structure trees, tree embedding techniques can be used to convert the tree structure into a vector. For example, the tree structure can be converted into a sequence, and then a model such as a recurrent neural network or Transformer can be used to encode the sequence to obtain a vector representing the semantic structure tree.

[0117] Finally, the vectors obtained above are concatenated to form a table feature set, which contains various structural and semantic information of the table.

[0118] Step S150: performing cross-modal alignment processing on the text semantic feature set, the image feature set, and the table feature set to determine a dynamic association feature set between text semantics, image elements, and table elements.

[0119] The purpose of cross-modal alignment is to find the relationship between text semantics, image elements, and table elements. The specific steps are as follows: Step S151: extracting the text description segment corresponding to the enhanced semantic vector in the text semantic feature set.

[0120] The enhanced semantic vectors in the text semantic feature set are derived through hierarchical semantic parsing. Each enhanced semantic vector corresponds to an independent text paragraph unit. By analyzing the correspondence between the enhanced semantic vectors and the original text, the text description segment corresponding to each enhanced semantic vector can be extracted.

[0121] During hierarchical semantic parsing, the start and end positions of each independent text paragraph unit can be recorded. Once the enhanced semantic vector is obtained, the corresponding text description segments can be extracted from the original text based on the recorded position information. These text description segments contain specific text content related to the enhanced semantic vector, providing textual basis for subsequent cross-modal alignment.

[0122] Step S152: extracting the header text features of the table area and the annotation text features of the illustration area from the image feature set.

[0123] The image feature set includes various feature information for the table and illustration areas. For the header text features in the table area, feature vectors related to the header can be extracted from the image feature set. These feature vectors, obtained through visual element extraction processing of the table area, reflect the semantic and visual characteristics of the header text.

[0124] For the annotation text features in the illustration area, feature vectors related to the annotation text can be extracted from the image feature set. These feature vectors are obtained after optical character recognition and feature encoding processing on the annotation text area, reflecting the semantic and visual characteristics of the annotation text.

[0125] Step S153: extracting header text features and numerical distribution features from the table feature set.

[0126] The table feature set contains various structural and semantic information of the table. The header text features are extracted from the table feature set. These features reflect the semantic information of the table header, such as field names.

[0127] At the same time, the numerical distribution characteristics of the table are extracted. These characteristics are obtained when the table data is structured and parsed, and reflect the statistical characteristics of the table data, such as maximum value, minimum value, average value, etc.

[0128] Step S154: performing semantic similarity matching processing on the text description segment and the table header text feature to determine a first alignment score between the text description segment and the table area.

[0129] Semantic similarity matching can use a pre-trained semantic matching model, such as Sentence-BERT. This semantic matching model converts text description segments and header text features into vector representations and then calculates the similarity score between the two vectors.

[0130] The text description segment and header text features are fed into the Sentence-BERT model to generate corresponding vectors. The cosine similarity score between the two vectors is then calculated. This score ranges from -1 to 1, with higher scores indicating greater semantic similarity.

[0131] Based on the similarity score, a first alignment score between the text description segment and the table area is determined. A threshold can be set. When the similarity score exceeds the threshold, the text description segment is considered to have a high degree of alignment with the table area and a higher score is given; otherwise, a lower score is given.

[0132] Step S155: performing keyword coverage analysis on the text description segment and the annotated text feature to determine a second alignment score between the text description segment and the illustration area.

[0133] Keyword coverage analysis is done by comparing keywords in the text description segment and the annotated text features to determine alignment. First, keywords are extracted from the text description segment and the annotated text features using algorithms such as TF-IDF or TextRank.

[0134] After extracting keywords from the text description segment and the annotated text features, the keyword overlap ratio between the two is calculated. The keyword overlap ratio refers to the ratio of the number of keywords that appear in both texts to the total number of keywords.

[0135] A second alignment score between the text description segment and the illustration region is determined based on the keyword overlap ratio. A threshold can be set such that when the keyword overlap ratio exceeds the threshold, the text description segment and the illustration region are considered to have a high degree of alignment and are given a higher score; otherwise, a lower score is given.

[0136] Step S156: performing a numerical correlation analysis process on the text description segment and the numerical distribution feature to determine a third alignment score between the text description segment and the table data.

[0137] Numerical correlation analysis can be achieved through the following steps: Step S1561: extracting quantitative description phrases and numerical entities from the text description segment.

[0138] Quantitative descriptive phrases are phrases in text that contain quantitative information such as quantity, proportion, or degree, such as "most," "a few," and "increased." Numerical entities are numbers explicitly expressed in text, such as "10%" and "200."

[0139] Natural language processing techniques, such as part-of-speech tagging and named entity recognition, can be used to identify quantitative descriptive phrases and numerical entities in text descriptions. Part-of-speech tagging determines the part of speech of each word, and by analyzing the part of speech, words that represent quantitative information can be found. Named entity recognition can identify numerical entities in text.

[0140] Step S1562: extracting the value range and statistical features of the numeric cells from the table feature set.

[0141] The table feature set contains the table's numerical distribution features, from which the numerical range and statistical features of the numeric cells are extracted, such as the maximum value, minimum value, average value, etc. This information reflects the numerical characteristics of the table data.

[0142] Step S1563: establishing a mapping relationship between the quantitative description phrase and the numerical cell, and calculating the degree of fit between the numerical range covered by the description phrase and the distribution of the table data.

[0143] For quantitative description phrases, they can be mapped to a numerical range based on their semantic meaning. For example, "most" can be mapped to a larger numerical range, and "a few" can be mapped to a smaller numerical range.

[0144] Compare the numerical intervals mapped to the quantitative description phrases with the numerical ranges of the tabular data and calculate the degree of agreement between the two. The degree of agreement can be measured by calculating the overlap ratio of the numerical intervals.

[0145] Step S1564: Perform table positioning processing on the numerical entity to determine its specific position coordinates in the table.

[0146] For numeric entities in a text description, locate them in the table data. You can find the matching cell by comparing the value of the numeric entity with the value of a numeric cell in the table and determine its position coordinates in the table.

[0147] Step S1565: constructing a numerical correlation matrix based on the fit and position coordinates to generate the third alignment score.

[0148] A numerical association matrix is ​​constructed based on the degree of fit and the position coordinates. The elements in the matrix represent the degree of association between the quantitative description phrases or numerical entities in the text description segment and the numerical cells in the table.

[0149] A comprehensive analysis of the numerical correlation matrix is ​​performed to generate a third alignment score for the text description segment and the table data. Different weights can be set based on the values ​​of the elements in the matrix, and a weighted score is calculated as the third alignment score.

[0150] Step S157: constructing a cross-modal attention weight matrix based on the first alignment score, the second alignment score, and the third alignment score.

[0151] The cross-modal attention weight matrix is ​​used to represent the strength of the association between text semantics, image features, and table features. A matrix is ​​constructed based on the first alignment score, the second alignment score, and the third alignment score.

[0152] The rows of the matrix represent text description segments, and the columns represent image elements and table elements. Each element in the matrix represents the strength of the association between the corresponding text description segment and the image element or table element. The specific construction process is as follows: First, the first, second, and third alignment scores need to be normalized to ensure that they are in the same dimension and value range to avoid the influence of different scoring scales on the construction of the weight matrix. Normalization can use the common min-max normalization method to map each score to the range of 0 to 1.

[0153] Let the first alignment score set be A, the second alignment score set be B, and the third alignment score set be C. For each score a in set A, normalize it using the formula: Normalized a = (minimum value in aA) / (maximum value in A - minimum value in A). The same method is used to normalize the scores in sets B and C.

[0154] Then, assign a weight to each alignment score. Assume the weight of the first alignment score is w1, the weight of the second alignment score is w2, and the weight of the third alignment score is w3, where w1 + w2 + w3 = 1. These weights can be adjusted based on specific business needs and data characteristics. For example, if you are more concerned about the relationship between text and table areas, you can increase the value of w1 appropriately.

[0155] For each element in the cross-modal attention weight matrix, its value is calculated as follows: the first normalized alignment score of the text description segment corresponding to the element and the image element or table element multiplied by w1, the second normalized alignment score multiplied by w2, and the third normalized alignment score multiplied by w3. This results in a complete cross-modal attention weight matrix that comprehensively reflects the strength of the association between text semantics, image elements, and table elements.

[0156] Step S158: Perform multimodal feature interaction processing on the text semantic feature set, image feature set and table feature set according to the cross-modal attention weight matrix to generate the dynamic association feature set.

[0157] The goal of multimodal feature interaction processing is to allow text semantic features, image features, and table features to interact and explore potential correlations between them. First, the text semantic feature set, image feature set, and table feature set are represented as feature matrices, where each row corresponds to a feature vector and the column corresponds to the feature dimension.

[0158] For the feature matrix T corresponding to the text semantic feature set, the feature matrix I corresponding to the image feature set, and the feature matrix G corresponding to the table feature set, feature interaction operations are performed based on the cross-modal attention weight matrix W. For each feature vector t in the text semantic feature matrix T, the cross-modal attention weight matrix W is used to find its association weight with each feature vector in the image feature matrix I and the table feature matrix G.

[0159] Specifically, the association weight between t and a particular eigenvector i in the image feature matrix I is the value of the element corresponding to t and i in the cross-modal attention weight matrix W. This association weight is multiplied by the eigenvector i to obtain a weighted eigenvector. This operation is repeated for all eigenvectors in the image feature matrix I, and the weighted eigenvectors are then concatenated. The same method is used to associate t with the table feature matrix G.

[0160] A similar approach is used for the eigenvectors in the image feature matrix I and the table feature matrix G to interact with the text semantic feature matrix T. Through this multimodal feature interaction, features from different modalities can complement and enhance each other.

[0161] When performing feature interaction, nonlinear transformations can be introduced, such as using activation functions (such as the ReLU function) to process the weighted feature vectors to increase the model's expressive power. After multimodal feature interaction, the resulting new feature vectors are combined to form a dynamically correlated feature set. This dynamically correlated feature set contains dynamic correlation information between text semantics, image elements, and table elements.

[0162] Step S160: performing multimodal fusion processing on the text semantic feature set, the image feature set and the table feature set based on the dynamic association feature set to generate target summary content of the document to be processed.

[0163] Multimodal fusion processing integrates the feature information of different modalities to generate target summary content with high quality and completeness. The specific steps are as follows: Step S161: performing importance ranking processing on the dynamic association feature set to determine key semantic units in the text semantic feature set, key image elements in the image feature set, and key table elements in the table feature set.

[0164] Importance ranking can be performed using machine learning-based ranking algorithms, such as LambdaMART. This algorithm builds a ranking model by learning the features of sample data and their corresponding importance labels. Taking a dynamically associated feature set as input, the ranking model assigns an importance score to each feature vector.

[0165] The dynamically associated feature set is sorted based on importance scores, with feature vectors with higher scores considered more important. For the text semantic feature set, the semantic units in the text description corresponding to feature vectors with high importance scores are considered key semantic units. For example, if a text description segment emphasizes the core objectives and key deliverables of a project, then the relevant semantic units in that description segment are considered key semantic units.

[0166] For image feature sets, image elements corresponding to feature vectors with high importance scores are considered key image elements. For example, in a visualization of project progress, graphical objects representing key milestones are considered key image elements. For table feature sets, table elements corresponding to feature vectors with high importance scores are considered key table elements, such as rows or columns of data in a table that reflect key project metrics.

[0167] Step S162: performing context integrity verification on the key semantic units, and removing redundant semantic units whose correlation with the key image elements and key table elements is lower than a preset threshold.

[0168] The context integrity verification process is to ensure that the key semantic units are complete and coherent in the context. First, the semantic relationship between the key semantic units is analyzed to determine whether they can form a complete semantic paragraph.

[0169] Then, the correlation between the key semantic units and the key image elements and key table elements is calculated. The correlation can be calculated based on the cross-modal attention weight matrix obtained above, by extracting the weight values ​​between the feature vectors corresponding to the key semantic units and the feature vectors corresponding to the key image elements and key table elements.

[0170] A preset threshold is set. If the correlation between a key semantic unit and key image elements and key table elements is lower than the threshold, the semantic unit is considered redundant and removed from the key semantic unit set. This allows the key semantic unit to focus more on important information related to the image and table, improving the quality of the summary content.

[0171] Step S163: performing content interpretability conversion processing on the key image elements, converting the cell layout features of the table area into text description statements, and converting the graphic object categories of the illustration area into keyword labels.

[0172] For the cell layout characteristics of the table area, analyze information such as cell position, size, merging status, and header associations. Generate a text description based on this information. For example, if there is a merged cell in the table that merges two rows and three columns and is located in the upper left corner of the table, and the table header displays the area as "Project Principal Investigator Information," a text description statement can be generated: "The area in the upper left corner of the table contains the project principal investigator information, and this area merges two rows and three columns."

[0173] For graphic object categories in the illustration area, we convert them into keyword labels. For example, if the illustration area contains a bar chart representing project progress and the graphic object category is identified as "project progress bar chart," we convert it into the keyword label "project progress bar chart." These keyword labels can concisely express the main content of the illustration and facilitate integration with text information.

[0174] Step S164: Perform statistical description conversion processing on the key table elements to convert the numerical distribution characteristics of the table into data summary statements.

[0175] The statistical description transformation process can be performed as follows: Step S1641: Identify the header row and data rows of the table area, and extract the field names of the header row and the value distribution characteristics of the data rows.

[0176] Through the previous structured parsing process, we have obtained the row and column structure of the table and the association between the headers. From this information, we can identify the header row and data rows. The content of each cell in the header row is the field name, such as "Product Sales Volume" or "Sales Amount".

[0177] The numerical distribution characteristics of the data rows include maximum value, minimum value, average value, standard deviation, etc. These characteristics are extracted when performing structured parsing on the table data.

[0178] Step S1642: Perform statistical analysis on the numerical distribution characteristics to determine maximum value, minimum value and average value characteristics.

[0179] Further analyze the numerical distribution characteristics to identify the fields and specific values ​​corresponding to the maximum, minimum, and average values. For example, for the "Product Sales" field, determine the product corresponding to its maximum value, the product corresponding to its minimum value, and the average sales volume.

[0180] Step S1643: Generate a statistical description template statement for the table area based on the field name and the maximum value, minimum value and average value features.

[0181] Generate a statistical description template statement based on the field name and statistical characteristics. For example, for the "Product Sales" field, if the maximum value is the sales of a certain product A, the minimum value is the sales of product B, and the average value is a specific value, you can generate the statement "In terms of product sales, product A has the highest sales, product B has the lowest sales, and the average sales is [specific value]."

[0182] Step S1644: performing semantic conversion processing on the hierarchical relationship graph in the table feature set, and converting the logical relationship of the merged cells into a correlation description phrase.

[0183] The hierarchical relationship map records the hierarchical and relational relationships between cells, especially when cells are merged. These relationships are converted into relational descriptive phrases. For example, if a merged cell represents summary information for multiple subprojects, a relational descriptive phrase like "This area summarizes relevant information for multiple subprojects" can be generated.

[0184] Step S1645: performing sentence structure integration processing on the statistical description template sentence and the relevance description phrase to generate a data summary sentence corresponding to the table element.

[0185] Combine the statistical description template statements and the associated description phrases to form a complete data summary statement. During this integration process, pay attention to the coherence and logic of the statements to ensure they accurately convey the core information of the table data. For example, combine the statistical description statements and associated description phrases generated previously to form "Among the multiple sub-items summarized in this region, product A has the highest sales volume, product B has the lowest sales volume, and the average sales volume is [specific value]."

[0186] Step S165: performing multimodal content enhancement processing on the key semantic unit based on the text description sentence, keyword tag and data summary sentence to generate an enhanced semantic paragraph that integrates text information, image information and table information.

[0187] Multimodal content enhancement involves integrating text descriptions, keyword tags, and data summaries with key semantic units. First, keyword tags are inserted into appropriate locations within the key semantic units to better connect the text to the image information. For example, if the key semantic unit describes a project's progress, inserting the keyword tag "project progress bar chart" allows readers to intuitively connect the text to the image.

[0188] Then, the data summary statement is merged with the key semantic unit to supplement the information about the table data in the text. For example, the table data summary statement generated earlier is inserted into the description of the project data in the key semantic unit to make the text richer and more accurate.

[0189] Finally, the text description is integrated with key semantic units to improve the text's description of information such as the table cell layout. Through these operations, an enhanced semantic paragraph is generated that integrates text, image, and table information and fully reflects the core content of the official document.

[0190] Step S166: Optimizing the coherence of the enhanced semantic paragraphs, adjusting the paragraph order and adding logical conjunctions, to generate target summary content that complies with summary specifications.

[0191] For example, the consistency optimization process can be performed as follows: Step S1661: performing semantic role labeling on the enhanced semantic paragraphs to determine the action subject features and action object features of each paragraph of the enhanced semantic paragraphs.

[0192] Semantic role labeling can be done using a pre-trained semantic role labeling model, such as the semantic role labeler in AllenNLP. This semantic role labeling model can analyze each sentence in the enhanced semantic paragraph and determine the action subject and action object in the sentence.

[0193] The action subject is the entity that performs the action, and the action object is the recipient of the action. For example, in the sentence "The company has developed a project plan," "company" is the action subject, and "project plan" is the action object. By performing semantic role labeling on each paragraph in the enhanced semantic paragraph, we obtain the action subject and action object features of each paragraph.

[0194] Step S1662: constructing an entity continuity chain between paragraphs based on the action subject feature, and determining the breakpoint position in the entity continuity chain.

[0195] The entity continuity chain refers to the continuity relationship between the action subjects of paragraphs. According to the characteristics of the action subjects of each paragraph, paragraphs with the same action subjects are connected to form an entity continuity chain.

[0196] When building an entity continuity chain, the subject of the action may suddenly change. These locations are breakpoints in the entity continuity chain. For example, if the subject of the action in the previous paragraph is "company" and the subject of the action in the next paragraph suddenly changes to "department", then there is a breakpoint between the two paragraphs.

[0197] Step S1663: inserting a transition statement at the breakpoint position, wherein the transition statement includes a summary description of the action object in the previous paragraph and an introduction description of the action subject in the next paragraph.

[0198] Inserting transition statements at breakpoints can help create a more natural flow between paragraphs. When designing transition statements, consider the action of the previous paragraph and the subject of the next paragraph. For example, if the previous paragraph describes the company developing a project plan and the next paragraph describes the departments executing the plan, the transition statement could be, "The company's project plan lays the foundation for subsequent work. Next, each department will begin executing according to the plan."

[0199] Step S1664: Perform pronoun replacement processing on repetitive action objects in adjacent paragraphs, use a preset synonym library to replace repeated nouns, and generate processed candidate paragraphs.

[0200] To avoid excessive repetition in paragraphs, replace the pronouns of repeated actions in adjacent paragraphs. For example, if the previous paragraph mentions "project plan", you can replace it with "the plan" when it is mentioned again in the next paragraph.

[0201] At the same time, we use a pre-set synonym library to replace repeated nouns. For example, if the word "project" is mentioned multiple times, we can replace it with synonyms such as "engineering" or "scheme." Through these processing, the paragraph is made more concise and clear, and the processed candidate paragraph is generated.

[0202] Step S1665: Input the candidate paragraph into a pre-trained language coherence model to extract the context embedding vector of the candidate paragraph.

[0203] The pre-trained language coherence model can be a Transformer-based model, such as BERT. The candidate paragraph is input into the language coherence model, which encodes the candidate paragraph and extracts its contextual embedding vector.

[0204] The context embedding vector can reflect the semantic information of each word in the candidate paragraph in the context. By analyzing these context embedding vectors, the coherence of the candidate paragraph can be judged.

[0205] Step S1666: Perform mutation point detection processing on the context embedding vector to determine the semantic jump degree between adjacent sentences.

[0206] The mutation point detection process can be achieved by calculating the similarity between the context embedding vectors of adjacent sentences. If the similarity is low, it means that the semantic jump between adjacent sentences is large.

[0207] Cosine similarity can be used to calculate the similarity between the context embedding vectors of adjacent sentences. When the similarity is lower than a certain threshold, it is considered that there is a semantic jump between adjacent sentences.

[0208] Step S1667: Calculate the average fluency deviation value of the candidate paragraph based on the semantic jump degree.

[0209] The average fluency deviation is a measure of the overall coherence of a candidate paragraph. The average fluency deviation is calculated by summing the semantic jumps of all adjacent sentences in a candidate paragraph and dividing the sum by the number of adjacent sentences.

[0210] Step S1668: If the average fluency deviation value exceeds a preset threshold, the sentence with the largest semantic jump is reconstructed by adding connecting adverbs or adjusting the word order.

[0211] If the average fluency deviation exceeds the preset threshold, it means that the coherence of the candidate paragraph is poor. In this case, the sentence with the largest semantic jump is found and reconstructed.

[0212] You can add connecting adverbs, such as "however," "therefore," and "and," to indicate the logical relationship between sentences. You can also adjust the word order of sentences to make them more logical and consistent with expression habits.

[0213] Step S1669: Repeat the detection and reconstruction process until the average fluency deviation value does not exceed a preset threshold, and generate the final optimized target summary content.

[0214] The process of detecting mutation points and reconstructing sentences is repeated until the average fluency deviation of the candidate paragraphs does not exceed a preset threshold. At this point, the candidate paragraph's coherence meets the requirements and is selected as the final optimized target summary. This target summary integrates key information from text, images, and tables, achieving high quality and readability, accurately reflecting the core content of the document being processed.

[0215] In the above embodiments, a pre-trained semantic parsing model like BERT is used to perform hierarchical semantic parsing on text modal data, which mainly consists of an input layer, a multi-layer Transformer encoder layer, and an output layer. The input layer processes the input text data into a format suitable for model processing. The multi-layer Transformer encoder layer is the core. Each layer has a multi-head self-attention mechanism and a feedforward neural network. The multi-head self-attention mechanism can capture the dependencies between different positions in the text. The feedforward neural network performs a nonlinear transformation on the output of the attention mechanism. The output layer processes the encoder layer output according to specific task requirements. Between each layer, the input layer outputs to the first encoder layer, and the output of each encoder layer is given to the next layer, and the last layer outputs to the output layer.

[0216] Training uses large-scale unsupervised text data, such as XX Encyclopedia, and includes tasks such as masked language modeling and next sentence prediction. The masked language model task involves randomly masking some words in the input text and asking the model to predict them. This learns semantic information by minimizing the loss between the predicted results and the actual words. The next sentence prediction task involves inputting two sentences and asking the model to determine whether they are consecutive, thereby learning the logical relationship between the sentences. Regarding training parameters, the learning rate can be set high initially and then gradually reduced. The batch size determines the amount of data input for each training session, and the number of training epochs is the number of times the entire training data is traversed. These parameters should be adjusted as needed.

[0217] In this application scenario, the input data is composed of independent text paragraphs, which require word segmentation and special markup to conform to the input format requirements. The output is a vector representation of each word. After aggregation, the core semantic vector of the paragraph is obtained, which is used for subsequent topic clustering and semantic weight graph construction.

[0218] Pre-trained visual feature extraction models, such as ResNet, are used to process image modality data. They consist of convolutional layers, residual blocks, and fully connected layers. Convolutional layers extract local image features by performing convolution operations using different convolution kernels. Residual blocks are a core innovation, using skip connections to address the vanishing gradient problem in deep neural networks. Fully connected layers integrate the features extracted by the convolutional layers and residual blocks to produce the final output. Each layer is connected through a stacked structure. The input image first passes through a convolutional layer, then through multiple residual blocks in sequence. The output of each residual block is added to the input and passed to the next residual block before being output through a fully connected layer.

[0219] Training is done using large-scale image datasets such as ImageNet, using stochastic gradient descent or its variant optimizers. Image data is fed into the model, and predictions are obtained through forward propagation. The loss between the predictions and the true labels is calculated, and the model parameters are then updated through backpropagation.

[0220] In this scenario application, the input is the segmented image area, such as the table area, illustration area, etc., and the output is the feature map of the image. These feature maps contain the visual feature information of the image and are used for subsequent row and column structure analysis, object detection and other processing.

[0221] The pretrained character segmentation model uses a convolutional neural network combined with a recurrent neural network (such as a CRNN) to perform line and character segmentation on annotated text regions. It consists of convolutional layers, recurrent layers, and transcription layers. The convolutional layers extract image features, the recurrent layers process sequence information, and the transcription layers convert feature sequences into character sequences. Each layer is connected sequentially, with the input image first passing through the convolutional layers, then the recurrent layers, and finally the transcription layer outputting the character segmentation results.

[0222] Training uses a large amount of text image data, annotating each image with character information. During training, the image and the corresponding character label are input. The model uses forward propagation to obtain a predicted character sequence, calculates the loss between the predicted sequence and the true label, and then uses backpropagation to update the parameters.

[0223] In this scenario, the input is a binarized labeled text area image, and the output is a single character image block for subsequent character feature extraction and recognition.

[0224] The character recognition model is based on a convolutional neural network, consisting of convolutional, pooling, and fully connected layers. The convolutional layers extract local features from character images, the pooling layers reduce feature dimensionality, and the fully connected layers map features to different character categories. Each layer is connected sequentially. After the input character image is processed by the convolutional and pooling layers, the fully connected layers output the character category prediction result.

[0225] Training uses a large amount of character image data, annotating each image with the character category. During training, the character image and the corresponding category label are input, and the predicted category is obtained through forward propagation. The loss between the predicted category and the true label is calculated, and then backpropagation is used to update the parameters.

[0226] In this scenario application, the input is a comprehensive feature vector obtained by concatenating the local texture features and global shape features of a single character image block, and the output is the character category for subsequent text correction and combination.

[0227] A pre-trained language model is used to contextually correct the recognized character categories. It learns from a large amount of text data and understands the probability distribution of different character combinations. The model structure consists of an input layer, a hidden layer, and an output layer. The input layer receives the character sequence, the hidden layer processes the sequence information, and the output layer outputs the probability score for each character sequence.

[0228] Large-scale text data is used for training, and model parameters are adjusted through methods such as maximum likelihood estimation so that the model can accurately predict the probability of character sequences.

[0229] In this scenario application, the input is the recognized character sequence, and the output is the corrected character sequence, which improves the accuracy of character recognition.

[0230] Pre-trained semantic matching models, such as Sentence-BERT, are used to perform semantic similarity matching between text description segments and header text features. Based on the BERT model, it consists of an input layer, an encoder layer, and an output layer. The input layer processes the input text, the encoder layer extracts text features, and the output layer calculates the text similarity score.

[0231] Training uses a large amount of text pair data, annotating each pair with a similarity label. During training, the text pairs are input, and a similarity score is obtained through forward propagation. The loss between the score and the true label is calculated, and the parameters are updated through backpropagation.

[0232] In this scenario application, the input is the text description segment and the table header text features, and the output is the semantic similarity score, which is used to determine the alignment score of the text description segment and the table area.

[0233] The LambdaMART-based ranking algorithm is used to prioritize the importance of dynamically correlated feature sets. It consists of an ensemble of decision trees, each of which learns the relationship between sample data features and importance labels. During training, multiple decision trees are iteratively trained using sample data and corresponding importance labels, with each tree fitting the residuals of the previous tree.

[0234] In this scenario application, the input is a set of dynamically associated features, and the output is the importance score of each feature vector, which is used to determine key semantic units, key image elements, and key table elements.

[0235] Pre-trained language coherence models are based on the Transformer architecture, such as BERT. They consist of an input layer, an encoder layer, and an output layer. The input layer processes the input paragraph, the encoder layer extracts the paragraph context embedding vector, and the output layer can perform further processing as needed.

[0236] The training uses large-scale text data and unsupervised learning to allow the model to learn language coherence.

[0237] In this scenario, the input is the processed candidate paragraph, and the output is the context embedding vector of the paragraph, which is used to judge the paragraph coherence and calculate the semantic jump degree and average fluency deviation value.

[0238] Finally, regarding data collection, the multimodal data sets of official documents must adhere to legal and regulatory compliance principles and must be collected and analyzed with user authorization. If privacy-sensitive data is involved, encryption technology must be used to encrypt and store the data, and secure transmission protocols must be used during transmission to prevent data leaks. Strict permission management must be implemented for access to multimodal data, ensuring that only authorized personnel have access to the relevant data.

[0239] Figure 2 A schematic diagram illustrates exemplary hardware and software components of a system 100 for generating key summaries of official documents based on multimodal feature extraction, which can implement the concepts of the present invention, according to some embodiments of the present invention. For example, a processor 120 can be used in the system 100 for generating key summaries of official documents based on multimodal feature extraction, and can be used to perform the functions of the present invention.

[0240] The document key summary generation system 100 based on multimodal feature extraction can be a general-purpose server or a special-purpose server, both of which can be used to implement the document key summary generation method based on multimodal feature extraction of the present invention. Although only one server is shown in the present invention, for convenience, the functions described in the present invention can be implemented in a distributed manner on multiple similar platforms to balance the processing load.

[0241] For example, the document key summary generation system 100 based on multimodal feature extraction may include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and storage media 140 in various forms, such as a disk, ROM, or RAM, or any combination thereof. Exemplarily, the document key summary generation system 100 based on multimodal feature extraction may also include program instructions stored in ROM, RAM, or other types of non-transitory storage media, or any combination thereof. The method of the present invention may be implemented based on these program instructions. The document key summary generation system 100 based on multimodal feature extraction also includes an I / O interface 150 between the computer and other input and output devices.

[0242] For ease of explanation, only one processor is described in the document key summary generation system 100 based on multimodal feature extraction. However, it should be noted that the document key summary generation system 100 based on multimodal feature extraction in the present invention can also include multiple processors, so the steps performed by one processor described in the present invention can also be performed jointly or individually by multiple processors. For example, if the processor of the document key summary generation system 100 based on multimodal feature extraction executes step A and step B, it should be understood that step A and step B can also be executed jointly by two different processors or individually in one processor. For example, the first processor executes step A, the second processor executes step B, or the first processor and the second processor execute steps A and B together.

[0243] In addition, an embodiment of the present invention further provides a readable storage medium, in which computer-executable instructions are preset. When a processor executes the computer-executable instructions, the above-mentioned method for generating a key summary of an official document based on multimodal feature extraction is implemented.

[0244] It should be noted that in order to simplify the description of the present invention and thus help understand one or more embodiments of the invention, in the foregoing description of the embodiments of the present invention, multiple features are sometimes combined into one embodiment, figure or description thereof.

Claims

1. A method for generating key summaries of official documents based on multimodal feature extraction, characterized in that: The method comprises: Acquire a multimodal data set of the document to be processed, wherein the multimodal data set includes text modal data, image modal data, and table modal data; Performing hierarchical semantic parsing on the text modal data to generate a text semantic feature set of the document to be processed; Performing visual element extraction processing on the image modality data to generate an image feature set of the document to be processed; Performing structured parsing on the form modal data to generate a form feature set of the document to be processed; Performing cross-modal alignment processing on the text semantic feature set, the image feature set, and the table feature set to determine a dynamic association feature set between text semantics, image elements, and table elements; Based on the dynamic association feature set, multimodal fusion processing is performed on the text semantic feature set, the image feature set and the table feature set to generate the target summary content of the document to be processed.

2. The method for generating key summaries of official documents based on multimodal feature extraction according to claim 1, characterized in that: The step of performing hierarchical semantic parsing on the text modal data to generate a text semantic feature set of the document to be processed includes: Performing paragraph division processing on the text modal data to obtain a plurality of independent text paragraph units; Calling a pre-trained semantic parsing model to perform context dependency analysis on each of the independent text paragraph units, and extracting the core semantic vector and modifier association features of the independent text paragraph unit; Performing topic clustering processing on the core semantic vector to generate topic distribution weights of the independent text paragraph units; Performing syntax tree traversal processing on the modifier-related features to determine the logical connective features and the modification and limitation relationships in the independent text paragraph units; Constructing a semantic weight graph of the independent text paragraph unit according to the topic distribution weight, the logical connective features and the modification and limitation relationship; Performing dynamic weighted aggregation on the core semantic vector based on the semantic weight map to generate an enhanced semantic vector for the independent text paragraph unit; The enhanced semantic vectors of the independent text paragraph units are merged to obtain a text semantic feature set of the document to be processed.

3. The method for generating key summaries of official documents based on multimodal feature extraction according to claim 1 is characterized in that: The performing visual element extraction processing on the image modality data to generate the image feature set of the document to be processed includes: Performing region segmentation processing on the image modality data to divide the image modality data into a table region, an illustration region, and an annotation text region; Calling a pre-trained visual feature extraction model to perform row and column structure analysis on the table area, and extracting cell layout features and table header association relationships in the table area; performing object detection processing on the illustration area to identify the categories of graphic objects in the illustration area and the spatial position features between the objects; Performing optical character recognition processing on the annotated text area to convert the annotated text area into structured text data; Feature encoding processing is performed on the cell layout features, the header association relationship, the graphic object category, the spatial position features and the structured text data to generate an image feature set of the image modality data.

4. The method for generating key summaries of official documents based on multimodal feature extraction according to claim 3 is characterized in that: The performing optical character recognition processing on the annotated text area to convert the annotated text area into structured text data includes: performing binarization processing on the annotated text area to enhance the contrast between the characters and the background of the annotated text area; Calling a pre-trained character segmentation model to perform line segmentation and character segmentation processing on the binarized annotated text area to obtain a single character image block; Performing multi-scale feature extraction processing on the single character image block to generate local texture features and global shape features of the character image block; Calling a character recognition model to perform character classification processing based on the local texture features and the global shape features to determine the character category corresponding to the character image block; Performing contextual correction processing on the recognized character categories to correct misrecognized characters based on semantic relevance of adjacent characters; The corrected characters are combined into a complete sentence according to the segmentation order to generate structured text data of the annotated text area.

5. The method for generating key summaries of official documents based on multimodal feature extraction according to claim 1 is characterized in that: The performing structured parsing processing on the form modal data to generate the form feature set of the document to be processed includes: Performing row and column structure analysis on the table modal data to extract the cell layout features and table header association relationships of the table; Identify merged cell areas in a table and establish a hierarchical relationship map between cells; Perform type recognition on table data to distinguish between numeric cells and text cells; Extract the field names of the header row and the numerical distribution characteristics of the data rows to build a semantic structure tree of the table; The cell layout features, hierarchical relationship graph and semantic structure tree are vectorized and encoded to generate the table feature set.

6. The method for generating key summaries of official documents based on multimodal feature extraction according to any one of claims 1 to 5, characterized in that: The cross-modal alignment processing of the text semantic feature set, the image feature set, and the table feature set to determine a dynamic association feature set between text semantics, image elements, and table elements includes: Extracting a text description segment corresponding to the enhanced semantic vector in the text semantic feature set; Extracting header text features of the table area and annotation text features of the illustration area from the image feature set; Extracting header text features and numerical distribution features from the table feature set; Performing semantic similarity matching processing on the text description segment and the table header text feature to determine a first alignment score between the text description segment and the table area; Performing keyword coverage analysis on the text description segment and the annotated text feature to determine a second alignment score between the text description segment and the illustration area; performing a numerical correlation analysis on the text description segment and the numerical distribution feature to determine a third alignment score between the text description segment and the table data; constructing a cross-modal attention weight matrix based on the first alignment score, the second alignment score, and the third alignment score; The text semantic feature set, image feature set and table feature set are subjected to multimodal feature interaction processing according to the cross-modal attention weight matrix to generate the dynamic association feature set.

7. The method for generating key summaries of official documents based on multimodal feature extraction according to claim 6, characterized in that: The performing a numerical correlation analysis on the text description segment and the numerical distribution feature to determine a third alignment score between the text description segment and the table data includes: Extracting quantitative description phrases and numerical entities from the text description segment; Extracting the value range and statistical features of the numeric cells from the table feature set; Establishing a mapping relationship between the quantitative description phrase and the numerical cell, and calculating the degree of fit between the numerical range covered by the description phrase and the distribution of the table data; Performing table positioning processing on the numerical entity to determine its specific position coordinates in the table; A numerical correlation matrix is ​​constructed based on the fit and position coordinates to generate the third alignment score.

8. The method for generating key summaries of official documents based on multimodal feature extraction according to claim 1 is characterized in that: The multimodal fusion processing of the text semantic feature set, the image feature set, and the table feature set based on the dynamic association feature set to generate the target summary content of the document to be processed includes: Performing importance sorting processing on the dynamic association feature set to determine key semantic units in the text semantic feature set, key image elements in the image feature set, and key table elements in the table feature set; Performing context integrity verification on the key semantic units, and removing redundant semantic units whose correlation with the key image elements and key table elements is lower than a preset threshold; Performing content interpretability conversion processing on the key image elements, converting the cell layout features of the table area into text description sentences, and converting the graphic object categories of the illustration area into keyword labels; Performing statistical description conversion processing on the key table elements to convert the numerical distribution characteristics of the table into data summary statements; Performing multimodal content enhancement processing on the key semantic unit based on the text description sentence, keyword tag and data summary sentence to generate an enhanced semantic paragraph that integrates text information, image information and table information; The enhanced semantic paragraphs are subjected to coherence optimization processing, the paragraph order is adjusted and logical conjunctions are added to generate target summary content that meets summary specifications.

9. The method for generating key summaries of official documents based on multimodal feature extraction according to claim 8, characterized in that: The statistical description conversion process is performed on the key table elements to convert the numerical distribution characteristics of the table into a data summary statement, including: Identifying the header row and data rows of the table area, and extracting the field names of the header row and the numerical distribution characteristics of the data rows; Performing statistical analysis on the numerical distribution characteristics to determine maximum, minimum and average value characteristics; Generate a statistical description template statement for the table area based on the field name and the maximum value, minimum value and average value features; Performing semantic conversion processing on the hierarchical relationship graph in the table feature set, converting the logical relationship of the merged cells into a correlation description phrase; The statistical description template sentence and the association description phrase are subjected to sentence structure integration processing to generate a data summary sentence corresponding to the table element.

10. A document key summary generation system based on multimodal feature extraction, characterized in that: It includes a processor and a memory, the memory is connected to the processor, the memory is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the memory to implement the method for generating key summaries of official documents based on multimodal feature extraction as described in any one of claims 1 to 9.

Images