Document processing method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of computers, and particularly discloses a document processing method and device, computer equipment and a storage medium. The document processing method comprises the following steps: acquiring a multi-modal document; decomposing the multi-mode document into a plurality of document fragment blocks; the document fragment blocks are associated with first position information of the document fragment blocks in the multi-modal document; splicing the plurality of document fragment blocks according to the reading sequence of the multi-modal document to form a target text; dividing the target text into a plurality of logic blocks based on the semantic structure of the target text; wherein the logic block comprises the content of the target text; establishing an association relationship between the logic block and the document fragment block, and aggregating the first position information by using the association relationship to determine second position information of the logic block corresponding to the logic block in the multi-modal document; and the content of the logic block and the second position information are stored in an associated manner, so that the technical problems of fragmented information overload and overall document positioning failure in document processing are solved, and improvement of semantic integrity and positioning accuracy is facilitated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and in particular to a document processing method and device, computer equipment, and a storage medium. BACKGROUND

[0002] With the continuous improvement of digital office and knowledge management needs, the analysis and query of multi-modal documents has become a key technology for information extraction and retrieval. The current related technology cuts the document into over-finely granular fragment blocks based on visual features, which destroys the semantic coherence between chapters, titles and texts, resulting in fragmentation of the logical structure of the document. Or, it retains semantic integrity, but loses the layout and location information of the multi-modal document, and the complete text cannot realize the mapping of content and location, so that the user cannot reverse locate to the specific location of the multi-modal document through the search results, resulting in overload of fragmented information and failure of overall document positioning in multi-modal document processing. SUMMARY

[0003] The present application provides a document processing method, a document processing method, a document processing device, computer equipment, and a computer readable storage medium to at least solve the technical problems of overload of fragmented information and failure of overall document positioning in document processing.

[0004] The present application provides a document processing method, comprising: acquiring a multi-modal document; decomposing the multi-modal document to generate a plurality of document fragment blocks; wherein the document fragment block is associated with first location information thereof in the multi-modal document; splicing the plurality of document fragment blocks in accordance with the reading order of the multi-modal document to form a target text; dividing the target text into a plurality of logical blocks based on the semantic structure of the target text; wherein the logical block comprises the content of the target text; establishing an association relationship between the logical block and the document fragment block, and aggregating the first location information using the association relationship to determine second location information of the logical block in the multi-modal document; and storing the content of the logical block and the second location information in association.

[0005] The present application also provides a document processing device, comprising: an acquisition module for acquiring a multi-modal document; a decomposition module for decomposing the multi-modal document to generate a plurality of document fragment blocks; wherein the document fragment block is associated with first location information thereof in the multi-modal document; a splicing module for splicing the plurality of document fragment blocks in accordance with the reading order of the multi-modal document to form a target text; a division module for dividing the target text into a plurality of logical blocks based on the semantic structure of the target text; wherein the logical block comprises the content of the target text; an aggregation module for establishing an association relationship between the logical block and the document fragment block, and aggregating the first location information using the association relationship to determine second location information of the logical block in the multi-modal document; and a storage module for storing the content of the logical block and the second location information in association.

[0006] The application further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the document processing methods when executing the computer program.

[0007] The application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executable on a processor to implement the steps of any of the document processing methods.

[0008] The application generates a series of document fragments carrying accurate coordinate positioning capability through decomposition of the multi-modal document, that is, the first position information carried by the document fragments can reflect the specific position of the document fragments in the multi-modal document, thereby establishing an underlying data basis for subsequent physical position tracing; further, the document fragments are spliced in accordance with the reading order of the multi-modal document to form a target text, the original semantic flow and logical coherence of the multi-modal document are completely reconstructed, and information discontinuity caused by fragmented analysis is prevented; the logical blocks are dynamically divided based on the semantic structure of the target text, the title nodes and the associated content units such as paragraphs and tables are aggregated, the theme semantic modules retaining chapter integrity are formed, and information fragmentation in the fragmented processing is prevented; thereby, the association mapping between the logical blocks and the document fragments is established, and the bidirectional link between the semantic units and the physical positions is constructed; the coordinate information of the document fragments is aggregated through the association relationship, the second position information of the logical blocks in the original document is generated, and the complete theme units have cross-page and cross-column positioning capability; finally, the logical block content and the second position information are bound and stored, the minimum knowledge unit with semantic coherence and physical traceability is formed, thereby forming complete semantic expression, meeting the demand of the knowledge base construction for complete semantic expression, supporting the positioning and reverse tracing of the search results, solving the technical problems of fragmented information overload and overall document positioning failure in the document processing, and being beneficial to improving semantic integrity and positioning accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0009] In order to more clearly illustrate the embodiments of the application, the drawings required to be used in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0010] Figure 1 An application environment diagram of a document processing method provided by an embodiment of the application; Figure 2 A flowchart of a document processing method provided by an embodiment of the application; Figure 3 A flowchart of another document processing method provided by an embodiment of the application; Figure 4This is a schematic diagram of the structure of a document processing device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0011] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0012] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0013] It should be noted that the terms "S1," "S2," etc., are used only for descriptive purposes and do not specifically refer to the order or sequence, nor are they intended to limit this application. They are merely for the convenience of describing the method of this application and should not be construed as indicating the sequential order of the steps. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0014] The document processing method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 12 communicates with server 14 via a network. Terminal 12 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices, and server 14 can be a standalone server or a server cluster consisting of multiple servers.

[0015] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0016] The embodiments of this application provide a document processing method, and the method is described in detail below in conjunction with the execution flow of the document processing method.

[0017] In one embodiment, such as Figure 2 As shown, Figure 2 This is a flowchart illustrating a document processing method provided in an embodiment of this application.

[0018] S201: Obtain multimodal documents.

[0019] In this embodiment, a multimodal document refers to a document containing heterogeneous elements such as text, tables, images, and formulas, such as PDF (Portable Document Format) and scanned documents, whose content needs to parse both visual layout and semantic information.

[0020] Specifically, multimodal parsers, such as OCR (Optical Character Recognition) engines or layout analysis algorithms, can be used to parse and scan multimodal documents, extracting the text content and the corresponding position coordinates of the text content. For example, the text content is "Example", and the position coordinates are [x1,y1,x2,y2]. By acquiring and parsing the multimodal document, the original document is transformed into structured data, providing structured document input for subsequent processing and preserving the multimodal characteristics of the original multimodal document, such as the spatial relationships between text positions, tables, and images.

[0021] S202: Decompose the multimodal document to generate multiple document fragments; wherein the document fragments are associated with their first position information in the multimodal document.

[0022] In this embodiment, a document fragment represents the smallest semantic unit after the document has been segmented, such as a single sentence or a table cell. The document fragment carries its physical location coordinates in the multimodal document.

[0023] Specifically, decomposing a multimodal document to generate multiple document fragments can cut text paragraphs according to sentence boundaries, with each sentence fragment labeled with its original position. For example, the text content is "This is a sentence.", and the position coordinates are [100, 200, 300, 220]. This generates basic content units carrying position anchors.

[0024] S203: Combine multiple document fragments into the target text according to the reading order of the multimodal document.

[0025] In this embodiment, the reading order refers to a text sequence that conforms to human reading habits, such as from left to right or from top to bottom. The target text refers to the complete text stream formed by piecing together document fragments in the reading order.

[0026] Specifically, assembling target text according to the reading order of a multimodal document refers to reassembling multiple fragmented document pieces to ensure the assembled document conforms to the normal reading order of a multimodal document. This prevents the numerous fragmented document pieces from failing to integrate into a document with complete semantic meaning. In this case, the target text assembled according to the reading order already carries the first positional information of each document piece within the multimodal document, and the target text conforms to the normal reading order. The target text retains both semantic integrity and positional information, thus completing the reconstruction of a seamless, continuous text flow.

[0027] S204: Based on the semantic structure of the target text, divide the target text into multiple logical blocks; wherein, the logical blocks include the content of the target text.

[0028] In this embodiment, semantic structure represents the hierarchical logical organization of a multimodal document, such as the logical hierarchy of elements like headings, paragraphs, and lists; logical blocks represent complete thematic units that aggregate headings and related content, such as chapters.

[0029] Specifically, by parsing the syntax tree of the target text, the heading nodes, tables, and other elements in the target text can be detected and aggregated into paragraphs under the heading to form a logical block. This facilitates the generation of complete thematic units that retain the complete text of the chapter or images and list elements belonging to the text content.

[0030] S205: Establish the association between logical blocks and document fragment blocks, and use the association to aggregate the first position information to determine the second position information of the logical block in the multimodal document.

[0031] In this embodiment, the association relationship represents the mapping relationship between the logical block and the document fragment blocks that make up it; the second location information represents the coordinates of the smallest bounding region of the logical block in the multimodal document.

[0032] Specifically, the association between logical blocks and document fragment blocks can be established by calculating the longest common substring between the logical block text and the document fragment block text, thus constructing a mapping between semantic units and physical locations.

[0033] In this embodiment, document fragments carrying location information in a multimodal document are matched with logical blocks divided by semantic structure in a one-to-one, one-to-many, many-to-one, or many-to-many manner. This ensures that, while maintaining semantic integrity, each logical block possesses its location information within the multimodal document, thus establishing a mapping relationship between document content and location information. This mapping relationship can be established by calculating the coordinates of the minimum bounding box region.

[0034] S206: Associate the contents of the logic block with the second location information and store them together.

[0035] In this embodiment, associated storage refers to the bound storage of logical block content and its second location information. Vector storage, non-vector storage, or a combination of both storage methods can be used to allow users to display related elements such as text, images, and lists together when querying document content, thus achieving the joint output of multimodal data.

[0036] In this embodiment, by decomposing the multimodal document, a series of document fragments carrying precise coordinate positioning capabilities are generated. The first location information carried by each document fragment reflects its specific location within the multimodal document, establishing a foundational data basis for subsequent physical location tracing. Furthermore, the document fragments are assembled according to the reading order of the multimodal document to form the target text, completely reconstructing the original semantic flow and logical coherence of the multimodal document, preventing information gaps caused by fragmented parsing. Logical blocks are dynamically divided based on the semantic structure of the target text, aggregating title nodes and their associated paragraphs, tables, and other content units to form a thematic semantic module that preserves the integrity of the chapters, preventing information fragmentation during fragmented processing. By establishing an association mapping between logical blocks and document fragments, a bidirectional link between semantic units and physical locations is constructed. By aggregating the coordinate information of document fragments through association relationships, a second location information of logical blocks in the original document is generated, enabling complete topic units to have cross-page and cross-column positioning capabilities. Finally, the content of logical blocks is bound and stored with the second location information to form the smallest knowledge unit with both semantic coherence and physical traceability, thereby forming a complete semantic expression. This meets the requirements of knowledge base construction for complete semantic expression and supports the positioning and reverse tracing of search results. It solves the technical problems of fragmented information overload and overall document positioning failure in document processing, which is conducive to improving semantic integrity and positioning accuracy.

[0037] In one embodiment, such as Figure 3 As shown, Figure 3 This is a flowchart illustrating another document processing method provided in an embodiment of this application.

[0038] In response to receiving a multimodal document processing request from a user, the system executes the local service requested by the user using a document parser, which can be a Local Document Parser that coordinates the entire processing flow.

[0039] Further, obtain a multimodal document from a specified source, such as a web URL (Uniform Resource Locator) or an internal system. This document can be a PDF.

[0040] Once a multimodal document is obtained, a multimodal parser can be used to decompose it into multiple document fragments. This multimodal parser simultaneously launches three subtasks to deconstruct and analyze the PDF document, processing content of different natures within the document.

[0041] Specifically, this includes structural analysis of the multimodal document, parsing its physical layout, and generating intermediate JSON (JavaScript Object Notation, a data exchange format) containing metadata such as the coordinates of each element (coordinates as primary location information), page number, and type. Simultaneously, text extraction is performed to identify the text content within the document, preparing for the generation of target text. Furthermore, image elements are separated from the document, processed, and uploaded to the cloud to obtain their network addresses, completing the replacement of image references within the document. Each image is treated as a special document fragment, with its primary location information being its coordinates and its content being its URL.

[0042] In one embodiment, when generating a document fragment, it can be determined whether the number of characters in the document fragment is less than a preset second number of characters; in response to the number of characters in the document fragment being less than the preset second number of characters, the position of the document fragment is obtained; in response to the document fragment being located at the end boundary of a paragraph, it is merged backward into an adjacent logical block, and merging backward indicates that the target text corresponding to the document fragment is at the end of the reading order; in response to the document fragment being located at the beginning boundary of a paragraph, it is merged forward into an adjacent logical block, and merging forward indicates that the target text corresponding to the document fragment is at the beginning of the reading order.

[0043] Specifically, an isolated punctuation mark, page number, small icon, or figure caption has very few characters and does not constitute complete semantics on its own. This is addressed by checking if the number of characters in a document fragment is less than a preset second character count, for example, if the second character count is set to 3 characters. The document fragment "." has a character count of 1. Figure 1 The number of characters in the "" block is 2, which is less than 3, thus triggering a merging decision. The relationship between this excessively small document fragment and its adjacent text is analyzed to determine whether it is located at the beginning or end boundary of a paragraph.

[0044] Terminating boundaries, such as periods "." and ellipses "...", are usually located at the end of a complete sentence; beginning boundaries, such as... Figure 1 Numbers or labels such as ":", "①", etc., are usually located at the beginning of a paragraph or description. Based on the position of document fragments, they are merged into adjacent logical blocks to maintain semantic integrity.

[0045] Furthermore, for example, if a document fragment is a period "." at the end of a paragraph, this period is merged into the preceding logical block, that is, the end of the logical block before it, ensuring the integrity of the period as a sentence end marker. Similarly, a document fragment is " Figure 1 :, it is located at the beginning of a chart description, which will " Figure 1 "Document fragments are merged into the next logical block to which they belong, ensuring that they are together with the following figure caption text."

[0046] This embodiment identifies and eliminates fragments that cannot independently express a complete meaning, and merges them with adjacent logical blocks, ensuring that each text block contains as complete semantic information as possible, thus avoiding information fragmentation.

[0047] Furthermore, multiple document fragments are pieced together according to the reading order of the multimodal documents to form the target text.

[0048] Specifically, the text content obtained through text extraction is used, and the document fragments are pieced together according to the reading order based on the coordinate information provided by structural analysis. Simultaneously, Markdown (markup language) syntax is used to preserve the document's semantic structure, such as headings and lists, and image links provided by URL replacement are embedded. The final result is a coherent target text rich in semantic and multimedia information, which can be in Markdown format.

[0049] Markdown stands for Lightweight Markup Language, designed for the rapid creation and conversion of structured documents in plain text format. Markdown text uses simple symbols to express formatting semantics, allowing content creators to generate clearly structured documents without worrying about complex formatting. OCR or direct text extraction technologies are used to recognize the text content in the document. These text fragments are then pieced together in reading order and converted into a semantically rich Markdown format. Markdown text uses "#" for headings and "-" for lists, providing a foundation for subsequent logical block division.

[0050] Furthermore, based on the semantic structure of the target text, the target text is divided into multiple logical blocks; where each logical block includes the content of the target text.

[0051] In this embodiment, the division into multiple logical blocks can be done in various ways, such as intelligent block segmentation, custom block segmentation, or QA (Question Answering) block segmentation. The specific method is not limited here. Taking intelligent block segmentation as an example, the semantic structure of the Markdown format is parsed.

[0052] In one embodiment, the target text is parsed to construct an abstract syntax tree, and segmentation nodes are identified through the abstract syntax tree. The segmentation nodes include at least one of the following: heading nodes, paragraph nodes, and table nodes. For example, segmentation nodes are identified by symbols such as "#" and "-".

[0053] Furthermore, a current logical block is created associated with the segment node, and paragraph nodes and / or table nodes that are consecutive to the segment node (which is a title node) are included in this current logical block. Specifically, logical blocks are created based on segment nodes; if the segment node is a title node, then consecutive paragraph, list, or table nodes under that title are included in the same logical block. Tables can be converted to HTML (Hypertext Markup Language) format to maintain table integrity.

[0054] In one embodiment, it can be determined whether the number of characters in the current logical block exceeds a preset first number of characters. In response to the fact that the number of characters in the current logical block exceeds the preset first number of characters, the statement boundary that is closest to the end of the text of the current logical block is obtained, and the statement boundary is used as the dividing node between the current logical block and the next logical block to generate a first logical block and a second logical block adjacent to the first logical block.

[0055] Specifically, the number of characters in the logical block is determined. For example, the first character count is preset to 256 characters. If the number of characters in the logical block exceeds 256 characters, it will be split at the nearest statement boundary, such as a period, and the logical block that exceeds the preset number of characters will be divided into two adjacent logical blocks, namely the first logical block and the second logical block adjacent to the first logical block.

[0056] Furthermore, a predetermined number of words at the end of the first logical block text are used as words at the beginning of the second logical block text.

[0057] Specifically, overlapping words can be retained to ensure contextual coherence. For example, 10 words can be retained at the end of the first logic block, and these 10 words can be used as the 10 words at the beginning of the second logic block.

[0058] This embodiment determines whether the number of characters in a logical block exceeds a preset threshold and segments it at the nearest statement boundary, while retaining overlapping words to avoid exceeding the context window limit of downstream processing modules due to excessive length of a single logical block, thus ensuring the feasibility of system processing. By prioritizing segmentation at natural statement boundaries, the integrity of semantic units is maintained to the greatest extent, preventing semantic breaks or information distortion caused by truncation, thereby improving the readability and logical coherence of the segmented content. The introduction of overlapping words alleviates the problem of context loss that may be caused by segmentation, enabling subsequent vector retrieval or content analysis to more accurately capture cross-block language dependencies.

[0059] If the user configures custom chunking, they can set specific rules for logical block division, such as the number of characters in a logical block, the number of overlapping lexical units, or force chunking at delimiters. If the user configures QA chunking, they can filter key content, selecting high-information-density paragraphs such as definitions and conclusions. High information density refers to information with many keywords, such as core concepts in document titles, bold or italic text, content in ordered and unordered lists, sentences containing phrases like "defined as," "refers to," and "means," recurring concepts, or parameters, configurations, and specifications in technical documents. The key content is then input into a large language model, which generates structured question-and-answer pairs.

[0060] Furthermore, the association between logical blocks and document fragment blocks is established, and the association is used to aggregate the first position information to determine the second position information of the logical block in the multimodal document.

[0061] In one embodiment, the text content of each logical block generated by the block processing is matched with the document fragment blocks. This matching process evaluates the association between the logical blocks and the document fragment blocks using algorithms such as calculating the longest common substring, and filters out invalid associations based on the hit score.

[0062] Specifically, the longest common subsequence (LCS) of the logical block and the document fragment block is evaluated; the ratio of the length of the longest common subsequence to the content length of the logical block is calculated, and the ratio is used as the first factor; the product of the first factor and the length of the longest common subsequence is calculated, and the value of the product is used as the hit score of the longest common subsequence; it is determined whether the hit score is less than a preset hit score threshold; in response to the hit score being less than the preset hit score threshold, the association between the logical block and the document fragment block corresponding to the hit score is filtered; in response to the hit score being not less than the preset hit score threshold, the first position information is aggregated to determine the second position information of the logical block in the multimodal document.

[0063] Aggregating the first position information to determine the second position information of a logical block in a multimodal document includes obtaining document fragment blocks associated with the logical block, obtaining the first position information of at least some of the document fragment blocks, where the first position information at least includes a first coordinate and a second coordinate having a first direction value and a second direction value, and the first direction value and the second direction value corresponding to the first coordinate are less than the first direction value and the second direction value corresponding to the second coordinate; traversing the first coordinate and the second coordinate, obtaining the minimum value of the first direction corresponding to the first coordinate and the minimum value of the second direction value, forming the coordinates composed of the minimum value of the first direction and the minimum value of the second direction value as the target first coordinate, obtaining the maximum value of the first direction corresponding to the first coordinate and the maximum value of the second direction value, forming the coordinates composed of the maximum value of the first direction and the maximum value of the second direction value as the target second coordinate, and using the outer bounding box coordinates determined by the target first coordinate and the target second coordinate as the second position information of the logical block in the multimodal document.

[0064] Specifically, for example, the content of the logical block is "The present invention relates to document processing technology and aims to solve the fragmentation problem.", with a length of 20 characters; the content of the document fragment block is "Document processing technology.", with a length of 6 characters; the preset hit score threshold is 5.0.

[0065] First, find the longest continuous identical string sequence between the logical block text and the document fragment block text, which is "Document processing technology", and the length of this substring is 6. Calculate the first factor through Equation 1-1: First factor = LCS length / Logical block content length (Equation 1-1); That is, 6 / 20 = 0.3. The first factor measures the importance of this document fragment block to the entire logical block. The larger the ratio, the higher the proportion of the content of this document fragment block in the logical block and the more important it is.

[0066] Furthermore, calculate the hit score through Equation 1-2: Hit score = First factor * LCS length (Equation 1-2); That is, 0.3 * 6 = 1.8. The hit score takes into account both the matching length and the importance of the document fragment block. That is, a very long matching string can get a high score even if its proportion in the logical block is not particularly high; a shorter matching string can also get a high score if it is almost the entire content of the logical block. Based on this, it is possible to filter out accidentally occurring and very short synonymous words or common words, such as the match of "of".

[0067] Compare the calculated hit score 1.8 with the preset threshold 5.0. Since 1.8 < 5.0, it is determined that the association relationship between the current document fragment block and this logical block is too weak. It may be just an accidental match or an unimportant segment. Therefore, filter out this association relationship and do not use it for subsequent position aggregation.

[0068] Conversely, if the score is ≥5.0, the document fragment is considered a valid part of the logical block, and the association is retained.

[0069] The location information of all document fragments that are identified as belonging to the same logical block is merged into an outer bounding box that can cover the entire visual area of ​​the logical block, i.e., the minimum bounding box.

[0070] For example, the aforementioned logical block consists of 3 document fragment blocks, each of which has its first position information. The 3 document fragment blocks include document fragment block A: (x1=100, y1=200, x2=300, y2=220); document fragment block B: (x1=100, y1=230, x2=320, y2=250); and document fragment block C: (x1=110, y1=260, x2=280, y2=280).

[0071] Obtain the coordinates of document fragments A, B, and C, iterate through them, and extract the extreme values. The first direction value typically refers to the horizontal (X-axis) coordinate, and the second direction value typically refers to the vertical (Y-axis) coordinate. The first coordinate usually refers to the top-left corner coordinate (x1, y1) of the rectangle, and the second coordinate usually refers to the bottom-right corner coordinate (x2, y2). Iterate through the top-left corner coordinates (x1, y1) of all document fragments. The minimum value of X1 is min(100, 100, 110) = 100, and the minimum value of Y1 is min(200, 230, 260) = 200, meaning the target first coordinate is (100, 200). Similarly, iterate through the bottom-right corner coordinates (x2, y2) of all document fragments. The maximum value of X2 is max(300, 320, 280) = 320, and the maximum value of Y2 is max(220, 250, 280) = 280, meaning the target second coordinate is (320, 280). The first and second coordinates of the target, determined by the minimum and maximum values, are combined to form the final bounding box coordinates. The second position information of this logic block is [100, 200, 320, 280].

[0072] In this embodiment, the smallest rectangle encompasses the content of document fragments A, B, and C, accurately representing the visual area occupied by the logical block on the multimodal document page. This coordinate information provides a basis for subsequent highlighting, positioning, or visualization. By calculating the longest common substring hit score between the logical block and the document fragments, and filtering low-score associations based on a preset threshold, noisy matches and accidental overlaps are eliminated, ensuring strong semantic relevance of the associations. Furthermore, by aggregating the position coordinates of all high-score document fragments, the macroscopic position of the logical block in the multimodal document is reconstructed, improving the accuracy of block associations and the reliability of positional location.

[0073] In one embodiment, a logic block has an identifier used to identify the corresponding logic block. The content of the target text corresponding to the logic block is input to the encoder to generate the corresponding vector code. The identifier and vector code corresponding to the logic block are stored as key-value pairs. Alternatively, the identifier and second location information corresponding to the logic block are stored as key-value pairs. In response to the association of a logic block with an image, the identifier and a list of image addresses are stored as key-value pairs.

[0074] Specifically, the second location information corresponding to the content of each logical block and the image URLs contained within the logical block are used as metadata and associated with the corresponding logical block. The content of the logical block is encoded as a vector and stored in a vector database along with the metadata, forming searchable key-value pairs. Alternatively, metadata can be used to store location JSON associated with the logical block identifier, such as page number or bounding box; or a list of image URLs associated with the logical block identifier; or the type identifier of the logical block division, such as intelligent segmentation, custom segmentation, or QA segmentation type. When vector retrieval returns the logical block identifier, the metadata and image URLs or the type of logical block division are loaded synchronously to achieve multimodal data joint output. In knowledge base construction, complete topic units with location information can be stored; in document retrieval, the entire relevant chapter can be highlighted instead of fragments; in question-answering systems, specific areas of documents can be directly referenced as the basis for answers; and in large-scale model question answering, vector blocks can be located through questions, and answers can be inferred from the content of vector blocks.

[0075] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0076] Embodiments of this application also provide a document processing apparatus, such as... Figure 4 As shown, Figure 4 This is a schematic diagram of a document processing device provided in an embodiment of this application. The document processing device may include an acquisition module 21, a decomposition module 22, a splicing module 23, a division module 24, an aggregation module 25, and a storage module 26.

[0077] Module 21 is used to acquire multimodal documents.

[0078] The decomposition module 22 is used to decompose the multimodal document to generate multiple document fragments; wherein the document fragments are associated with their first position information in the multimodal document.

[0079] The splicing module 23 is used to splice multiple document fragments into target text according to the reading order of the multimodal document.

[0080] This is used to determine whether the number of characters in a document fragment is less than a preset second number of characters; in response to the number of characters in a document fragment being less than the preset second number of characters, the position of the document fragment is obtained; in response to the document fragment being located at the end boundary of a paragraph, it is merged backward into an adjacent logical block, and merging backward indicates that the target text corresponding to the document fragment is at the end of the reading order; in response to the document fragment being located at the beginning boundary of a paragraph, it is merged forward into an adjacent logical block, and merging forward indicates that the target text corresponding to the document fragment is at the beginning of the reading order.

[0081] The segmentation module 24 is used to divide the target text into multiple logical blocks based on the semantic structure of the target text; wherein, the logical blocks include the content of the target text.

[0082] Used to parse target text and construct an abstract syntax tree; identify segmentation nodes through the abstract syntax tree, where segmentation nodes include at least one of the following: heading nodes, paragraph nodes, and table nodes; create a current logical block associated with the segmentation node, and include paragraph nodes and / or table nodes that are consecutive to the heading node in the current logical block.

[0083] Used to determine whether the number of characters in the current logical block exceeds a preset first character count; in response to the number of characters in the current logical block exceeding the preset first character count, the statement boundary closest to the end of the text of the current logical block is obtained, and the statement boundary is used as the dividing node between the current logical block and the next logical block, generating a first logical block and a second logical block adjacent to the first logical block; a preset number of words at the end of the text of the first logical block are used as words at the beginning of the text of the second logical block.

[0084] The aggregation module 25 is used to establish the association between logical blocks and document fragment blocks, and to aggregate the first position information using the association to determine the second position information of the logical block in the multimodal document.

[0085] The algorithm evaluates the longest common substring between logical blocks and document fragments; calculates the ratio of the length of the longest common substring to the content length of the logical block, and uses this ratio as the first factor; calculates the product of the first factor and the length of the longest common substring, and uses the product as the hit score of the longest common substring; determines whether the hit score is less than a preset hit score threshold; if the hit score is less than the preset hit score threshold, it filters the association between the logical block and the document fragment corresponding to the hit score; if the hit score is not less than the preset hit score threshold, it aggregates the first position information to determine the second position information of the logical block in the multimodal document.

[0086] This method is used to obtain document fragments associated with logical blocks; obtain first position information of at least some document fragments, the first position information including at least a first coordinate and a second coordinate having a first direction value and a second direction value, wherein the first direction value and the second direction value corresponding to the first coordinate are less than the first direction value and the second direction value corresponding to the second coordinate; traverse the first coordinate and the second coordinate; obtain the minimum value of the first direction and the minimum value of the second direction corresponding to the first coordinate, and use the coordinate formed by the minimum value of the first direction and the minimum value of the second direction as the target first coordinate; obtain the maximum value of the first direction and the maximum value of the second direction corresponding to the first coordinate, and use the coordinate formed by the maximum value of the first direction and the maximum value of the second direction as the target second coordinate; use the bounding box coordinates determined by the target first coordinate and the target second coordinate as the second position information of the logical block in the multimodal document.

[0087] Storage module 26 is used to associate and store the contents of the logic block with the second location information.

[0088] This is used to input the content of the target text corresponding to the logic block into the encoder to generate the corresponding vector code; store the identifier corresponding to the logic block and the vector code in key-value pairs; and / or, store the identifier corresponding to the logic block and the second position information in key-value pairs; in response to the association of the logic block with an image, store the identifier corresponding to the logic block and the address list of the image in key-value pairs.

[0089] Each module in the aforementioned setting control device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0090] Embodiments of this application also provide a computer device, including a memory for storing computer programs.

[0091] A processor, when executing a computer program, can perform at least the following steps: The process involves: acquiring a multimodal document; decomposing the multimodal document to generate multiple document fragments; where each document fragment is associated with its first position information within the multimodal document; concatenating the multiple document fragments according to the reading order of the multimodal document to form the target text; dividing the target text into multiple logical blocks based on its semantic structure; where each logical block includes the content of the target text; establishing a relationship between the logical blocks and the document fragments, and using this relationship to aggregate the first position information to determine the second position information corresponding to each logical block within the multimodal document; and storing the content of each logical block in association with the second position information.

[0092] In one embodiment, the computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores software management data. The network interface allows for communication with external terminals via a network connection.

[0093] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0094] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, can perform at least the following steps: The process involves: acquiring a multimodal document; decomposing the multimodal document to generate multiple document fragments; where each document fragment is associated with its first position information within the multimodal document; concatenating the multiple document fragments according to the reading order of the multimodal document to form the target text; dividing the target text into multiple logical blocks based on its semantic structure; where each logical block includes the content of the target text; establishing a relationship between the logical blocks and the document fragments, and using this relationship to aggregate the first position information to determine the second position information corresponding to each logical block within the multimodal document; and storing the content of each logical block in association with the second position information.

[0095] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0096] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, can perform at least the following steps: The process involves: acquiring a multimodal document; decomposing the multimodal document to generate multiple document fragments; where each document fragment is associated with its first position information within the multimodal document; concatenating the multiple document fragments according to the reading order of the multimodal document to form the target text; dividing the target text into multiple logical blocks based on its semantic structure; where each logical block includes the content of the target text; establishing a relationship between the logical blocks and the document fragments, and using this relationship to aggregate the first position information to determine the second position information corresponding to each logical block within the multimodal document; and storing the content of each logical block in association with the second position information.

[0097] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0098] It will also be appreciated that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0099] The foregoing has provided a detailed description of a document processing method, system, device, and computer-readable storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A document processing method, characterized in that, The document processing method includes: Obtain multimodal documents; The multimodal document is decomposed to generate multiple document fragments; wherein each document fragment is associated with its first position information within the multimodal document; The multiple document fragments are assembled into the target text according to the reading order of the multimodal documents; Based on the semantic structure of the target text, the target text is divided into multiple logical blocks; wherein, the logical block includes the content of the target text; Establish the association between the logical block and the document fragment block, and use the association to aggregate the first location information to determine the second location information of the logical block in the multimodal document; The contents of the logic block are associated with and stored in relation to the second location information.

2. The document processing method according to claim 1, characterized in that, The division of the target text into multiple logical blocks based on the semantic structure of the target text includes: Parse the target text to construct an abstract syntax tree; The segmentation nodes are identified by the abstract syntax tree, and the segmentation nodes include at least one of the following: heading nodes, paragraph nodes, and table nodes. Create a current logical block associated with the segment node, and include the paragraph nodes and / or table nodes that are consecutive to the segment node as a title node into the current logical block.

3. The document processing method according to claim 2, characterized in that, The step of incorporating the paragraph nodes and / or table nodes following the title node into the current logical block includes: Determine whether the number of characters in the current logic block exceeds a preset first character count; In response to the current logical block having more than the preset first number of characters, the statement boundary closest to the end of the text of the current logical block is obtained, and the statement boundary is used as the dividing node between the current logical block and the next logical block, thereby generating a first logical block and a second logical block adjacent to the first logical block. A predetermined number of words at the end of the text of the first logic block are used as words at the beginning of the text of the second logic block.

4. The document processing method according to claim 3, characterized in that, The method further includes: Determine whether the number of characters in the document fragment is less than a preset second number of characters; If the number of characters in the document fragment is less than a preset second number of characters, the position of the document fragment is obtained. In response to the document fragment block being located at the end boundary of a paragraph, it is merged backwards into the adjacent logical block, wherein the backward merging indicates that the target text corresponding to the document fragment block is at the end of the reading order; In response to the document fragment block being located at the beginning boundary of a paragraph, it is merged forward to the adjacent logical block, wherein the forward merging indicates that the target text corresponding to the document fragment block is at the beginning of the reading order.

5. The document processing method according to claim 1, characterized in that, The process of establishing the association between the logical block and the document fragment block includes: Evaluate the longest common substring between the logical block and the document fragment block; Calculate the ratio of the length of the longest common substring to the content length of the logical block, and use the ratio as the first factor; Calculate the product of the first factor and the length of the longest common substring, and use the value of the product as the hit score of the longest common substring; Determine whether the hit score is less than a preset hit score threshold; In response to the hit score being less than a preset hit score threshold, the association between the logical block corresponding to the hit score and the document fragment block is filtered. In response to the hit score being not less than a preset hit score threshold, the first location information is aggregated to determine the second location information of the logic block in the multimodal document.

6. The document processing method according to claim 5, characterized in that, The aggregation of the first location information to determine the second location information of the logical block in the multimodal document includes: Obtain the document fragment block that has the aforementioned association with the logical block; Obtain first location information of at least a portion of the document fragment blocks, the first location information including at least a first coordinate and a second coordinate having a first direction value and a second direction value, wherein the first direction value and the second direction value corresponding to the first coordinate are less than the first direction value and the second direction value corresponding to the second coordinate; Iterate through the first coordinate and the second coordinate; Obtain the minimum value of the first direction and the minimum value of the second direction corresponding to the first coordinate, and use the coordinate formed by the minimum value of the first direction and the minimum value of the second direction as the target first coordinate; Obtain the maximum value of the first direction and the maximum value of the second direction corresponding to the first coordinate, and use the coordinate formed by the maximum value of the first direction and the maximum value of the second direction as the target second coordinate; The bounding box coordinates determined by the first and second coordinates of the target are used as the second position information of the logic block in the multimodal document.

7. The document processing method according to claim 1, characterized in that, The logic block has an identifier, which is used to identify the corresponding logic block. The step of associating and storing the content of the logic block with the second location information includes: The content of the target text corresponding to the logic block is input into the encoder to generate the corresponding vector code; the identifier corresponding to the logic block and the vector code are stored in key-value pair form. And / or, The identifier corresponding to the logical block and the second location information are stored in key-value pairs; in response to the association of the logical block with an image, the identifier corresponding to the logical block and the address list of the image are stored in key-value pairs.

8. A document processing apparatus, characterized in that, include: The acquisition module is used to acquire multimodal documents; A decomposition module is used to decompose the multimodal document to generate multiple document fragments; wherein the document fragments are associated with their first position information in the multimodal document; The splicing module is used to splice the multiple document fragments according to the reading order of the multimodal document to form the target text; The segmentation module is used to divide the target text into multiple logical blocks based on the semantic structure of the target text; wherein, the logical block includes the content of the target text; An aggregation module is used to establish the association between the logical block and the document fragment block, and to aggregate the first location information using the association to determine the second location information of the logical block in the multimodal document; A storage module is used to associate and store the contents of the logical block with the second location information.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the document processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the document processing method as described in any one of claims 1 to 7.

Images