Multi-modal document comparison method and system, storage medium and program product

Abstract

The invention discloses a multi-modal document comparison method and system, a storage medium and a program product, and relates to the technical field of data processing, and the method comprises the following steps: constructing a first index database based on a first document, and constructing a second index database based on a second document; traversing each first document block in the first document, retrieving a matching block matched with the first document block in each second document block of the second document based on the second index database, and determining a block difference based on the matching block; traversing each second document block, and determining a deleted block in each second document block based on the first index database; and writing the block difference and the deleted block into a difference knowledge graph, and generating a document comparison report based on the difference knowledge graph. According to the method, comprehensive and structured presentation of semantic-level difference information is realized through the architecture of the multi-modal block index-bidirectional matching retrieval-difference mapping knowledge domain, so that the semantic-level difference recognition accuracy of a multi-modal document comparison technology is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a multi-modal document comparison method and system, a storage medium and a program product. BACKGROUND

[0002] In high-risk industries such as finance, government affairs and medicine, a large number of scanned documents or PDF (Portable Document Format) documents containing text, images, charts, signatures and other multi-modal elements need to be processed daily. These documents often carry key information such as contract terms, financial data, approval records and medical reports, and the differences between their versions are directly related to business compliance, risk control and audit traceability. Existing multi-modal document comparison techniques attempt to combine OCR (Optical Character Recognition), image recognition and large language models for cross-modal analysis, but are limited by the limitations of the technical path and are difficult to achieve true semantic-level difference recognition.

[0003] On the one hand, there are natural differences in the feature spaces of text, images, charts, signatures and other modalities: text carries abstract semantics in the form of discrete character sequences, images / charts present concrete information in the form of pixel distribution or spatial structure, and signatures convey validity information in the form of visual features. Existing technologies often simply concatenate features of each modality, such as directly concatenating text vectors recognized by OCR with image pixel features, or independently processing single-modal data such as separately comparing text character differences and separately calculating image pixel deviations, resulting in the semantic association between multi-modalities being fragmented. On the other hand, existing document comparison techniques are difficult to cope with the challenges posed by document layout changes: when documents undergo changes in layout such as cross-page splitting and page order adjustment, traditional line-level or single-page static comparison methods will produce a large number of false differences due to the positional offset of block-level elements, such as line breaks and font changes being misjudged as content changes, and are unable to determine true semantic-level differences. In summary, existing multi-modal document comparison techniques have low accuracy in semantic-level difference recognition, which affects the results of multi-modal document comparison.

[0004] The above content is only used to assist in understanding the technical solutions of the present application and does not represent an admission that the above content is prior art. SUMMARY

[0005] The main purpose of the present application is to provide a multi-modal document comparison method, system, storage medium and program product, which aims to solve the technical problem that existing multi-modal document comparison techniques have low accuracy in semantic-level difference recognition, which affects the results of multi-modal document comparison.

[0006] To achieve the above-mentioned purpose, the present application provides a multi-modal document comparison method, which comprises: constructing a first index database based on the first document, and constructing a second index database based on the second document, wherein the first index database comprises first inverted indexes corresponding to text blocks in the first document and first vector indexes corresponding to non-text blocks in the first document, and the second index database comprises second inverted indexes corresponding to text blocks in the second document and second vector indexes corresponding to non-text blocks in the second document; traversing each first document block in the first document, retrieving matching blocks matching the first document block in each second document block of the second document based on the second index database, and determining block differences between the matching blocks and the first document block; traversing each second document block, determining deletion blocks in each second document block based on the first index database; writing the block differences and the deletion blocks into a difference knowledge graph, and generating a document comparison report based on the difference knowledge graph.

[0007] In an embodiment, the first index database comprises a character index database and a vector index database. The step of constructing a first index database based on the first document comprises: extracting text blocks in the first document and identifying text characters, clustering the text characters into a plurality of first semantic text blocks based on projection clustering, establishing first inverted indexes and storing them into the character index database, wherein in the inverted indexes, keywords of the first semantic text blocks are mapped to block identifiers of document blocks where the first semantic text blocks are located, coordinate positions of the first semantic text blocks in the first document, and page numbers; determining block boundaries of non-text blocks in the first document, calling a preset feature extraction model to perform feature coding on the non-text blocks in the first document based on the block boundaries to generate first coding vectors, establishing first vector indexes and storing them into the vector index database, wherein in the first vector indexes, the first coding vectors are mapped to block identifiers, coordinate positions, and page numbers of the non-text blocks in the first document.

[0008] In an embodiment, the step of retrieving matching blocks matching the first document block in each second document block of the second document based on the second index database comprises: if the first document block is a text block, calling a text retrieval tool to recall block identifiers of the top N similar semantic text blocks based on the second inverted indexes, and obtaining block contents based on the block identifiers of the top N similar semantic text blocks; if the block content of the first document block is non-text content, calling a visual retrieval tool to recall block identifiers of top N similar vectors based on the second vector index, and obtaining the block content based on the block identifiers of the top N similar vectors; inputting the block content into a preset difference determination model to obtain a difference type, wherein the difference type includes addition, modification, movement, and image replacement.

[0009] In an embodiment, the step of determining the deletion block in each of the second document blocks based on the first index database comprises: querying whether there is a block matching the second document block in each of the first document blocks based on the first index database; if there is no block matching the second document block, determining the second document block as a deletion block.

[0010] In an embodiment, before the step of writing the block difference and the deletion block into the difference knowledge graph, further comprising: creating nodes for each block in the first document and the second document respectively, storing the block identifier of each block, the coordinate position in the corresponding document, the page number, and the content type of the document block; The step of writing the block difference and the deletion block into the difference knowledge graph comprises: determining a difference level corresponding to the block difference, creating a connection edge between the nodes corresponding to the blocks associated with the block difference, and binding the difference type, difference semantic explanation, and difference level of the block difference to the edge attribute of the connection edge; creating a self-loop edge on the node corresponding to the deletion block, and binding the edge attribute of the self-loop edge to deletion.

[0011] In an embodiment, the step of determining the difference level corresponding to the block difference comprises: inputting the blocks associated with the block difference in the first document and the second document into a preset semantic analysis model to obtain a semantic difference degree, determining a visual difference degree and a structural difference degree between the blocks associated with the block difference, wherein the semantic difference degree represents the semantic conflict degree between the blocks associated with the block difference, the visual difference degree represents the appearance feature difference between the blocks associated with the block difference, and the structural difference degree represents the physical layout difference of the blocks associated with the block difference in the first document and the second document; weighting the semantic difference degree, the visual difference degree, and the structural difference degree to obtain a three-dimensional score corresponding to the block difference, and determining a difference level based on the three-dimensional score.

[0012] In an embodiment, the step of generating a document comparison report based on the difference knowledge graph comprises: In response to a natural language question input by a user, retrieving and generating a difference location evidence chain in the difference knowledge graph based on the natural language question, and generating a question answer based on the difference location evidence chain; Calling a comment tool to superimpose a comment layer at the difference block position in the first document and a comment layer at the deletion block position in the second document to obtain a comment document; Merging the comment document and the question answer to obtain a document comparison report.

[0013] In addition, to achieve the above-mentioned purpose, the present application also provides a multi-modal document comparison system, which comprises: A pre-indexing module is configured to construct a first index database based on a first document and a second index database based on a second document, wherein the first index database comprises a first inverted index corresponding to a text block in the first document and a first vector index corresponding to a non-text block in the first document, and the second index database comprises a second inverted index corresponding to a text block in the second document and a second vector index corresponding to a non-text block in the second document; A traversal module is configured to traverse each first document block in the first document, retrieve a matching block matching the first document block in each second document block in the second document based on the second index database, and determine a block difference between the matching block and the first document block; The traversal module is further configured to traverse each second document block, and determine a deletion block in each second document block based on the first index database; A document comparison module is configured to write the block difference and the deletion block into a difference knowledge graph, and generate a document comparison report based on the difference knowledge graph.

[0014] In addition, to achieve the above-mentioned purpose, the present application also provides a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium, and the computer program is executed by a processor to implement the steps of the multi-modal document comparison method as described above.

[0015] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the multi-modal document comparison method as described above.

[0016] In the present application, a first index database is constructed based on the first document, and a second index database is constructed based on the second document, wherein the first index database includes first inverted indexes corresponding to text blocks in the first document and first vector indexes corresponding to non-text blocks in the first document, and the second index database includes second inverted indexes corresponding to text blocks in the second document and second vector indexes corresponding to non-text blocks in the second document. The inverted indexes of the text blocks can convert discrete character sequences into semantic identifiers that can be efficiently retrieved, and the vector indexes of the non-text blocks can convert pixel distribution, spatial structure and other concrete information into quantifiable feature vectors that can be compared, and the two kinds of structured indexes correspond to feature attributes of different modalities, so that different modalities are associated under a unified index framework, thereby avoiding semantic fragmentation caused by simple splicing or independent processing, and improving the accuracy of semantic difference recognition. Each first document block in the first document is traversed, and matching blocks matching the first document block are retrieved in each second document block of the second document based on the second index database, and block differences between the matching blocks and the first document block are determined; each second document block is traversed, and deletion blocks in each second document block are determined based on the first index database. By retrieving matching items based on the index database in block units, the spatial limitation of format changes is broken, and when the document is split across pages or the order is adjusted, the content of the block does not change in nature, and the corresponding relationship between the two can still be recognized through index matching, and then the substantial difference in the block can be accurately determined, thereby eliminating the interference of non-content formatting factors such as line breaks and font changes, and reducing the generation of pseudo-differences. And through bidirectional retrieval, whether it is the removal of the entire text block or the deletion of non-text blocks such as images and signatures, it can be accurately recognized through index comparison, avoiding the deletion of information caused by independent processing of modalities, and ensuring the comprehensiveness of semantic difference recognition. The block differences and the deletion blocks are written into a difference knowledge graph and a comparison report is generated, wherein the knowledge graph can structurally integrate scattered multi-modal difference information according to semantic association, intuitively display the internal relationship between the differences, solve the problem of scattered difference results and difficult comprehensive analysis, and enable the generated report to clearly present all semantic differences. In summary, through the architecture of multi-modal block index-bidirectional matching retrieval-difference knowledge graph, the present application avoids the loss of cross-modal semantic association and pseudo-differences under formatting changes, realizes the comprehensive and structured presentation of semantic difference information, and thereby improves the accuracy of multi-modal document comparison technology in semantic difference recognition. BRIEF DESCRIPTION OF DRAWINGS The accompanying drawings, which are incorporated into and form a part of the specification, illustrate one embodiment consistent with the present application and, together with the description, serve to explain the principles of the application.

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative effort.

[0018] Figure 1 The flowchart provided by the first embodiment of the multi-modal document comparison method of the present application; Figure 2 The flowchart provided by the second embodiment of the multi-modal document comparison method of the present application; Figure 3 The flowchart provided by the third embodiment of the multi-modal document comparison method of the present application; Figure 4 The brief flowchart of the multi-modal document comparison method provided by an embodiment of the present application; Figure 5 The module structure diagram of the multi-modal document comparison system of the embodiment of the present application; Figure 6 The device structure diagram of the hardware running environment involved in the multi-modal document comparison method in the embodiment of the present application.

[0019] The purpose implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0020] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application, and are not used to limit the present application.

[0021] In order to better understand the technical solutions of the present application, the following will be described in detail with reference to the drawings in the specification and the specific embodiments.

[0022] It should be noted that the execution subject of the present embodiment can be a computing service device with data processing, network communication and program running functions, such as tablet computers, personal computers, mobile phones, etc., or an electronic device capable of realizing the above functions. The present embodiment and the following embodiments will be described taking the electronic device as an example.

[0023] Based on this, the present embodiment provides a multi-modal document comparison method, which will be described in detail with reference to Figure 1 , Figure 1 The flowchart of the first embodiment of the multi-modal document comparison method of the present application.

[0024] In the present embodiment, the multi-modal document comparison method comprises steps S10-S40: At step S10, a first index database is constructed based on the first document, and a second index database is constructed based on the second document, wherein the first index database comprises first inverted indexes corresponding to the text blocks in the first document and first vector indexes corresponding to the non-text blocks in the first document, and the second index database comprises second inverted indexes corresponding to the text blocks in the second document and second vector indexes corresponding to the non-text blocks in the second document.

[0025] The first document and the second document are documents that need to be compared, and the first and the second are only distinguished in name and do not constitute a limitation on the order. The text block is a document area that uses a discrete character sequence as a carrier to convey abstract semantic information, which is composed of characters, letters, numbers, symbols, etc., and can be recognized as editable text content by OCR technology, such as clause descriptions in contracts, analysis conclusions in reports, and word explanations in tables. The non-text block is a document area that uses visual features or spatial structures as a carrier to convey concrete information, including image, chart, signature, formula, etc. non-text content, such as product design drawings, data statistical charts, electronic signatures, mathematical formulas, etc., the information of which needs to be reflected through visual features or spatial layout.

[0026] The index database is a data collection organized according to a specific data structure, which is used for fast data lookup and access. The inverted index maps each word or phrase in the text to a list of documents containing the word or phrase. For text blocks, inverted indexes can quickly locate the document part containing specific text content, which helps subsequent text comparison. The vector index is a non-text data such as image, chart, signature, etc. converted into vector form through a specific algorithm, and then an index is constructed according to the characteristics of the vector, so that similar non-text content can be quickly found in the index using distance and other measurement methods between vectors.

[0027] The first document is divided into text blocks and non-text blocks. For text blocks, inverted indexes are established to record the document location of the text segment and other information to form the first inverted index. For non-text blocks, image feature extraction algorithms are used to convert image pixel distribution, spatial structure, and other information into vector representation, and then the first vector index is constructed according to the characteristics of these vectors and stored in the first index database. Specifically, the first document is read and divided into text blocks and non-text blocks using text recognition technology and document segmentation technology. For text blocks, a word segmentation algorithm in natural language processing is used for word segmentation, and then each word is associated with the text block location containing it and other information through an inverted index construction algorithm to form the first inverted index. For non-text blocks, features are extracted according to their types and converted into vectors, and then a vector index construction algorithm is used to construct the first vector index. The two indexes are stored in the first index database. Similarly, the same operation is performed on the second document to construct the second index database.

[0028] Step S20, traversing each first document block in the first document, retrieving a matching block matching the first document block in each second document block of the second document based on the second index database, and determining a block difference between the matching block and the first document block.

[0029] The matching block is a document block found in the second document that has similarity in content or features with the first document block, determined by retrieval of the index database. The block difference refers to the difference in content, structure, etc. between the matching first document block and the second document block.

[0030] In this embodiment, each first document block is sequentially obtained from the starting position of the first document. The block type is determined, if it is a text block, the text content is processed by word segmentation, etc., and then the second document text block containing these segmented words is queried in the second inverted index. If it is a non-text block, its vector features are extracted, and similar second document non-text blocks are found in the second vector index by calculating vector distance, etc. After determining the matching block, for text blocks, a text comparison algorithm is used, such as the longest common subsequence algorithm based on dynamic programming, to determine the differences in words, sentences, etc.; for non-text blocks, the corresponding difference calculation method is used according to its type, such as calculating pixel difference, feature vector difference, etc. to determine the difference.

[0031] It can be understood that retrieval and difference determination in block units break through the limitations of traditional line-level or single-page static comparison affected by layout changes. When the document appears to be split across pages, page order adjustment, etc. layout changes, since attention is paid to the block content itself rather than its position, the true content difference can be accurately identified, avoiding the misjudgment of line breaks, font changes, etc. as content changes due to layout changes, and improving the accuracy of semantic-level difference identification.

[0032] Step S30, traversing each second document block, determining a deletion block in each second document block based on the first index database.

[0033] The deletion block is a document block that exists in the second document but does not appear in the first document, determined by comparison with the first index database.

[0034] From the first block of the second document, each second document block is sequentially obtained. The block type is judged, if it is a text block, the text content is processed, such as word segmentation, and it is queried in the first inverted index whether there is the same or similar text block; if it is a non-text block, its vector feature is extracted, and it is queried in the first vector index whether there is a similar vector feature, if no match is found in the first index database, it is indicated that the second document block is deleted in the first document, and the second document block is marked as a deleted block.

[0035] It can be understood that through this reverse retrieval mode, the deletion of both text blocks and non-text blocks can be accurately identified, avoiding the problem of missing cross-modal deletion information in traditional multi-modal document comparison due to independent processing of single-modal data, ensuring the comprehensiveness of semantic-level difference identification, and making the document comparison result more accurate and complete.

[0036] In step S40, the block difference and the deleted block are written into the difference knowledge graph, and a document comparison report is generated based on the difference knowledge graph.

[0037] The difference knowledge graph is a knowledge graph that represents document difference information in a graphical structure, taking each block in the document as a node and constructing edges between the differences between the blocks, thereby realizing the presentation of the difference situation of the document in a intuitive and structured manner. The document comparison report is a report generated according to the information in the difference knowledge graph, which can include all differences between the first document and the second document, including the location, type, specific content, etc. of the difference, so that the user can clearly understand the change of the document.

[0038] Nodes are respectively created for each block in the first document and the second document, connection edges are created between the nodes corresponding to the block difference associated blocks, and the difference type, difference semantic explanation and difference level of the block difference are bound to the edge attribute of the connection edge, a self-loop edge is created on the node corresponding to the deleted block, and the edge attribute of the self-loop edge is bound to deletion, forming the difference knowledge graph. It should be noted that in the process of writing into the difference knowledge graph, the calling parameters of the knowledge graph tool can be generated through the multi-modal large language model and the prompt word engineering, including the difference type mapping rule and the risk level assignment logic, so as to realize flexible tool calling. Specifically, in the creation of nodes and connection edges and the binding of edge attributes, the multi-modal large language model can generate corresponding tool calling parameters according to the preset prompt word template, for example, for difference type mapping, the multi-modal large language model will generate mapping rules according to the prompt word template to classify the original difference into predefined types, and for risk level assignment, the multi-modal large language model will dynamically generate risk level assignment logic in combination with business rules and prompt word templates to ensure the accuracy and flexibility of risk assessment.

[0039] After the construction of the difference knowledge graph is completed, information is extracted from the difference knowledge graph according to a preset report template and format requirement, and a document comparison report is generated. In the process of generating the document comparison report, the report generation tool calling parameters can also be generated by the multi-modal large language model and the prompt word engineering. Specifically, the information in the difference knowledge graph, such as the difference type, the difference semantic explanation, the difference level, etc., is input into the multi-modal large language model; the multi-modal large language model is guided by the preset prompt word template to generate the calling parameters of the report generation tool, including but not limited to the content, format and logical order of the report, etc., and the calling parameters of the report generation tool are input into the report generation tool to automatically generate a customized document comparison report according to the needs and scenarios of different users, thereby improving the intelligent level and practicality of the document comparison system.

[0040] It should be particularly pointed out that the multi-modal large language model involved in generating the difference knowledge graph and generating the document comparison report in the present example can be the same or different, and is specifically set according to actual needs; the prompt word template involved in generating the difference knowledge graph and generating the document comparison report can also be set according to actual needs, which is not limited herein. It can be understood that the flexibility of parameter generation is realized by using the multi-modal large language model and the prompt word engineering to guide the tool calling of the large model for report generation, so that the document comparison can flexibly adapt to different scenarios and needs, and the intelligent level and practicality of the document comparison are significantly improved.

[0041] It can be understood that the difference knowledge graph structurally integrates scattered multi-modal difference information, and intuitively shows the internal relationship between the differences, solving the problem of scattered difference results and difficulty in comprehensive analysis in the prior art. The document comparison report generated based thereon can clearly and comprehensively present all semantic-level differences, improving the readability and practicality of the document comparison result, facilitating users to quickly and accurately understand the change of the document and make corresponding decisions.

[0042] In the present embodiment, a suitable index database is constructed for different modal document information, solving the problem of fragmented semantic association caused by multi-modal feature space difference and laying a foundation for accurate comparison; bidirectional retrieval and difference determination are performed in units of blocks, effectively overcoming the challenge brought by document layout changes and avoiding the generation of pseudo-differences, thereby significantly improving the accuracy of semantic-level difference identification; the difference information is integrated by the difference knowledge graph and a report is generated, so that the difference result is more clear, comprehensive, easy to understand and analyze. Therefore, the present embodiment improves the accuracy, comprehensiveness and effectiveness of result presentation in semantic-level difference identification.

[0043] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as the above embodiment one can refer to the above introduction, and the subsequent will not be described. On this basis, the first index database includes a text index database and a vector index database; please refer to Figure 2 Step S10: based on the first document, the step of constructing the first index database, including: Step S101, extracting the text block in the first document and identifying the text, clustering the text into a plurality of first semantic text blocks based on projection clustering, establishing a first inverted index and storing it to the text index database, wherein in the inverted index, the keywords of the first semantic text block are mapped to the block identifier of the document block where the first semantic text block is located, the coordinate position of the first semantic text block in the first document and the page number.

[0044] Projection clustering is a clustering algorithm, which can effectively process high-dimensional data by projecting data into a specific subspace and clustering in the subspace. In the present embodiment, it is used to cluster text according to semantic association. The first semantic text block is a set of text with semantic association obtained by projection clustering, which is a semantic unit further subdivided from the text block. The block identifier is a unique identifier assigned to each document block, which is used to accurately distinguish different blocks. The coordinate position is the specific coordinate information of the text block in the document page, which is accurate to the X, Y axis position in the page.

[0045] The document parsing tool is used to extract the text block in the first document, and the OCR recognition technology is used to convert the image text in the text block into editable text. The text is preprocessed, such as removing redundant symbols and unifying formats. The projection clustering algorithm is used to project the preprocessed text into a semantic related subspace, calculate the semantic similarity between the texts, and cluster the texts with high similarity into a class to form a plurality of first semantic text blocks. The keywords are extracted from each first semantic text block, the first inverted index is established, the keywords are associated with the block identifier of the block where the semantic text block is located, the coordinate position in the first document, such as X=100px, Y=200px in the page, and the page number, and finally the inverted index is stored in the text index database. It can be understood that the first semantic text block obtained by projection clustering makes the text processing more consistent with semantic logic, avoiding the semantic fragmentation caused by simply dividing by line or paragraph. The mapping of block identifier, coordinate position and page number in the inverted index retains the semantic association of the text, and can quickly locate the specific position during retrieval, providing more detailed information for subsequent accurate comparison of text differences, further improving the accuracy and efficiency of text modal difference identification.

[0046] In step S102, the block boundary of the non-text block in the first document is determined, a preset feature extraction model is called to perform feature coding on the non-text block in the first document based on the block boundary to generate a first coding vector, a first vector index is established and stored in a vector index database, wherein in the first vector index, the first coding vector is mapped to the block identifier, coordinate position and page number of the non-text block in the first document.

[0047] The block boundary is the boundary range of the non-text block in the document page, which is usually represented in the form of a rectangular box and contains information such as the upper left corner and lower right corner coordinates of the block. The feature extraction model is a preset model for extracting features of the non-text block. It should be noted that the feature extraction model can be a pre-trained vision-language model (Vision-Language Model, VLM), such as CLIP, BLIP-2 or LLaVA model. The feature extraction model generates a 512-dimensional semantic embedding vector by associating the visual features of the non-text block, such as images, charts, signatures, etc. with text semantics. Specifically, the feature extraction model receives image data within the block boundary and outputs a vector representing the semantic content of the block, i.e. the first coding vector. The first coding vector shares the same embedding space as the text modality, thereby achieving cross-modal semantic alignment. The first coding vector is a vector obtained by feature coding of the non-text block, which can represent the core features of the non-text block.

[0048] Through image segmentation technology or document structure analysis tools, the non-text block in the first document is identified, and the block boundary of each non-text block is determined, such as the upper left corner coordinates (x1, y1) and the lower right corner coordinates (x2, y2). According to the type of the non-text block, the non-text data within the block boundary is input into the feature extraction model, which extracts features such as image edges, texture features, chart coordinate axes, data point features, etc. and encodes these features into the first coding vector. A first vector index is established, the first coding vector is associated with the block identifier, coordinate position such as X=300px, Y=400px to X=500px, Y=600px in the page, and the page number, and is stored in the vector index database.

[0049] It can be understood that by determining the boundary of the non-text block, the range of feature extraction is accurate and the interference of irrelevant surrounding content is avoided. The preset feature extraction model can extract key features of different types of non-text blocks, and the first coding vector generated can effectively represent the non-text content. The association of the block identifier, coordinate position and page number in the vector index makes the retrieval of the non-text block more accurate and efficient, and the position information can assist in determining whether the non-text block has a false difference due to a change in position, further improving the reliability of the non-text modal difference identification.

[0050] In this embodiment, the text is clustered into semantic text blocks by projection clustering, and rich position mapping information is added in the inverted index, thereby enhancing the semantic correlation and positioning accuracy of the text modal processing; the non-text block boundary is determined, and a special feature extraction model is used to generate an encoding vector, and a vector index is established in combination with detailed position information, thereby improving the accuracy and retrieval efficiency of non-text modal feature extraction. Therefore, the subsequent block matching and difference identification are more accurate and efficient, the problem of pseudo-difference caused by the fragmentation of multi-modal semantic association and the change of layout in the background technology is solved, the overall performance of multi-modal document comparison is improved, and stronger practicality and innovation are achieved.

[0051] In an embodiment, the step S20 of retrieving, based on the second index database, matching blocks matching the first document block in each second document block of the second document, comprises: In step S201, if the first document block is a text block, a text retrieval tool is called to retrieve the block identifiers of the top N similar semantic text blocks based on the second inverted index, and the block content is obtained based on the block identifiers of the top N similar semantic text blocks.

[0052] The text retrieval tool is a tool for retrieving similar content in a text index, which can quickly find matching text data based on keywords, semantics, etc. When a certain block in the first document is determined to be a text block, the text retrieval tool is started for indexing. The indexing process of the text block can be: extracting semantic features such as keywords, topics, etc. of the text block, calling the text retrieval tool, inputting these semantic features into the text retrieval tool, and the matching engine in the text retrieval tool retrieves based on the second inverted index, calculates the semantic similarity between the text block and each text block in the second document; according to the order from high to low, the top N similar semantic text blocks are selected, and their block identifiers are recorded, wherein the top N similar semantic text blocks are the top N text blocks with the highest semantic similarity to the text block of the first document in the retrieval result, and N is a predetermined positive integer which can be set according to actual needs; according to these block identifiers, the corresponding text block in the second document is located, and its text content is extracted as the block content. It should be noted that in this embodiment, the content and semantic context of the first document block can be analyzed based on a preset multi-modal large language model, and the parameters of the text retrieval tool such as the retrieval range, keyword weight, similarity threshold K value are generated by the multi-modal large language model, and the parameters of the text retrieval tool are bound to the text retrieval tool.

[0053] It can be understood that, by the text retrieval tool based on the second inverted index, the second document text block similar to the first document text block can be quickly and accurately found, and by limiting the top N results, the highly similar content is ensured not to be missed, the retrieval range is controlled, and the retrieval efficiency is improved.

[0054] In step S202, if the block content of the first document block is non-text content, a visual retrieval tool is called to recall the block identifiers of the top N similar vectors based on the second vector index, and the block content is obtained based on the block identifiers of the top N similar vectors.

[0055] The visual retrieval tool is a tool for retrieving similar visual content in the vector index, and can match and find based on the vector features of non-text data such as images and charts. When the block in the first document is non-text content such as images, charts, and signatures, the visual retrieval tool is called for block matching. The process of block matching of non-text blocks can be: obtaining the first encoding vector of the non-text block, calling the visual retrieval tool, and inputting the first encoding vector into the tool; the matching engine in the visual retrieval tool retrieves based on the second vector index, calculates the similarity between the vector and the vector of each non-text block in the second document; according to the order from high to low of the similarity, the top N similar vectors are filtered out, and their block identifiers are recorded; according to the block identifiers, the corresponding non-text blocks in the second document are found, and their contents are extracted as block contents.

[0056] It can be understood that, by the visual retrieval tool based on the second vector index, the non-text content is retrieved, the vector features of the non-text block are fully utilized, the similar second document non-text block can be efficiently and accurately found, the limitation of the top N similar vectors improves the efficiency while ensuring the retrieval quality, provides reliable comparison content for the difference judgment of the non-text block, and is beneficial to accurately identifying the visual feature difference of the non-text block.

[0057] In step S203, the block content is input into a preset difference judgment model to obtain a difference type, wherein the difference type includes addition, modification, movement, and image replacement.

[0058] The difference determination model is a pre-trained model for determining the content difference type of two document blocks. It can analyze multi-modal data such as text and images, and can be a deep learning model that integrates text processing and image processing capabilities, such as a visual language model. In this embodiment, the difference determination model can be based on a multi-modal large language model such as GPT-4V, LLaVA, etc. The model outputs structured difference types guided by a pre-set prompt template. The specific process of the difference determination model for model type determination can include: inputting the content of the first document block to be compared and the matching document block content into the difference determination model and inputting the prompt. For example, the prompt can be: "Please compare the following two block contents: [content of the first document block] and [content of the matching document block]; determine the difference type: addition, modification, movement, or image replacement, where content change only is movement; image complete replacement is image replacement." The difference determination model outputs structured difference labels based on semantic analysis. In this embodiment, the difference types include addition, modification, movement, and image replacement. Specifically, addition refers to a content block that exists in the first document but does not exist in the second document; modification refers to a change in content between the first document block and the corresponding block in the second document, but not complete replacement; movement refers to a change in the location of the block content in the document, such as page number or coordinate; image replacement refers to a non-text content in the first document, such as an image, that is completely different from the corresponding content in the second document.

[0059] After obtaining the first N block contents similar to the first document block in the second document, these block contents are input into the difference determination model together with the corresponding block content of the first document. The model first determines whether the block of the first document exists in the second document; if not, it is determined as addition; if it exists, it further compares whether the content has changed; if the content has changed, it is determined as modification; if the content is completely the same but the position is different, it is determined as movement; for non-text blocks, if the content is completely different, it is determined as image replacement; finally, the determined difference type is output.

[0060] It can be understood that the pre-set difference determination model can comprehensively and objectively analyze the block content and accurately determine different difference types, making the document comparison result more detailed and clear. The clear difference type division helps users quickly understand the changes of the document, meets the fine-grained needs of difference information in different scenarios, and improves the practical value of the document comparison result.

[0061] In this embodiment, by using corresponding retrieval tools and indexes for text and non-text blocks respectively to recall the top N similar blocks, the relevance and efficiency of retrieval are improved, the accuracy of matching blocks is ensured, the problems of low retrieval efficiency and ambiguous difference types in traditional comparison techniques are solved, and the precision and practicality of multi-modal document comparison are significantly improved.

[0062] In an embodiment, the step S30 of determining the deletion block in each second document block based on the first index database comprises: In step S301, it is determined whether there is a block matching the second document block in each first document block based on the first index database.

[0063] The first index database is an index database constructed based on the first document, which contains a first inverted index corresponding to the text blocks of the first document and a first vector index corresponding to the non-text blocks, and is used for quickly retrieving block information in the first document.

[0064] Each block of the second document is traversed, and the type of the block is first determined. If it is a text block, the keywords and semantic features of the text content are extracted, and then the first inverted index of the first index database is searched to check whether there is a semantically similar text block in the first document. If it is a non-text block, its feature vector is extracted, and the first vector index of the first index database is searched to calculate the similarity between the feature vector and the feature vector of the non-text block in the first document, and to determine whether there is a similar non-text block.

[0065] In step S302, if there is no block matching the second document block, the second document block is determined as a deletion block.

[0066] When it is found that there is no block matching the second document block in the first document, it means that there is no corresponding content in the first document for the second document block, and therefore it can be determined as a deletion block.

[0067] When there is a matching block, further difference comparison can be performed through a visual language model. The specific comparison process can refer to steps S201-S203, which will not be described here. Further, in an embodiment, the difference obtained by comparison can be compared with the existing relevant difference records in the difference knowledge graph. If there is a new difference or the original record needs to be corrected, an update operation on the knowledge graph is triggered, and the new difference information is added to the knowledge graph or the original record is modified.

[0068] In this embodiment, the efficient and accurate determination of the deletion block is realized by querying the matching block based on the first index database, the problem of difficult comprehensive identification of the deletion content in the traditional technology is solved, the process of deletion block identification and matching block deep difference analysis in multi-modal document comparison is improved, the accuracy, comprehensiveness and intelligent level of the document comparison are improved, and the accuracy of the document comparison result is improved.

[0069] Based on the first and / or second embodiments of the present application, in the third embodiment of the present application, the same or similar contents as the above embodiment one and / or embodiment two can be referred to the above introduction, and the subsequent will not be described in detail. On this basis, please refer to Figure 3 , before the step S40 of writing the block difference and the deletion block into the difference knowledge graph, further comprising: Step S01, creating nodes for each block in the first document and the second document respectively, storing the block identifier, the coordinate position in the corresponding document, the page number and the content type of the document block of each block.

[0070] The node is a basic unit for representing an independent document block in the difference knowledge graph. The content type refers to the category of the content contained in the document block, such as text, image, chart, signature, etc.

[0071] Traverse all document blocks of the second document, for each block, the system automatically generates a unique node; extract the block identifier, the coordinate position in the second document, the page number and the content type of the block, and store these information into the node. Traverse all document blocks of the first document in the same way, create nodes for each block and store the corresponding information, thereby forming a difference knowledge graph including the basic node data of all blocks in the second document and the first document.

[0072] Step S40: the step of writing the block difference and the deletion block into the difference knowledge graph, comprising: Step S401, determining the difference level corresponding to the block difference, creating a connection edge between the nodes corresponding to the block difference associated blocks, and binding the difference type, difference semantic explanation and difference level of the block difference to the edge attribute of the connection edge.

[0073] The difference level is an index for measuring the importance of the block difference, which can be divided into different levels according to the influence degree of the difference on the overall semantics and function of the document, such as slight, general, important, serious, etc. The connection edge is a line in the difference knowledge graph, which is used to connect two related nodes and represent the association relationship between the nodes. The edge attribute is the information carried on the connection edge, which is used to describe the specific content of the association between the two nodes, at least including the difference type.

[0074] For each identified block difference, determine its difference level according to preset criteria, such as whether the difference affects the core logic of the document, whether it involves key data, and the like; find the second document block node and the first document block node involved in the difference in the difference knowledge graph; create a connection edge between the two nodes; extract the difference type of the block difference, generate a difference semantic explanation, and store the difference type, difference semantic explanation, and difference level in the edge attribute of the connection edge. It can be understood that by determining the difference level and binding various difference information to the edge attribute of the connection edge, the difference knowledge graph not only shows the differences between the blocks, but also details the nature, importance, and semantic meaning of the differences, which helps users quickly understand the key information of the differences and distinguish the importance of different differences, improves the efficiency of grasping the document change situation, and also enhances the information carrying capacity and practical value of the difference knowledge graph.

[0075] In step S402, a self-loop edge is created on the node corresponding to the deleted block, and the edge attribute of the self-loop edge is bound to deletion.

[0076] The self-loop edge is an edge in the difference knowledge graph that starts from a node and returns to the node, used to represent the special state or relationship of the block corresponding to the node itself, and in this embodiment, it is used to represent that the block is deleted.

[0077] After identifying the deleted block, find the node corresponding to the deleted block in the difference knowledge graph; create a self-loop edge on the node, i.e., the start and end points of the edge are both the node; set the edge attribute of the self-loop edge to deletion to clearly identify the deletion state of the block. It can be understood that the deleted block is marked by the self-loop edge and its edge attribute, so that users can quickly locate all deleted blocks, avoiding the deletion information being missed or confused in a complex knowledge graph, and ensuring the clarity and accuracy of the difference information display.

[0078] In an embodiment, step S401 of determining the difference level corresponding to the block difference comprises: In step S4011, input the blocks associated with the block difference in the first document and the second document into a preset semantic analysis model to obtain a semantic difference degree, and determine a visual difference degree and a structural difference degree between the blocks associated with the block difference, wherein the semantic difference degree represents the semantic conflict degree between the blocks associated with the block difference, the visual difference degree represents the appearance feature difference between the blocks associated with the block difference, and the structural difference degree represents the physical layout difference of the blocks associated with the block difference in the first document and the second document.

[0079] The semantic analysis model is a preset model for analyzing semantic differences of text or cross-modal content, which can be constructed based on a large language model or a visual language model, and can understand the meaning of text and semantic information carried by non-text content. The semantic difference degree is used to quantitatively represent the degree of conflict or difference between two associated blocks in the semantic level, and the value range is usually 0 to 1, for example, 0 represents complete semantic consistency, and 1 represents complete semantic conflict. The visual difference degree is used to quantitatively represent the difference between two associated blocks in appearance features such as color, shape, and texture. The visual difference degree of the non-text block needs to be calculated. It can be understood that the visual difference degree of the text block does not affect the semantics due to the appearance difference, so the visual difference degree is preset to 0. The structural difference degree is used to quantitatively represent the difference between two associated blocks in the physical layout of the document, such as coordinate position, page number, and relative position to other blocks, and the value range is 0 to 1.

[0080] The second document block associated with the block difference and the first document block are extracted; the two blocks are input into a preset semantic analysis model, the model performs semantic analysis on the text content, and analyzes the semantic information carried by the non-text content, and outputs the semantic difference degree; for the visual difference degree, if it is a non-text block, the visual feature difference value is obtained by comparing the image features, such as calculating the feature vector distance, as the visual difference degree; if it is a text block, the visual difference degree is directly set to 0; for the structural difference degree, the coordinate position, page number and relative position relationship with the surrounding blocks of the two blocks are extracted, and the structural difference degree is obtained by a preset algorithm, such as calculating the position coordinate deviation rate, page number difference weight and the like.

[0081] In step S4012, the semantic difference degree, the visual difference degree and the structural difference degree are weighted to obtain a three-dimensional score corresponding to the block difference, and the difference level is determined based on the three-dimensional score.

[0082] In this embodiment, different weights can be given to the importance of the three difference degrees to calculate the block difference, and the sum of the weights is 1, such as a semantic difference degree weight of 0.6, a visual difference degree weight of 0.2, and a structural difference degree weight of 0.2.

[0083] Detailed explanation and layout explanation: Due to the different influence degrees of semantic, visual and structural differences on the overall difference of the block, the difference degrees of the three dimensions can be integrated into a three-dimensional score through weighted processing. The weights can be set according to the actual application scene, for example, in contract document comparison, the weight of semantic difference is higher; in design drawing comparison, the weight of visual difference may be higher. In this embodiment, the weights of the preset semantic difference degree, visual difference degree and structural difference degree are obtained, the semantic difference degree, visual difference degree and structural difference degree are multiplied by the corresponding weights respectively, and then the products are added to obtain a three-dimensional score; according to the interval where the three-dimensional score is located, the difference level is determined. It can be understood that the weighted processing reasonably integrates the difference degrees of the three dimensions according to the importance, avoiding the result deviation caused by the equal treatment of the dimensions. The determination of the three-dimensional score and the difference level converts the complex multi-dimensional difference information into a simple level identifier, so that the user can quickly grasp the importance of the difference without analyzing the specific difference data, thereby improving the practicability and decision efficiency of the difference information.

[0084] In an embodiment, the step S40 of generating a document comparison report based on the difference knowledge graph comprises: Step S403, in response to the natural language question input by the user, retrieving and generating a difference position evidence chain in the difference knowledge graph based on the natural language question, and generating a question answer based on the difference position evidence chain.

[0085] The natural language question is a question about the document difference in the form of daily language proposed by the user, for example, "What modifications are there in the price clause of the contract?" and "Is there any change in the chart on page 3?" etc. The difference position evidence chain is a series of related chains of difference information retrieved in the difference knowledge graph and related to the user's question, including block identifier, coordinate position, page number, difference type, difference level, etc., forming a complete evidence chain.

[0086] The natural language question input by the user through the interactive interface is received, and the multi-modal large language model such as GPT-4V analyzes the user's natural language question to identify the key information in the question; according to the key information, a knowledge graph query instruction is generated automatically, and the difference knowledge graph is retrieved to match the related nodes, which are called target nodes below, and the edge attributes of the associated edges of the target nodes are extracted; the multi-modal large language model integrates the query results, that is, organizes the target nodes and the edge attributes of the associated edges into a difference position evidence chain in a logical order; the multi-modal large language model converts the evidence chain into a natural language to generate a question answer that conforms to the user's understanding habit. It can be understood that the difference position evidence chain ensures the accuracy and traceability of the answer, so that the user not only gets the answer, but also knows the source of the answer, so that the user can quickly obtain the target difference information, avoid manual screening in a large amount of difference data, and improve the information acquisition efficiency.

[0087] At step S404, the annotation tool is invoked to superimpose the annotation layer at the difference block position in the first document and the deletion block position in the second document to obtain an annotated document.

[0088] The annotation tool is a software module for adding annotation information in a document. The difference identification and description text can be superimposed and displayed without changing the original content of the document. The annotation layer is a transparent layer superimposed on the original content of the document, used to display difference information, including difference type identification and brief description. The annotated document is a collection of the first document and the second document superimposed with the annotation layer, which intuitively displays the position and basic type of the difference.

[0089] In this embodiment, the annotation tool can be invoked by the multi-modal large language model. The specific process can be as follows: the multi-modal large language model determines the annotation rules according to the difference type, risk level, etc.; the multi-modal large language model dynamically generates the calling parameters required by the annotation tool, including but not limited to the tool name, specific annotation position information such as the page number where the difference is located, annotation content, annotation color, etc.; performing the annotation operation: after receiving the parameterized instructions generated by the multi-modal large language model, the annotation tool performs the corresponding operation according to the instruction content, superimposes the annotation at the specified position of the document, completes the annotation process, for example, superimposes the corresponding annotation layer in the first document according to the coordinates and page number of the difference block, and superimposes the deletion annotation layer at the original position of the deletion block in the second document; after completing the annotation, the processed first document and second document are integrated into an annotated document. It can be understood that the difference is directly marked in the document through the annotation layer, so that the user can quickly locate the difference position and identify the difference type at a glance, solving the problem of abstract description of difference position in traditional text reports and difficulty in corresponding to the specific position of the document. The annotation layer is separated from the original document, which not only retains the originality of the document but also clearly presents the difference information, improving the convenience of difference viewing.

[0090] At step S405, the annotated document and the answer to the question are merged to obtain a document comparison report.

[0091] The annotated document and the answer content of the user question are integrated to form a document comparison report with clear structure. The report usually includes an annotated document preview area and a question answer area. The user can view the annotated document in the preview area and view the detailed explanation of the question in the answer area. The document comparison report combines visual information and textual answers to meet different acquisition needs of users for difference information, improving the comprehensive utility of the report.

[0092] In this embodiment, personalized and accurate information interaction is realized by responding to user natural language questions and generating evidence chains and answers; the visualized presentation of differences is realized by superimposing the annotation layer through the annotation tool; the comprehensive and easy-to-use comparison report is formed by merging the annotated document and the question and answer, thereby improving the usability of the document comparison result and the user experience, enabling the user to obtain and understand the difference information of the multi-modal document in a natural and efficient manner, solving the problems of poor interactivity, scattered information and lack of intuition of the traditional report, and improving the practicality of multi-modal document comparison.

[0093] By way of example, in order to facilitate the understanding of the implementation process of the multi-modal document comparison method obtained after the above embodiment one, please refer to Figure 4 , Figure 4 A schematic diagram of the brief process of the multi-modal document comparison method is provided, specifically: First, task initialization and strategy setting. After inputting the new version (version A), the old version (version B) file, the user question and the resource limit, the system will generate a session ID (Identifier, identifier), parse and load the user and system configuration, including the default loop step number, the threshold default value, the user question, the resource limit, the tool registry and the parameter constraint, etc., and initialize the working memory and the tool invocation log (TIL, Tool Invocation Log) for this session at the same time, which is used to record each tool invocation and the evidence returned thereby, and finally output the session object.

[0094] Input receiving and legality verification. After the system receives the session object and the file stream, SHA-256 (Secure Hash Algorithm 256-bit, secure hash algorithm 256-bit) is calculated. If the file is encrypted, damaged or fails to read, an error will be thrown and the process will be exited; if the file can be normally read, it will be loaded into the memory, and the page number, size and other metadata will be extracted, and the document object will be output.

[0095] Enter the unified preprocessing stage. After inputting the document object, each page is rendered into a 300DPI (Dots Per Inch, dots per inch) PNG (Portable Network Graphics, portable network graphics) image, binarization and skew correction operations are performed through OpenCV, and page-level ROIs (Region of Interest, region of interest) are generated, and finally the rendered result and the ROI result are output.

[0096] The pre-indexing stage is divided into two parts: text indexing and visual indexing. In the text indexing process, A and B documents are operated independently. Optical character recognition is used to extract text, and projection clustering is used to obtain text blocks. The inverted index is calculated and the text index database is output. In the visual indexing process, all pictures, charts, and signatures are encoded using a visual language model based on feature extraction, resulting in a 512-dimensional vector. The vector database is written and the index is calculated, and the image index database is output. Then, the system records the index id and corresponding block position obtained from the above index processing to form a layout table record.

[0097] The main loop first traverses the A document. After inputting the A document and the B document, the agent (note that in this embodiment, the agent is an artificial intelligence application based on a multi-modal large language model, which uses prompt word engineering to guide self-service decision-making to call tools) calls the document reading tool to open the pth page of the A document. For each block of content on this page, the corresponding block in the B document is retrieved. Specifically, the agent reads the blocked results from the index and layout records, generates a text retrieval tool call to recall the top K relevant blocks of the B document, and generates an image retrieval tool call if the block to be processed in the current A document contains an image. The top K matching results are recorded according to user configuration. Then, the retrieved document blocks and image blocks are input into the visual language model for difference analysis. The large model generates a structured output based on a pre-set prompt word template, records the blocks as nodes and the differences between the blocks as edges, and calls the knowledge graph recording tool to record the difference knowledge graph. If a match is found, the visual language model compares the differences and writes them into the knowledge graph. If no match is found, the new content is recorded into the knowledge graph, and the final output is the difference knowledge graph.

[0098] The main loop then traverses the B document to verify and fill in the differences. The agent calls the document reading tool to open the pth page of the B document, and retrieves the corresponding block in the A document for each block of content on this page. If a match is found, the visual language model compares the differences and determines whether to update the difference record in the knowledge graph. If no match is found, the deleted content is recorded into the knowledge graph, and the updated difference knowledge graph is output.

[0099] In the difference scoring and explanation stage, the system generates scores and analysis explanations in three aspects: semantic difference, visual difference, and structural difference, for each difference, through a visual language model based on semantic analysis. The final difference score is a weighted value, and the weighting formula is λ1•semantic difference + λ2•visual difference + λ3•structural difference. Based on the final score, the difference level is determined, for example, the top 10% is considered as a serious difference, the middle 70% is considered as an ordinary difference, and the last 20% is considered as a minor difference. The difference list is output.

[0100] Finally, the annotated PDF and the final report are generated. The agent calls the annotation tool to fill the difference score and analysis explanation of each difference block into the PDF annotation area; meanwhile, the knowledge graph retrieval tool answers the user's initial question, and the answer and the analysis of all difference blocks are combined to form the final report. The annotated PDF and the final report are output.

[0101] It should be noted that the above examples are only for understanding the present application and do not constitute a limitation on the multi-modal document comparison method of the present application. More forms of simple transformation based on this technical concept are within the protection scope of the present application.

[0102] The present application also provides a multi-modal document comparison system, which refers to Figure 5 , and the multi-modal document comparison system comprises: A pre-indexing module 10 is configured to construct a first index database based on a first document and a second index database based on a second document. The first index database comprises a first inverted index corresponding to a text block of the first document and a first vector index corresponding to a non-text block of the first document. The second index database comprises a second inverted index corresponding to a text block of the second document and a second vector index corresponding to a non-text block of the second document. A traversal module 20 is configured to traverse each first document block of the first document, retrieve a matching block matching the first document block in each second document block of the second document based on the second index database, and determine a block difference between the matching block and the first document block. The traversal module 20 is further configured to traverse each second document block, and determine a deletion block in each second document block based on the first index database. A document comparison module 30 is configured to write the block difference and the deletion block into a difference knowledge graph, and generate a document comparison report based on the difference knowledge graph.

[0103] Optionally, the first index database comprises a character index database and a vector index database. The pre-indexing module 10 is configured to: extract a text block from the first document and identify text characters, cluster the text characters into a plurality of first semantic text blocks based on projection clustering, establish a first inverted index and store it into the character index database, wherein the keywords of the first semantic text blocks are mapped to the block identifier of the document block where the first semantic text blocks are located, the coordinate position of the first semantic text blocks in the first document, and the page number in the inverted index. determining a block boundary of a non-text block in the first document, calling a preset feature extraction model to perform feature coding on the non-text block in the first document based on the block boundary to generate a first coding vector, establishing a first vector index and storing the first vector index to the vector index database, wherein the first coding vector is mapped to a block identifier, a coordinate position and a page number of the non-text block in the first document in the first vector index.

[0104] Optionally, the traversal module 20 is configured to: if the first document block is a text block, calling a text retrieval tool to recall block identifiers of top N similar semantic text blocks based on the second inverted index, and obtaining block content based on the block identifiers of the top N similar semantic text blocks; if the block content of the first document block is non-text content, calling a visual retrieval tool to recall block identifiers of top N similar vectors based on the second vector index, and obtaining the block content based on the block identifiers of the top N similar vectors; inputting the block content into a preset difference determination model to obtain a difference type, wherein the difference type includes addition, modification, movement and image replacement.

[0105] Optionally, the traversal module 20 is configured to: querying whether there is a block matching the second document block in each of the first document blocks based on the first index database; if there is no block matching the second document block, determining that the second document block is a deletion block.

[0106] Optionally, the apparatus further comprises a construction module configured to: creating a node for each block in the first document and the second document respectively, and storing a block identifier of the each block, a coordinate position in a corresponding document, a page number and a content type of the document block; The document comparison module 30 is configured to: determining a difference level corresponding to the block difference, creating a connection edge between nodes corresponding to the block difference, and binding a difference type, a difference semantic explanation and the difference level of the block difference to an edge attribute of the connection edge; creating a self-loop edge on the node corresponding to the deletion block, and binding an edge attribute of the self-loop edge as deletion.

[0107] Optionally, the document comparison module 30 is further configured to: inputting the blocks associated with the block difference in the first document and the second document into a preset semantic analysis model to obtain a semantic difference degree, determining a visual difference degree and a structural difference degree between the blocks associated with the block difference, wherein the semantic difference degree represents a semantic conflict degree between the blocks associated with the block difference, the visual difference degree represents an appearance feature difference between the blocks associated with the block difference, and the structural difference degree represents a physical layout difference of the blocks associated with the block difference in the first document and the second document; performing weighted processing on the semantic difference degree, the visual difference degree and the structural difference degree to obtain a three-dimensional score corresponding to the block difference, and determining a difference level based on the three-dimensional score.

[0108] Optionally, the document comparison module 30 is further configured to: in response to a natural language question input by a user, retrieve and generate a difference location evidence chain in the difference knowledge graph based on the natural language question, and generate a question answer based on the difference location evidence chain; invoke a comment tool to superimpose a comment layer at a difference block position in the first document and a deletion block position in the second document to obtain a comment document; merge the comment document and the question answer to obtain a document comparison report.

[0109] The multi-modal document comparison system provided in the present application adopts the multi-modal document comparison method in the above embodiments, and can solve the technical problem that the existing multi-modal document comparison technology has low semantic level difference recognition accuracy, which affects the result of multi-modal document comparison. Compared with the prior art, the multi-modal document comparison system provided in the present application has the same beneficial effects as the multi-modal document comparison method provided in the above embodiments, and other technical features in the multi-modal document comparison system are the same as the features disclosed in the above embodiments, which will not be repeated here.

[0110] The present application provides an electronic device, which comprises at least one processor and a memory connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the multi-modal document comparison method in the above embodiment one.

[0111] Reference will be made to the following Figure 6The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of this application. The electronic devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0112] like Figure 6 As shown, the electronic device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. The communication device 1009 allows the electronic device to communicate wirelessly or wiredly with other devices to exchange data. Although the diagrams show electronic devices with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented alternatively.

[0113] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network through a communication device, or installed from the storage device 1003, or installed from the read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiments disclosed in the present application are executed.

[0114] The electronic device provided by the present application adopts the multi-modal document comparison method in the above-mentioned embodiments, which can solve the technical problem that the existing multi-modal document comparison technology has low recognition accuracy for semantic level differences, resulting in the result of multi-modal document comparison being affected. Compared with the prior art, the beneficial effects of the electronic device provided by the present application are the same as those of the multi-modal document comparison method provided by the above-mentioned embodiments, and other technical features in the electronic device are the same as those disclosed in the previous embodiment method, which will not be repeated here.

[0115] It should be understood that parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above-mentioned embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0116] The above is merely specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0117] The present application provides a computer readable storage medium having stored thereon computer readable program instructions (i.e. computer program) for executing the multi-modal document comparison method in the above-mentioned embodiments.

[0118] The computer readable storage medium provided in the present application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system or device, or any combination thereof. More specific examples of the computer readable storage medium can include, but are not limited to, an electric connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), an optical fiber, a portable compact disk read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present embodiment, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer readable storage medium can be transmitted in any suitable medium, including but not limited to an electric wire, an optical cable, an RF (Radio Frequency), and the like, or any suitable combination thereof.

[0119] The above computer readable storage medium can be contained in an electronic device, or can exist separately without being assembled into an electronic device.

[0120] The above computer readable storage medium carries one or more programs, which, when executed by an electronic device, cause the electronic device to implement various embodiments of the multi-modal document comparison method as described above.

[0121] Computer program code for carrying out operations of the present application can be written in one or more programming languages or combinations of languages including object oriented programming languages such as Java, Smalltalk, C++ or conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code can execute entirely on a user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0122] The flow and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flow and block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or combinations of hardware and software.

[0123] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the name of the module does not constitute a limitation on the unit itself.

[0124] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer programs) for executing the above-mentioned multi-modal document comparison method, and can solve the technical problem that the existing multi-modal document comparison technology has low recognition accuracy of semantic level differences, which affects the result of multi-modal document comparison. Compared with the prior art, the beneficial effects of the computer readable storage medium provided by the present application are the same as those of the multi-modal document comparison method provided by the above-mentioned embodiments, which will not be repeated here.

[0125] The present application also provides a computer program product comprising a computer program, which, when executed by a processor, implements the steps of the multi-modal document comparison method as described above.

[0126] The computer program product provided by the present application can solve the technical problem that the existing multi-modal document comparison technology has low recognition accuracy of semantic level differences, which affects the result of multi-modal document comparison. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as those of the multi-modal document comparison method provided by the above-mentioned embodiments, which will not be repeated here.

[0127] The above only describes some embodiments of the present application, and does not limit the patent scope of the present application, and any equivalent structural transformation made by using the content of the present application specification and drawings, or direct / indirect application in other related technical fields is included in the patent protection scope of the present application.

Claims

1. A multi-modal document alignment method, characterized by, The multi-modal document comparison method comprises: building a first index database based on a first document and a second index database based on a second document, wherein the first index database comprises first inverted indexes corresponding to text blocks in the first document and first vector indexes corresponding to non-text blocks in the first document, and the second index database comprises second inverted indexes corresponding to text blocks in the second document and second vector indexes corresponding to non-text blocks in the second document; traversing each first document block in the first document, retrieving matching blocks matching the first document block in each second document block in the second document based on the second index database, and determining block differences between the matching blocks and the first document block; traversing each second document block, determining a deletion block in each second document block based on the first index database; writing the block differences and the deletion block into a difference knowledge graph, and generating a document comparison report based on the difference knowledge graph.

2. The method of claim 1, wherein, The first index database comprises a character index database and a vector index database; The step of building the first index database based on the first document comprises: extracting text blocks in the first document and identifying text characters, clustering the text characters into a plurality of first semantic text blocks based on projection clustering, establishing first inverted indexes and storing them into the character index database, wherein in the inverted indexes, keywords of the first semantic text blocks are mapped to block identifiers of document blocks where the first semantic text blocks are located, coordinate positions of the first semantic text blocks in the first document, and page numbers; determining block boundaries of non-text blocks in the first document, calling a preset feature extraction model to perform feature coding on the non-text blocks in the first document based on the block boundaries to generate first coding vectors, establishing first vector indexes and storing them into the vector index database, wherein in the first vector indexes, the first coding vectors are mapped to block identifiers, coordinate positions, and page numbers of the non-text blocks in the first document.

3. The method of claim 1, wherein, The step of retrieving matching blocks matching the first document block in each second document block in the second document based on the second index database comprises: if the first document block is a text block, calling a text retrieval tool to recall block identifiers of the top N similar semantic text blocks based on the second inverted indexes, and obtaining block content based on the block identifiers of the top N similar semantic text blocks; if the block content of the first document block is non-text content, calling a visual retrieval tool to recall block identifiers of the top N similar vectors based on the second vector indexes, and obtaining the block content based on the block identifiers of the top N similar vectors; inputting the block content into a preset difference determination model to obtain a difference type, wherein the difference type comprises addition, modification, movement, and image replacement.

4. The method of claim 1, wherein, The step of determining a deletion block in each second document block based on the first index database comprises: querying, based on the first index database, whether there is a block matching the second document block in each of the first document blocks; if there is no block matching the second document block, determining the second document block as a deletion block.

5. The method of claim 1, wherein, Before the step of writing the block difference and the deletion block into the difference knowledge graph, the method further comprises: creating nodes respectively for each block in the first document and the second document, and storing block identifiers, coordinate positions in corresponding documents, page numbers, and content types of the document blocks; the step of writing the block difference and the deletion block into the difference knowledge graph comprises: determining a difference level corresponding to the block difference, creating a connection edge between nodes corresponding to blocks associated with the block difference, and binding the difference type, difference semantic interpretation, and the difference level of the block difference to the edge attribute of the connection edge; creating a self-loop edge on the node corresponding to the deletion block, and binding the edge attribute of the self-loop edge to deletion.

6. The method of claim 5, wherein, The step of determining the difference level corresponding to the block difference comprises: inputting blocks associated with the block difference in the first document and the second document into a preset semantic analysis model to obtain a semantic difference degree, determining a visual difference degree and a structural difference degree between blocks associated with the block difference, wherein the semantic difference degree represents the semantic conflict degree between blocks associated with the block difference, the visual difference degree represents the appearance feature difference between blocks associated with the block difference, and the structural difference degree represents the physical layout difference of blocks associated with the block difference in the first document and the second document; weighting the semantic difference degree, the visual difference degree, and the structural difference degree to obtain a three-dimensional score corresponding to the block difference, and determining a difference level based on the three-dimensional score.

7. The method of any one of claims 1 to 6, wherein, The step of generating a document comparison report based on the difference knowledge graph comprises: in response to a natural language question input by a user, retrieving and generating a difference location evidence chain based on the natural language question in the difference knowledge graph, generating a question answer based on the difference location evidence chain; calling a comment tool to superimpose a comment layer on the difference block position in the first document and a comment layer on the deletion block position in the second document to obtain a comment document; merging the comment document and the question answer to obtain a document comparison report.

8. A multi-modal document comparison system, characterized by, The multi-modal document comparison system comprises: a pre-indexing module configured to construct a first index database based on a first document and a second index database based on a second document, wherein the first index database comprises first inverted indexes corresponding to text blocks in the first document and first vector indexes corresponding to non-text blocks in the first document, and the second index database comprises second inverted indexes corresponding to text blocks in the second document and second vector indexes corresponding to non-text blocks in the second document; The traversing module is configured to traverse each first document block in the first document, retrieve a matching block matching the first document block in each second document block of the second document based on the second index database, and determine a block difference between the matching block and the first document block; The traversing module is further configured to traverse each second document block, and determine a deletion block in each second document block based on the first index database; The document comparison module is configured to write the block difference and the deletion block into a difference knowledge graph, and generate a document comparison report based on the difference knowledge graph.

9. A storage medium, characterized by The storage medium is a computer-readable storage medium, and the storage medium stores a computer program. The computer program is executed by the processor to implement the steps of the multi-modal document comparison method in any one of claims 1 to 7.

10. A computer program product, characterised in that, The computer program product includes a computer program. The computer program is executed by the processor to implement the steps of the multi-modal document comparison method in any one of claims 1 to 7.

Images