A multi-modal based document analysis method and apparatus

Abstract

The application provides a multi-modal based document analysis method and device, relates to the field of unstructured data processing, and comprises the following steps: performing multi-modal analysis on a received unstructured document to obtain document content and corresponding content labels; performing tree structure construction on the document content and the corresponding content labels according to a preset financial system document structure to obtain a directory structure tree; converting the directory structure tree into a knowledge graph, and performing document content retrieval based on the knowledge graph. The application can conveniently and efficiently analyze system documents and provide a convenient and accurate system retrieval approach.

Description

Technical Field

[0001] This application relates to the field of unstructured data processing, specifically a multimodal document analysis method and apparatus. Background Technology

[0002] In the daily operations of banks, employees frequently need to develop processes or check the compliance of work matters, requiring them to consult relevant regulations. Banks possess a large number of policy documents; a single policy may have obsolete versions, current versions, and often updated versions. Furthermore, a single policy may contain multiple explanatory appendices, and the main text may only explain the policy's content, with the main text itself sometimes appearing in the main text or appendices. Therefore, when users need to locate a specific clause, they must first locate the policy document, then the clause, which can lead to situations with similar policy names, multiple versions of the policy, or a single policy containing numerous clauses. Accurately locating the desired clause often consumes significant time. Therefore, there is an urgent need to parse policy documents and provide a convenient and accurate retrieval method.

[0003] This section is intended to provide background or context for the embodiments of the invention set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section. Summary of the Invention

[0004] To address the problems in the existing technology, this application provides a multimodal document analysis method and apparatus that can conveniently and efficiently parse institutional documents and provide a convenient and accurate institutional retrieval method.

[0005] To solve the above-mentioned technical problems, this application provides the following technical solution: Firstly, this application provides a multimodal document analysis method, including: The received unstructured document is parsed using multimodal methods to obtain the document content and its corresponding content tags; Based on the preset financial system document structure, a tree structure is constructed for the document content and corresponding content tags to obtain a directory structure tree; The directory structure tree is converted into a knowledge graph, and document content retrieval is performed based on the knowledge graph.

[0006] Furthermore, the step of performing multimodal parsing on the received unstructured document to obtain the document content and corresponding content tags includes: Multimodal feature extraction is performed on the unstructured document based on its visual layout and semantic content to obtain structural and semantic features. Based on the document's visual layout, cross-modal association is performed on the structural features and semantic features to obtain structural-semantic association relationships; Based on the structural features, the semantic features, and the structural semantic relationships, an overall document structure is generated, and tagged structured data is output; wherein, the tagged structured data includes the document content and corresponding content tags.

[0007] Furthermore, the step of constructing a tree structure for the document content and corresponding content tags according to a preset financial system document structure to obtain a directory structure tree includes: Based on the structure of the financial policy document, the hierarchical relationship between the content tags and the parent-child node association are identified to obtain a tree framework; wherein, the tree framework includes parent nodes and child nodes; Content tags with the same parent node are grouped into the same level, and the content tags in the same level are arranged according to the document content. Based on the correspondence between the content tags and the document content, the document content is written into the child nodes corresponding to the content tags to obtain the directory structure tree.

[0008] Furthermore, the step of converting the directory structure tree into a knowledge graph includes: The nodes in the directory structure tree are identified as entities in the knowledge graph; The relationships of the knowledge graph are constructed based on the tree framework; A graph structure is constructed from the entities and their relationships to obtain a network graph; The network graph is rendered to obtain a visualized knowledge graph.

[0009] Furthermore, the document content retrieval based on the knowledge graph includes: The retrieved search requirements information is parsed into structured query statements; The structured query statement is used to locate related entities and relationship paths in the knowledge graph to obtain matching information blocks; The matched information blocks are subjected to result reasoning and optimization, and the returned results are obtained to obtain the search content corresponding to the search requirements.

[0010] Furthermore, the aforementioned multimodal document analysis method also includes: The directory structure tree and the knowledge graph are reconstructed based on the received retrieval error information; The reconstructed directory structure tree and knowledge graph were used to analyze and resolve the causes of errors.

[0011] Secondly, this application provides a multimodal document analysis device, comprising: The multimodal parsing unit is used to perform multimodal parsing on the received unstructured document to obtain the document content and the corresponding content tags; The tree structure construction unit is used to construct a tree structure of the document content and corresponding content tags according to the preset financial system document structure, so as to obtain a directory structure tree. The document content retrieval unit is used to convert the directory structure tree into a knowledge graph and perform document content retrieval based on the knowledge graph.

[0012] Thirdly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the multimodal document analysis method.

[0013] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multimodal document analysis method.

[0014] Fifthly, this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the multimodal document analysis method.

[0015] To address the problems in existing technologies, this application provides a multimodal document analysis method and apparatus capable of receiving user-uploaded financial policy documents and comparing updated policies with existing policies. Following banking policy writing rules, it uses multimodal document analysis and assembly technology to disassemble and store policy clauses, including disassembling the main text by chapter and paragraph, constructing a directory structure tree, fine-tuning automatic numbering and inappropriate line breaks, and storing and managing basic policy information (including issuing agency, issuance date, etc.). Furthermore, it can construct a policy knowledge base based on the disassembly results, support knowledge base management functions, and provide interfaces for retrieving basic policy information and content. By parsing financial policy documents, banking policy assets can be reused, version management can be facilitated, and content retrieval efficiency and accuracy can be improved. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of the multimodal document analysis method in the embodiments of this application; Figure 2 This is a flowchart illustrating the multimodal feature extraction process in this application embodiment; Figure 3 This is a flowchart illustrating the directory structure tree obtained in this application embodiment; Figure 4 This is a flowchart illustrating the conversion of a directory structure tree into a knowledge graph in an embodiment of this application. Figure 5 This is a flowchart illustrating document content retrieval in an embodiment of this application; Figure 6 This is a flowchart illustrating the analysis of the causes of parsing errors in an embodiment of this application; Figure 7 This is a structural diagram of the multimodal document analysis device in the embodiments of this application; Figure 8 This application provides an application scenario diagram and architecture diagram for the intelligent parsing and retrieval of banking systems in this embodiment. Figure 9 The following are the algorithms and flowcharts in the embodiments of this application; Figure 10 This is a diagram illustrating the backend management process and architecture in the embodiments of this application; Figure 11 This is a block diagram of the intelligent analysis device in the embodiments of this application; Figure 12 This is a schematic diagram of the structure of the electronic device in the embodiments of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0019] The information collected in the technical solution of this application is information and data authorized by the user or fully authorized by all parties. The collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data all comply with the relevant laws, regulations and standards of the relevant countries and regions, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation portals are provided for users to choose to authorize or refuse.

[0020] Provide users with corresponding operation entry points, allowing them to choose to agree to or reject the automated decision results; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0021] In one embodiment, see Figure 1 To facilitate convenient and efficient parsing of policy documents and provide a convenient and accurate policy retrieval method, this application provides a multimodal document analysis method, including: S101: Perform multimodal parsing on the received unstructured document to obtain the document content and corresponding content tags; S102: Construct a tree structure for the document content and corresponding content tags according to the preset financial system document structure to obtain a directory structure tree; S103: Convert the directory structure tree into a knowledge graph, and perform document content retrieval based on the knowledge graph.

[0022] Understandably, see Figure 8 As shown, the application scenarios in this application mainly include: server 101, terminal device 102, server 103, terminal device 104 and server 105.

[0023] Servers 101, 103, and 105 are different system servers; terminal devices 102 and 104 are different terminal devices. The servers and terminal devices are interconnected via arrows.

[0024] Server 101 is a data source server, mainly providing various rules and regulations documents of the bank. It can push the rules and regulations documents and basic information to server 103 in batches in any form, such as scheduled tasks or real-time push.

[0025] Terminal device 102 is a background management terminal device. Management users can manually upload policy documents and basic information to server 103 through terminal device 102, and can also manage and schedule policy document parsing tasks running on server 103.

[0026] Server 103 is a policy parsing server, responsible for receiving policy documents and basic information pushed by servers 101 and 102, managing the pushed policies according to the version management algorithm, parsing the policy documents according to the policy parsing algorithm, generating a directory structure tree and knowledge graph, storing the parsing results and basic policy information in the database, and providing an interface for server 105 to query in real time.

[0027] Terminal device 104 is a system query terminal device. Users can transmit the information and parameters they need to query to server 105 through terminal device 104 and wait for server 105 to return the query results.

[0028] Server 105 is a business server. The business program running on it integrates a policy query function component. After obtaining the query parameters transmitted by terminal device 104, it can obtain the corresponding policy entries by calling the policy query interface provided by server 103 and return the query results to terminal device 104.

[0029] Terminal devices 102 and 104 can be mobile phones, tablets, laptops, PCs, etc. Users can access the above functions through a webpage via a browser, and this embodiment does not impose any restrictions on this.

[0030] Servers 101, 103, and 105 can be service servers, mainly requiring a processor, memory, and communication interface. They are capable of receiving instructions, running system parsing algorithms, storing system parsing results, and returning request results. This embodiment does not impose any restrictions on this.

[0031] Figure 8 The number of terminal devices and servers shown is merely illustrative; in actual use, it can be expanded as needed. Figure 8 The application scenarios and architecture described above will be explained in detail below, including the algorithms and backend management functions involved in the intelligent analysis of systems.

[0032] In this embodiment of the application, the process of parsing the content of the banking system includes: First, users upload the policy document and a basic information form. The policy document contains the main text of the policy, generally including a title, chapters, and articles. The basic information form includes multiple fields such as policy name, policy document number, policy release date, policy effective date, security classification, issuing unit, and whether it is expired. These files are uploaded to the server via a browser. The file can be an unstructured document.

[0033] Then, the system schedules the push of policy documents and basic information forms, with the upstream system pushing the latest policy documents and basic information forms daily.

[0034] Next, task scheduling is performed, and the task execution status can be displayed, including four statuses: parsed, parsing error, pending parsing, and parsing in progress. It also provides task management functions, including deleting tasks, starting tasks, and stopping tasks.

[0035] Subsequently, the policy document is parsed using multimodal document analysis and assembly techniques, including determining whether the policy already exists, locating the main text, document parsing, reconstructing the directory structure tree, and storing the content. Specific steps include: (1) To determine whether the system already exists, the system name and system document number are compared with the system basic information table in the database. According to the bank's system rules, the system is considered to exist in the system only when the system name and system document number are completely consistent with a certain item in the system basic information table.

[0036] (2) Locate the main text according to the naming rules of the appendix to the system.

[0037] (3) Perform document parsing, process the document using multimodal technology, and then break it down into entries according to rules. At the same time, record the structural information of the parsing results, including whether the entry belongs to a title, chapter, or section. In addition, perform a series of processing on the main text content, mainly including handling automatic number parsing, footnote processing, inappropriate line breaks, and symbol mismatch errors.

[0038] (4) Reconstruct the directory structure tree. Form a directory structure tree according to the system document structure specifications. For example, the directory structure of the title is 1, the directory structure of the first chapter is 1.1, and the directory structure of the first item is 1.1.1. Complete the reconstruction in this way.

[0039] (5) Store the parsed system entry information and basic information into the database.

[0040] In addition, this application provides a back-end management method for intelligent analysis and retrieval of banking systems, including: It provides a backend management system, including the ability to retrieve, start, and stop system analysis tasks; the ability to add, delete, modify, and query analysis results; the ability to view analysis task progress and prompts; and the ability to control access permissions for backend management accounts.

[0041] Provide adjustment solutions for the parsing model and the parsing process of institutional documents, including: classifying parsing errors, locating institutional documents, analyzing whether the parsed documents are correct, analyzing whether the parsed content is correct, analyzing whether the directory structure tree is correct, etc.; and adjusting the parsing model or the parsing process of institutional documents based on the identified causes.

[0042] In addition, see Figure 11 This application also provides an intelligent analysis device 400 for banking system content, comprising: The receiving module 401 receives the policy documents and basic information forms uploaded manually from the page and pushed by the scheduled task, and stores them in the server; The system parsing module 402 is used to locate, parse, and reconstruct the directory structure tree of system documents, parse basic information, and store the parsed item results and basic information respectively. The backend management module 403 is used for task scheduling, as well as adding, deleting, modifying, and querying parsing results and directory structure trees, and locating the cause of parsing errors; The result retrieval module 404 provides an interface for retrieving parsed results.

[0043] As described above, the multimodal document analysis method provided in this application can receive financial policy documents uploaded by users, compare updated policies with existing policies, and use multimodal document analysis and assembly technology according to banking policy writing rules to decompose and store policy clauses. This includes decomposing the main text by chapter and paragraph, constructing a directory structure tree, fine-tuning automatic numbering and inappropriate line breaks, and storing and managing basic policy information (including issuing agency, issuance time, etc.). It can also build a policy knowledge base based on the decomposition results, support knowledge base management functions, and provide interfaces for retrieving basic policy information and content. By parsing financial policy documents, banking policy assets can be reused, version management can be facilitated, and content retrieval efficiency and accuracy can be improved.

[0044] In one embodiment, see Figure 2 The step of performing multimodal parsing on the received unstructured document to obtain the document content and corresponding content tags includes: P201: Based on the document's visual layout and semantic content, perform multimodal feature extraction on the unstructured document to obtain structural and semantic features; P202: Based on the document's visual layout, perform cross-modal association on the structural features and semantic features to obtain structural-semantic association relationships; P203: Generate the overall document structure based on the structural features, the semantic features, and the structural semantic relationships, and output tagged structured data; wherein, the tagged structured data includes the document content and corresponding content tags.

[0045] Understandably, when processing unstructured documents, it's essential to first analyze their visual layout and semantic content. Multimodal feature extraction techniques are then used to combine these two aspects, capturing the rich information in both form and content. The visual layout encompasses the arrangement of page elements, font styles, colors, sizes, spacing, and other visual attributes. These elements collectively constitute the document's physical structure. The semantic content, on the other hand, involves the meaning of the text, keywords, contextual relationships, and underlying logical connections. Multimodal feature extraction allows for the extraction of structural and semantic features from both visual and semantic dimensions. The former describes the document's physical organization, while the latter reveals the text's inherent meaning. Next, based on the document's visual layout, cross-modal correlation analysis is performed on the extracted structural and semantic features to establish their intrinsic relationship. This correlation helps understand how visual elements carry and express semantic information; for example, the font size and position of a title may indicate its importance, while the spacing and indentation between paragraphs may reflect the logical hierarchy of the content. This cross-modal correlation forms a structural-semantic relationship, laying the foundation for subsequent document structuring. Finally, by leveraging these structural and semantic features and their interrelationships, the overall logical framework of the document is constructed, generating a hierarchical and logically coherent document structure, which is then transformed into tagged structured data. This structured data not only preserves the content of the original document but also assigns corresponding content tags to each content element, facilitating subsequent retrieval, analysis, and application, greatly improving the efficiency and accuracy of document processing.

[0046] For specific implementation, please refer to Figure 9 .

[0047] S201 is the step of manually uploading policy documents and basic information forms.

[0048] The policy document is the document containing the main text that needs to be analyzed. The content is the main text of the policy, generally including a title, chapters, and articles. Each chapter contains multiple articles. For example, Chapter 1 contains articles 1 to 5, Chapter 2 contains articles 6 to 8, and so on.

[0049] The basic information table is a table containing basic information about the system, including multiple fields such as system name, system document number, system release date, system effective date, security classification, issuing unit, and whether it is invalid, which together constitute the basic information of the system.

[0050] Users can upload the policy documents to be parsed and the corresponding basic information forms to the server through a browser. This method is suitable for use when uploading historical policy documents or when there are defects in the parsing of policy documents pushed by automatic tasks.

[0051] S202 is the procedure for scheduled tasks / real-time push of policy documents and basic information.

[0052] The data source server periodically or in real-time pushes policy documents and basic information forms to the server in batches. This method is suitable for automatic pushing when new policy documents are released or revised. The policy documents and basic information forms pushed are the same as those in S201, and will not be described again.

[0053] S203 is the task scheduling step.

[0054] When users upload, schedule, or push policy documents and forms in real time, the files need to be executed sequentially. Task scheduling displays the task execution status, including four states: Parsed, Parsing Error, Pending Parsing, and Parsing in Progress. Parsed means the parsing work is complete, but the result may be incorrect; Parsing Error means an error occurred during parsing, and no result has been obtained; Pending Parsing means there are still tasks being parsed ahead, and parsing must wait for them to complete before proceeding; Parsing in Progress means the task is currently being executed, but if it takes too long, it may get stuck.

[0055] In addition, it provides task management functions, mainly including deleting tasks, starting tasks, and stopping tasks. When the system receives a task, it will start parsing it sequentially by default. This manual management function is mainly for situations such as parsing errors, parsing time being too long, and the need to adjust priorities.

[0056] If a parsing error occurs, it means that the computer program is currently unable to parse the document. In this case, you need to use logs and other recording tools to locate the cause of the parsing error. After locating and resolving the problem, you can delete the task and manually upload and parse it again.

[0057] If the parsing process takes too long, you can pause the task and use logs or other recording tools to locate the reason why it continues to fail to produce results. Once the problem is located and resolved, you can restart the task or delete the task and manually upload and parse it again.

[0058] The need to adjust priorities is primarily for situations where there are too many policy documents to be parsed, but only one or a few documents urgently need parsing. In this case, select all and stop the task, check the tasks that need to be executed first, and click "Start Task." After the high-priority policies have been parsed, select all tasks again and click "Start Task."

[0059] S204 is a procedure to compare whether a system already exists.

[0060] When the server receives a newly transmitted policy document and basic information table, it compares the received policy name and policy document number with the policy basic information table in the database. According to the bank's policy rules, only when the policy name and policy document number completely match a certain entry in the policy basic information table is the policy considered to exist in the system. At this time, the parsing task will be marked as parsing complete, and the details will indicate that it already exists, indicating that the file processed by this task already exists.

[0061] In addition, it is important to note the presence of Chinese and English punctuation. These punctuation marks differ in their usage and capitalization; a strict match would be considered as two different systems. Therefore, the method used in the comparison process is to remove all these punctuation marks before performing the comparison.

[0062] As can be seen from the above description, the multimodal document analysis method provided in this application can perform multimodal parsing on received unstructured documents to obtain document content and corresponding content tags.

[0063] In one embodiment, see Figure 3 The step of constructing a tree structure for the document content and corresponding content tags according to a preset financial system document structure to obtain a directory structure tree includes: P301: Based on the structure of the financial policy document, identify the hierarchical relationship between the content tags and the parent-child node association to obtain a tree framework; wherein, the tree framework includes parent nodes and child nodes; P302: Group content tags with the same parent node into the same level, and arrange the content tags in the same level according to the document content; P303: Based on the correspondence between the content tags and the document content, the document content is written into the child nodes corresponding to the content tags to obtain the directory structure tree.

[0064] Understandably, given the structural characteristics of financial regulatory documents, the first step is to identify and analyze the content tags within the document, clarifying the hierarchical relationships between tags and the connections between parent and child nodes, thereby constructing a hierarchical tree framework. This process requires the system to be able to parse the logical relationships between tags, such as which tags are parent directories and which are child directories, thus forming a clear tree structure and ensuring that each node has a clear parent-child relationship. After the tree framework is constructed, content tags with the same parent node need to be grouped into the same level for subsequent organization.

[0065] To ensure the rationality and readability of the structure, tags at the same level need to be sorted according to the actual content of the document, conforming to logical order or business specifications. This sorting method not only helps improve the document's organization but also facilitates subsequent retrieval and maintenance. After completing the hierarchical division and sorting of tags, the next crucial step is to match the specific content in the document with the corresponding tags, ensuring that each piece of content is accurately placed into its respective tag node. This process requires in-depth analysis of the document content, identifying information related to each tag, and writing it into the corresponding child nodes, ultimately forming a complete directory structure tree. This directory structure tree not only reflects the hierarchical relationship of the document but also the mapping relationship between content and tags, making the entire financial policy document clearer in structure and more logically rigorous, facilitating subsequent review, updates, and management. Through the above steps, the entire document structure is systematically organized, providing a foundation for subsequent information extraction, content retrieval, and automated processing.

[0066] For specific implementation, please refer to Figure 9 .

[0067] Following S204, S205 is the step to locate the document that needs to be parsed, i.e., the position of the text.

[0068] When a policy document is uploaded to a server, it often includes multiple attachments. However, the main content of these attachments may not be the actual policy document; it might resemble an announcement. Therefore, it is necessary to first locate the main policy document.

[0069] The appendices to a policy may fall into the following categories: 1. The policy name includes quotation marks; 2. The appendix name includes the policy name; 3. The appendix name includes the policy name and a revision note, such as "Revision Notes on the 'xxx Policy'"; 4. The policy name includes "Supplementary Provisions" or "Some Clauses".

[0070] For the first case, use the content within the quotation marks for matching; otherwise, use the full name of the system for matching.

[0071] In the second scenario, if the attachment name contains this content, then the attachment is considered part of the main document. Conversely, if no attachment name contains this content, then the main document is considered part of the policy document.

[0072] In the third scenario, if the attachment title contains the words "revision notes", then the attachment is considered to be merely a revision note to the system and is not part of the main text. It is necessary to continue searching for other attachments to locate the main text.

[0073] In the fourth case, if the name of the system includes "supplementary provisions" or "partial clauses", it means that the system is a supplement or modification to the existing system. In this case, when matching attachments, it is not necessary to remove the quotation marks; you can directly match all the titles.

[0074] S206 is the procedure for determining whether a document is in a formal system format.

[0075] Generally, the format of a regulation is as described in S201, with a main structure of title, chapter, and article. If the regulation format is a formal format, the main text is divided according to title, chapter, and article. If it is not a formal format, it is broken down into paragraphs, each representing one article.

[0076] S207 outlines the steps for parsing policy documents.

[0077] Based on the S206 standard for determining whether a document has a proper format, the document is parsed. Multimodal processing is applied to the document, followed by text concatenation and then breakdown into entries according to rules. Simultaneously, structural information is recorded for the parsed results, including whether each entry belongs to a title, chapter, or section, thus constructing a directory structure tree.

[0078] In addition, a series of processing steps are performed on the main text content, mainly including handling errors such as automatic number parsing, footnote processing, inappropriate line breaks, and mismatched symbols.

[0079] For automatic numbering, it's necessary to differentiate based on document type; otherwise, it will result in garbled text during parsing. After parsing, numbers can be generated sequentially. For parts of the document that do not use automatic numbering, the numbers should be rearranged according to their order. For example, if 1.1, 1.2, and 1.4 are automatic numbers, and 1.3 is a manually entered number, then all numbers should be rearranged to prevent parsing errors caused by manual numbering.

[0080] For footnote processing, locate the footnote and replace the footnote tag in the main text with the content.

[0081] Inappropriate line breaks need to be located at their current position. Inappropriate line breaks can split the article content into two paragraphs. If a line break is used in a title, it may cause the title to be incomplete. In this case, the line break needs to be removed to combine the content into one paragraph.

[0082] Errors such as mismatched symbols mainly include mismatched Chinese and English symbols. In this case, it is necessary to match the punctuation and change one of the symbols to make the whole consistent.

[0083] As can be seen from the above description, the multimodal document analysis method provided in this application can construct a tree structure for the document content and corresponding content tags according to the preset financial system document structure, and obtain a directory structure tree.

[0084] In one embodiment, see Figure 4 The step of converting the directory structure tree into a knowledge graph includes: S401: Identify the nodes in the directory structure tree as entities in the knowledge graph; S402: Construct the relationships of the knowledge graph based on the tree framework; S403: Construct a graph structure for the entities and relationships to obtain a network graph; S404: Render the network graph to obtain a visualized knowledge graph.

[0085] Understandably, the first step in constructing a knowledge graph is to transform the nodes in the directory structure tree into entities within the knowledge graph. This step is fundamental to knowledge graph construction. Each node in the directory structure tree, whether a parent or child node, represents a specific concept or topic within the document. These nodes are interconnected through hierarchical relationships. By identifying these nodes, they can be mapped to entities in the knowledge graph, each carrying specific semantic information, such as clauses, definitions, or rules in a financial system. Next, the relationships within the knowledge graph are constructed based on the tree framework. This step involves analyzing the parent-child relationships and hierarchical structure between nodes to determine the connection methods between entities. For example, the subordinate relationship between a parent and child node can be transformed into a "containment" or "subclass" relationship in the knowledge graph, while the parallel relationship between sibling nodes may correspond to a "parallel" or "related" relationship. In this way, the hierarchical structure of the document can be transformed into a relational network in the knowledge graph, laying the foundation for subsequent graph structure construction. Subsequently, a graph structure is constructed for the entities and relationships, with entities as vertices and relationships as edges, forming a network graph. This step requires ensuring accurate mapping of entities and relationships, avoiding information loss or incorrect connections, thereby guaranteeing the integrity and consistency of the network graph. Finally, the network graph is rendered, transforming it into a visual knowledge graph. The rendering process involves selecting appropriate graphical representations, such as node shape, color, and size, as well as edge style and direction, to intuitively display the relationships and hierarchical structure between entities. Through visualization, users can more clearly understand the knowledge structure in financial policy documents, facilitating retrieval, analysis, and application.

[0086] For specific implementation, please refer to Figure 9 .

[0087] Following S208, S209 is the step of storing the institutional parsing entries into a database with an open-source distributed search and analysis engine.

[0088] Because the lengths of the parsed entries vary, the parsed results of unstructured data are suitable for storage in this type of database.

[0089] S210 is the step of storing basic system information into a database with transaction processing and structured data management functions.

[0090] The acquired basic information related to the system, including multiple fields such as system name, system document number, system release time, system effective time, security classification, issuing unit, and whether it is invalid, will be stored in this type of database in a fixed-length format.

[0091] As can be seen from the above description, the multimodal document analysis method provided in this application can convert the directory structure tree into a knowledge graph.

[0092] In one embodiment, see Figure 5 The document content retrieval based on the knowledge graph includes: S501: Parse the obtained search request information into a structured query statement; S502: Using the structured query statement, locate the associated entities and relationship paths in the knowledge graph to obtain the matching information block; S503: Perform result reasoning and optimization on the matched information block and return the result to obtain the search content corresponding to the search requirement information.

[0093] Understandable, Figure 10 This illustration shows the back-end management process and architecture of intelligent analysis and retrieval of banking regulations.

[0094] S301 describes the steps to log in to the backend management system.

[0095] The backend management system provides a login page and controls access permissions based on the logged-in user account. These permissions primarily include the ability to retrieve, start, and stop system analysis tasks, as well as the ability to add, delete, modify, and query analysis results, and view task progress and notifications.

[0096] S302 outlines the steps for retrieving and parsing the relevant policy documents in case of task failure or exceptions.

[0097] This step primarily filters for parsing error cases. The backend management system provides filtering for parsing task status, including four states: parsed, parsing error, pending parsing, and parsing in progress. Selecting the task with parsing error will complete the query.

[0098] S303 describes the steps for querying policy documents that are parsed normally but yield incorrect results.

[0099] This step mainly addresses the situation where, during actual use, the parsing task is completed normally, but the parsing results do not match expectations.

[0100] This situation occurs when a user finds errors in the returned clauses while searching for relevant terms. Examples include punctuation marks becoming garbled, missing content, multiple clauses being merged into one, or footnotes not being correctly populated back into the original text. In this case, it's necessary to search for the corresponding policy document name and locate the erroneous entry.

[0101] S304 describes the steps to locate the original policy document.

[0102] This step has two scenarios: one is to locate the system name to be re-parsed based on the failed parsing task found in S302; the other is to directly locate the system name based on step S303.

[0103] S305 is the step for analyzing whether the parsed document is correct.

[0104] This step corresponds to situation S302: the parsing task encountered an error, causing the document parsing to fail. This mainly includes situations such as the document already existing, the parsing document having an incorrect format, the attached document not matching the information recorded in the basic information table, errors occurring during file parsing, and the parsing process crashing.

[0105] The document already exists if it has been parsed in a previous task and is already in the database. If you still need to parse this document, you must delete the existing data.

[0106] The parsed document is in the following format: the main text document is not in the default file format, or the basic information table is not in the default file format. This usually only occurs when manually uploading and parsing the document.

[0107] The mismatch between the attached document and the basic information table is as follows: the name of the attached document, the name of the policy in the attachment, and the corresponding field name in the basic information table do not match exactly.

[0108] An error occurred during the parsing process, meaning an error occurred while the parsing algorithm was running. In this case, it's necessary to refer to the error messages printed in the logs to pinpoint the specific error step.

[0109] The parsing process can be checked for crashes based on the task's runtime. For example, if parsing a document takes more than 10 minutes, the parsing task will be considered stuck.

[0110] S306 is the step for analyzing whether the parsed content is correct.

[0111] This step corresponds to S303, where users encounter parsed content errors during use. These mainly include document content segmentation errors, character errors within the document, footnote backfilling errors, and automatic numbering errors.

[0112] Content segmentation errors refer to incorrect identification of paragraphs in the original text. Document structure includes headings, chapters, and sections, with each level potentially having multiple sub-levels or paragraphs. If the level distinction is incorrect, such as misidentification or omission, it can lead to an excessive number of paragraphs or incorrect merging of paragraphs.

[0113] Errors in the document's characters mean that when parsing the characters in the document, the characters are rendered as garbled text.

[0114] Footnote backfilling errors refer to the situation where footnotes in the main text of a document are often in the form of “[1]”. In order to facilitate users’ use of the banking system, the footnote content needs to be backfilled into the original text. At this time, there may be spaces, special characters retained, etc., which will affect users’ reading and citation.

[0115] In cases of automatic numbering errors, there may be situations where the automatic numbering parsing logic for two different document formats is inconsistent; different components need to be used for parsing. Additionally, there may be cases where automatic and manual numbering are mixed. For example, if items 1, 2, 4, and 5 are automatically numbered, while item 3 is manually numbered, the parsing result might interpret item 3 and item 4 as item 3 in both cases.

[0116] S307 is the step for analyzing whether the directory structure tree is correct.

[0117] The directory structure tree is formed according to the standard structure of the document. For example, the directory structure tree of the title is 1, the directory structure tree of the first chapter is 1.1, and the directory structure tree of the first item is 1.1.1. If the directory structure tree of a certain level is incorrect, it may cause the structure to be confused. For example, the directory structure tree of the fifth item should be 1.1.5, but the result after parsing is 1.2. At this time, the fifth item is mistakenly regarded as the second chapter.

[0118] As can be seen from the above description, the multimodal document analysis method provided in this application can perform document content retrieval based on the knowledge graph.

[0119] In one embodiment, see Figure 6 The multimodal document analysis method further includes: S601: Reconstruct the directory structure tree and the knowledge graph based on the received retrieval error information; S602: Analyze the causes of errors using the reconstructed directory structure tree and knowledge graph.

[0120] Understandably, the directory structure tree and knowledge graph are reconstructed based on the received retrieval error information. First, the error information needs in-depth analysis to identify the error type and its specific location within the directory structure. By parsing the error information, the file paths, module names, and related knowledge nodes involved in the problem can be located, thereby determining the directory hierarchy and relationships in the knowledge graph that need adjustment. During the reconstruction process, the logical clarity and hierarchical structure of the directory structure should be ensured, so that the error information can be accurately mapped to the corresponding directory nodes. Simultaneously, the entities and relationships related to the error in the knowledge graph are updated to ensure the integrity and accuracy of the knowledge system. Next, the reconstructed directory structure tree and knowledge graph are used for in-depth analysis of the error causes. By examining the association paths and dependencies in the graph, the root cause of the error is traced to determine whether it is an access conflict caused by an unreasonable directory structure or a logical error caused by inaccurate node relationships in the knowledge graph. Based on this, the reconstruction effect is further verified to ensure that the error information can be correctly parsed and located. At the same time, the organization of the directory structure and knowledge graph is optimized to improve the maintainability and scalability of the system, ultimately achieving accurate diagnosis and effective resolution of retrieval errors.

[0121] For specific implementation, please refer to Figure 9 .

[0122] Following S307, S308 is the step for adjusting the analytical model.

[0123] The adjustment of the parsing model mainly targets two cases, S306 and S307. It requires manual debugging based on the error messages printed in the logs and the parsing results that do not meet expectations. The program needs to be debugged to find the defective steps of the algorithm and then make adjustments.

[0124] S309 outlines the steps for adjusting the process of parsing policy documents.

[0125] The parsing model adjustment is mainly for the S305 case. It requires manual intervention based on the error messages printed in the logs to locate the error flow in the parsing process and then make adjustments.

[0126] As can be seen from the above description, the multimodal document analysis method provided in this application can analyze the cause of parsing errors by utilizing the reconstructed directory structure tree and knowledge graph.

[0127] Based on the same inventive concept, this application also provides a multimodal document analysis device, which can be used to implement the method described in the above embodiments, as described in the following embodiments. Since the principle of the multimodal document analysis device in solving the problem is similar to that of the multimodal document analysis method, the implementation of the multimodal document analysis device can refer to the implementation of the software performance benchmark determination method, and repeated details will not be elaborated further. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0128] In one embodiment, see Figure 7 To facilitate convenient and efficient parsing of institutional documents and provide a convenient and accurate means of institutional retrieval, this application provides a multimodal document analysis device, comprising: The multimodal parsing unit 701 is used to perform multimodal parsing on the received unstructured document to obtain the document content and the corresponding content tags; The tree structure construction unit 702 is used to construct a tree structure for the document content and corresponding content tags according to the preset financial system document structure, so as to obtain a directory structure tree. The document content retrieval unit 703 is used to convert the directory structure tree into a knowledge graph and perform document content retrieval based on the knowledge graph.

[0129] From a hardware perspective, in order to facilitate convenient and efficient parsing of institutional documents and provide a convenient and accurate means of institutional retrieval, this application provides an embodiment of an electronic device for implementing all or part of the content of the multimodal document analysis method, wherein the electronic device specifically includes the following: The system comprises a processor, a memory, a communications interface, and a bus; wherein the processor, memory, and communications interface communicate with each other via the bus; the communications interface is used to realize information transmission between the multimodal document analysis device and core business systems, user terminals, and related databases and other related devices; the logic controller can be a desktop computer, tablet computer, or mobile terminal, etc., and this embodiment is not limited to these. In this embodiment, the logic controller can be implemented with reference to the embodiments of the multimodal document analysis method and the multimodal document analysis device in the embodiments, the content of which is incorporated herein, and repeated details will not be described again.

[0130] It is understood that the user terminal may include smartphones, tablet computers, network set-top boxes, portable computers, desktop computers, personal digital assistants (PDAs), in-vehicle devices, smart wearable devices, etc. Among these, the smart wearable devices may include smart glasses, smartwatches, smart bracelets, etc.

[0131] In practical applications, parts of the multimodal document analysis method can be executed on the electronic device side as described above, or all operations can be completed on the client device. The choice can be made based on the processing power of the client device and the limitations of the user's usage scenario. This application does not impose any limitations on this. If all operations are completed on the client device, the client device may further include a processor.

[0132] The aforementioned client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission. The server may include a server on the task scheduling center side; in other implementation scenarios, it may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, a server cluster consisting of multiple servers, or a distributed server structure.

[0133] Figure 12 This is a schematic block diagram illustrating the system configuration of the electronic device 9600 according to an embodiment of this application. Figure 12 As shown, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It is worth noting that... Figure 12 This is an example; other types of structures can also be used to supplement or replace this structure to achieve telecommunications functions or other functions.

[0134] In one embodiment, the multimodal document analysis method functionality can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following control: S101: Perform multimodal parsing on the received unstructured document to obtain the document content and corresponding content tags; S102: Construct a tree structure for the document content and corresponding content tags according to the preset financial system document structure to obtain a directory structure tree; S103: Convert the directory structure tree into a knowledge graph, and perform document content retrieval based on the knowledge graph.

[0135] As described above, the multimodal document analysis method provided in this application can receive financial policy documents uploaded by users, compare updated policies with existing policies, and use multimodal document analysis and assembly technology according to banking policy writing rules to decompose and store policy clauses. This includes decomposing the main text by chapter and paragraph, constructing a directory structure tree, fine-tuning automatic numbering and inappropriate line breaks, and storing and managing basic policy information (including issuing agency, issuance time, etc.). It can also build a policy knowledge base based on the decomposition results, support knowledge base management functions, and provide interfaces for retrieving basic policy information and content. By parsing financial policy documents, banking policy assets can be reused, version management can be facilitated, and content retrieval efficiency and accuracy can be improved.

[0136] In another embodiment, the multimodal document analysis device can be configured separately from the central processing unit 9100. For example, the data composite transmission device can be configured as a chip connected to the central processing unit 9100, and the function of the multimodal document analysis method can be realized through the control of the central processing unit.

[0137] like Figure 12 As shown, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily need to include these components. Figure 12 All components shown; in addition, the electronic device 9600 may also include Figure 12 For components not shown, please refer to existing technologies.

[0138] like Figure 12 As shown, the central processing unit 9100, sometimes also referred to as a controller or operating control, may include a microprocessor or other processor device and / or logic device, which receives inputs and controls the operation of various components of the electronic device 9600.

[0139] The memory 9140 may be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It may store the aforementioned failure-related information, and also store a program for executing that information. The central processing unit 9100 may execute the program stored in the memory 9140 to perform information storage or processing, etc.

[0140] Input unit 9120 provides input to central processing unit 9100. Input unit 9120 may be, for example, a keypad or touch input device. Power supply 9170 provides power to electronic device 9600. Display 9160 displays images and text. Display may be, for example, an LCD display, but is not limited thereto.

[0141] The memory 9140 can be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as EPROMs. The memory 9140 can also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142 for storing application programs and function programs or processes for executing the operation of the electronic device 9600 via the central processing unit 9100.

[0142] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for the electronic device's communication functions and / or for performing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0143] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module (transmitter / receiver) 9110 is coupled to the central processing unit 9100 to provide input signals and receive output signals, which is the same as in a conventional mobile communication terminal.

[0144] Based on different communication technologies, multiple communication modules 9110 can be configured in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module (transmitter / receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby realizing typical telecommunications functions. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processor 9130 is also coupled to a central processing unit 9100, enabling on-device recording via the microphone 9132 and on-device playback of stored sound via the speaker 9131.

[0145] Embodiments of this application also provide a computer-readable storage medium capable of implementing all steps of the multimodal document analysis method with a server or client as the execution subject in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the multimodal document analysis method with a server or client as the execution subject in the above embodiments. For example, when the processor executes the computer program, it implements the following steps: S101: Perform multimodal parsing on the received unstructured document to obtain the document content and corresponding content tags; S102: Construct a tree structure for the document content and corresponding content tags according to the preset financial system document structure to obtain a directory structure tree; S103: Convert the directory structure tree into a knowledge graph, and perform document content retrieval based on the knowledge graph.

[0146] As described above, the multimodal document analysis method provided in this application can receive financial policy documents uploaded by users, compare updated policies with existing policies, and use multimodal document analysis and assembly technology according to banking policy writing rules to decompose and store policy clauses. This includes decomposing the main text by chapter and paragraph, constructing a directory structure tree, fine-tuning automatic numbering and inappropriate line breaks, and storing and managing basic policy information (including issuing agency, issuance time, etc.). It can also build a policy knowledge base based on the decomposition results, support knowledge base management functions, and provide interfaces for retrieving basic policy information and content. By parsing financial policy documents, banking policy assets can be reused, version management can be facilitated, and content retrieval efficiency and accuracy can be improved.

[0147] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0148] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0149] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0150] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0151] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A document analysis method based on multimodal analysis, characterized in that, include: The received unstructured document is parsed using multimodal methods to obtain the document content and its corresponding content tags; Based on the preset financial system document structure, a tree structure is constructed for the document content and corresponding content tags to obtain a directory structure tree; The directory structure tree is converted into a knowledge graph, and document content retrieval is performed based on the knowledge graph.

2. The document analysis method based on multimodal analysis according to claim 1, characterized in that, The step of performing multimodal parsing on the received unstructured document to obtain the document content and corresponding content tags includes: Multimodal feature extraction is performed on the unstructured document based on its visual layout and semantic content to obtain structural and semantic features. Based on the document's visual layout, cross-modal association is performed on the structural features and semantic features to obtain structural-semantic association relationships; Based on the structural features, the semantic features, and the structural semantic relationships, an overall document structure is generated, and tagged structured data is output; wherein, the tagged structured data includes the document content and corresponding content tags.

3. The document analysis method based on multimodal analysis according to claim 1, characterized in that, The step of constructing a tree structure for the document content and corresponding content tags according to a preset financial system document structure, resulting in a directory structure tree, includes: Based on the structure of the financial policy document, the hierarchical relationship between the content tags and the parent-child node association are identified to obtain a tree framework; wherein, the tree framework includes parent nodes and child nodes; Content tags with the same parent node are grouped into the same level, and the content tags in the same level are arranged according to the document content. Based on the correspondence between the content tags and the document content, the document content is written into the child nodes corresponding to the content tags to obtain the directory structure tree.

4. The document analysis method based on multimodal analysis according to claim 3, characterized in that, The step of converting the directory structure tree into a knowledge graph includes: The nodes in the directory structure tree are identified as entities in the knowledge graph; The relationships of the knowledge graph are constructed based on the tree framework; A graph structure is constructed from the entities and their relationships to obtain a network graph; The network graph is rendered to obtain a visualized knowledge graph.

5. The document analysis method based on multimodal analysis according to claim 1, characterized in that, The document content retrieval based on the knowledge graph includes: The retrieved search requirements information is parsed into structured query statements; The structured query statement is used to locate related entities and relationship paths in the knowledge graph to obtain matching information blocks; The matched information blocks are subjected to result reasoning and optimization, and the returned results are obtained to obtain the search content corresponding to the search requirements.

6. The document analysis method based on multimodal analysis according to claim 1, characterized in that, Also includes: The directory structure tree and the knowledge graph are reconstructed based on the received retrieval error information; The reconstructed directory structure tree and knowledge graph were used to analyze and resolve the causes of errors.

7. A document analysis device based on multimodal analysis, characterized in that, include: The multimodal parsing unit is used to perform multimodal parsing on the received unstructured document to obtain the document content and the corresponding content tags; The tree structure construction unit is used to construct a tree structure of the document content and corresponding content tags according to the preset financial system document structure, so as to obtain a directory structure tree. The document content retrieval unit is used to convert the directory structure tree into a knowledge graph and perform document content retrieval based on the knowledge graph.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the multimodal document analysis method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the multimodal document analysis method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program / instructions, characterized in that, When executed by a processor, the computer program / instructions implement the steps of the multimodal document analysis method according to any one of claims 1 to 6.

Images