A flowchart understanding method, system, device and storage medium

Abstract

The present application belongs to the technical field of data processing, and particularly relates to a flowchart understanding method, system, device and storage medium. The present application aims to solve the technical problems of the prior art, such as dependence on a large amount of high-quality artificial annotation data and lack of accurate modeling of high-order semantic relationships. The present application comprises the following steps: data analysis and data cleaning are performed on original flowchart data to obtain flowchart text data; a training data set is constructed, and a pre-trained visual multi-modal large model is fine-tuned through the training data set; flowchart entities and their logical relationships are extracted from the flowchart text data by using a flowchart domain large model, a structured flowchart knowledge graph is constructed, and the flowchart knowledge graph is stored in a graph database; a user question is obtained, relevant entities and relationships are retrieved in the graph database, and a preliminary retrieval answer is obtained; the preliminary retrieval answer and the user question are fused to obtain a prompt question; and the prompt question is input into the flowchart domain large model to obtain a final flowchart understanding answer.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, and in particular relates to a flowchart understanding method, system, device and storage medium. Background Technology

[0002] In the early days, Optical Character Recognition (OCR) and object detection technologies were widely used to extract unstructured elements, such as flowcharts, approval forms, system interface screenshots, and electronic document charts. Object detection models were used to detect flowchart units, and then OCR technology was applied to the flowchart for recognition. To ensure that the relationships between internal flowchart units were not lost, specific location algorithms or relationship segmentation algorithms were used to supplement the relationships between entities. This traditional approach required the use of multiple technical modules, resulting in poor generalization of the algorithm. Even slight changes in the paradigm of the business department's flowchart necessitated modifications to the processing between modules, making it extremely difficult to implement the technology for flowcharts in complex scenarios.

[0003] In existing technologies, image and text descriptions are implemented in stages. Specifically, image features are first extracted based on a pre-trained CNN or Transformer model, then RNN is used to extract features from the text, the user's query is converted into a semantic vector, entities in the query are extracted by combining rule templates, the text and image features are concatenated and fed into the encoder, and finally the decoder is used to generate the text word by word.

[0004] The existing technology has the following technical problems: 1. Existing technologies rely on a large amount of high-quality manually labeled data, which not only leads to long data preparation cycles and high labor costs, but also makes it difficult to achieve rapid migration and generalization when facing rapid business changes, new fields or new scenarios, as the labeling system often lags behind actual needs.

[0005] 2. Existing technologies, when dealing with relationships between entities in flowcharts, are often limited to simple linear or adjacency connections, lacking accurate modeling of higher-order semantic relationships. They perform poorly in relation extraction, reasoning, and alignment, leading to fragmented understanding of flowchart semantics.

[0006] 3. Existing technologies have a technical shortcoming: difficulty in integrating and understanding multimodal information from flowcharts. Summary of the Invention

[0007] This invention provides a flowchart understanding method, system, device, and storage medium, aiming to solve the technical problems in the prior art, such as reliance on a large amount of high-quality manually labeled data, resulting in long data preparation cycles and high labor costs; when dealing with the relationships between entities in a flowchart, it is often limited to simple linear or adjacency connections, lacking accurate modeling of higher-order semantic relationships; and the difficulty in integrating and understanding multimodal information of flowcharts.

[0008] The technical solution of the present invention to solve the above-mentioned technical problems is as follows: a flowchart understanding method, comprising: Obtain the original flowchart data, and perform data parsing and data cleaning on the original flowchart data to obtain flowchart text data; A training dataset is constructed using the flowchart data, and the pre-trained visual multimodal large model is fine-tuned using the training dataset to obtain a large model of the process domain. The process domain big model is used to extract process entities and their logical relationships from process graph data, construct a structured process knowledge graph, and store the process knowledge graph in a graph database. The system retrieves user questions, searches for relevant entities and relationships in the graph database to obtain preliminary answers. The preliminary search results are then combined with the user's question to generate a prompt question. The prompt question is input into the process domain model, and the process domain model generates the final flowchart understanding answer.

[0009] Furthermore, the above-mentioned data parsing and cleaning of the original flowchart data yields the following flowchart text data: Obtain the original flowchart data, perform data cleaning and parsing on the original flowchart data to obtain parsed data; Based on the parsed data, a question-answer pair set is generated; The question-and-answer pair set is subjected to noise reduction and compliance checks to obtain flowchart data.

[0010] Furthermore, the above-mentioned fine-tuning of the pre-trained visual multimodal large model using the training dataset to obtain the process domain large model is as follows: The flowchart data is converted into a training dataset in triplet form; Construct a large visual multimodal model and insert a low-rank adaptation module into the large visual multimodal model; The low-rank adaptation module is trained using the training dataset to achieve instruction fine-tuning and obtain a large model of the process domain.

[0011] Furthermore, the above-mentioned construction of a structured process knowledge graph and storage of the process knowledge graph in a graph database specifically involves: The flowchart data is input into the large process domain model to extract entities and relationships, and the extraction is assisted by a retrieval enhancement generation method to obtain extracted data in the form of triples. The extracted data is aggregated into triples to obtain a process knowledge graph. The process knowledge graph is stored in a graph database.

[0012] Furthermore, the above-mentioned method of retrieving relevant entities and relationships from the graph database to obtain preliminary search results is as follows: The user question is obtained, and the named entity recognition module and relation extraction module are used to parse the user question to obtain the target entity and relation; Transform the target entities and relationships into query statements; The query statement is used to perform a multi-way search in the graph database to obtain multiple candidate answers; The candidate answers are sorted by relevance and then filtered to obtain the filtered answers; The filtered answers are then structured to obtain preliminary search results.

[0013] Furthermore, the preliminary search results are combined with the user's question to obtain the following prompt: An enhanced prompt template is constructed, and the preliminary search answer is combined with the enhanced prompt template of the user's question input to obtain the prompt question.

[0014] Furthermore, the specific answer to the final flowchart comprehension obtained from the aforementioned large-scale model of the process domain is as follows: The system receives the prompt questions and parses the technical terms and logical structure to obtain process semantic data. Internal reasoning is performed based on the process semantic data to generate a structured response; wherein the structured response is the answer to the flowchart comprehension question.

[0015] Secondly, in order to solve the above-mentioned technical problems, the present invention also provides a flowchart understanding system, comprising: The data preprocessing module is used to acquire raw flowchart data, perform data parsing and data cleaning on the raw flowchart data, and obtain flowchart text data. The model training module is used to construct a training dataset using the flowchart data, and to fine-tune the pre-trained visual multimodal large model using the training dataset to obtain a large model of the process domain. The process knowledge graph module is used to extract process entities and their logical relationships from process graph data using the large model of the process domain, construct a structured process knowledge graph, and store the process knowledge graph in a graph database. The preliminary retrieval module is used to obtain user questions, retrieve relevant entities and relationships in the graph database to obtain preliminary retrieval answers; The enhanced prompt module is used to merge the preliminary search answer with the user's question to obtain a prompt question; The answer generation module is used to input the prompt question into the process domain model and obtain the final flowchart understanding answer generated by the process domain model.

[0016] Thirdly, in order to solve the above-mentioned technical problems, the present invention also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the flowchart understanding method of the present application.

[0017] Fourthly, in order to solve the above-mentioned technical problems, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the flowchart understanding method of the present application.

[0018] Compared with the prior art, the present invention has the following advantages: 1. This invention automatically parses flowcharts using a visual multimodal large model, and automatically extracts entities and relationships using GraphRAG or LightRAG, significantly reducing the need for manual annotation, and is especially suitable for complex, multi-branch business processes.

[0019] 2. The model based on multimodal instruction fine-tuning of this invention can accurately identify technical terms and implicit logic in flowcharts, solving the problem of understanding heterogeneous flowcharts that are difficult to handle by traditional OCR or rule engines.

[0020] 3. This invention dynamically updates the knowledge graph using GraphRAG or LightRAG, allowing the system to respond to process changes in real time without retraining the model. Business departments can directly query the latest processes using natural language, avoiding execution errors or communication delays caused by process updates.

[0021] 4. With this invention, users do not need to learn flowchart symbols or query syntax. They can directly ask questions in natural language, and the system, combined with Retrieval Enhancement Generation (RAG), returns structured answers within seconds, effectively improving the efficiency of process information processing.

[0022] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart illustration of a flowchart understanding method according to an embodiment of the present invention is shown; Figure 2 A flowchart illustrating a flowchart understanding method according to an embodiment of the present invention is shown; Figure 3 A schematic diagram of a flowchart understanding system according to an embodiment of the present invention is shown; Figure 4 A schematic diagram of an electronic device structure according to an embodiment of the present invention is shown. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Figure 1 A flowchart illustrating a method for understanding a flowchart according to an embodiment of the present invention is shown, such as... Figure 1 and Figure 2 As shown, an embodiment of the present invention provides a flowchart understanding method, including: Obtain the original flowchart data, and perform data parsing and data cleaning on the original flowchart data to obtain flowchart text data; A training dataset is constructed using the flowchart data, and the pre-trained visual multimodal large model is fine-tuned using the training dataset to obtain a large model of the process domain. The process domain big model is used to extract process entities and their logical relationships from process graph data, construct a structured process knowledge graph, and store the process knowledge graph in a graph database. The system retrieves user questions, searches for relevant entities and relationships in the graph database to obtain preliminary answers. The preliminary search results are then combined with the user's question to generate a prompt question. The prompt question is input into the process domain model, and the process domain model generates the final flowchart understanding answer.

[0027] Optionally, the original flowchart data is parsed and cleaned to obtain the flowchart text data as follows: Obtain the original flowchart data, perform data cleaning and parsing on the original flowchart data to obtain parsed data; Based on the parsed data, a question-answer pair set is generated; The question-and-answer pair set is subjected to noise reduction and compliance checks to obtain flowchart data.

[0028] In this embodiment, a preliminary question-answer pair set is automatically generated using a visual multimodal base model and an algorithm. During compliance checks, illegal or inappropriate content is filtered out according to preset rules. Finally, manual review can be performed to confirm the high quality of the data. After that, the resulting flowchart text data is saved to a database for later use.

[0029] Optionally, the pre-trained visual multimodal large model can be fine-tuned using the training dataset to obtain the process domain large model, specifically as follows: The flowchart data is converted into a training dataset in triplet form; Construct a large visual multimodal model and insert a low-rank adaptation module into the large visual multimodal model; The low-rank adaptation module is trained using the training dataset to achieve instruction fine-tuning and obtain a large model of the process domain.

[0030] In this embodiment, the structured information (such as nodes, connectors, icons, and text) in the flowchart can be effectively identified and parsed, and logical reasoning can be performed in conjunction with contextual semantics. This method uses visual multimodal analysis to achieve end-to-end understanding of the flowchart content. The visual multimodal large model significantly improves the accuracy and generalization ability of the model in professional flowchart understanding tasks by introducing domain knowledge constraints, graph structure-guided attention mechanisms, and transfer learning from a small number of labeled samples.

[0031] In this embodiment, the triplet format is an instruction-input-output format, specifically: Instruction: Please explain the approval path in this flowchart; Input: Flowchart image and related context; Output: Structured natural language answer or flowchart logic description; The data is formatted, deduplicated, and quality-checked to form a training set suitable for instruction fine-tuning. After fine-tuning the instructions using LoRA, the performance of the fine-tuned model is evaluated. If it does not meet the expected standard, the training strategy is adjusted (such as adding data, modifying the instruction template, optimizing the LoRA configuration) and fine-tuned again until the standard is met.

[0032] Optionally, constructing a structured process knowledge graph and storing the process knowledge graph in a graph database specifically involves: The flowchart data is input into the large process domain model to extract entities and relationships, and the extraction is assisted by a retrieval enhancement generation method to obtain extracted data in the form of triples. The extracted data is aggregated into triples to obtain a process knowledge graph. The process knowledge graph is stored in a graph database.

[0033] In this embodiment, the retrieval enhancement generation method is the GraphRAG framework or the LightRAG framework. Using the GraphRAG framework or the LightRAG framework, existing process knowledge fragments (such as terminology definitions, common patterns, and historical graph substructures) are dynamically retrieved during model inference, helping to improve the accuracy and consistency of entity and relation extraction. The large process domain model outputs a standard knowledge graph triple format: (head entity, relation, tail entity). All valid triples are aggregated to form a structured process knowledge graph oriented towards a specific business domain.

[0034] Optionally, the user's question can be retrieved from the graph database to find relevant entities and relationships, yielding a preliminary search result. The user question is obtained, and the named entity recognition module and relation extraction module are used to parse the user question to obtain the target entity and relation; Transform the target entities and relationships into query statements; The query statement is used to perform a multi-way search in the graph database to obtain multiple candidate answers; The candidate answers are sorted by relevance and then filtered to obtain the filtered answers; The filtered answers are then structured to obtain preliminary search results.

[0035] In this embodiment, based on the user's input question (e.g., what steps are required before financial approval in the reimbursement process?), target entities (such as "reimbursement process" and "financial approval") and relationships (such as "previous steps", "dependencies", and "triggering conditions") are extracted. Then, the extracted entities and relationships are transformed into a query form that the graph database can understand, such as Cypher queries, SPARQL, or vector search keywords.

[0036] Optionally, the preliminary search answer can be merged with the user's question to obtain the following prompt question: An enhanced prompt template is constructed, and the preliminary search answer is combined with the enhanced prompt template of the user's question input to obtain the prompt question.

[0037] Optionally, the final flowchart comprehension answer generated from the large model of the process domain is specifically as follows: The system receives the prompt questions and parses the technical terms and logical structure to obtain process semantic data. Internal reasoning is performed based on the process semantic data to generate a structured response; wherein the structured response is the answer to the flowchart comprehension question.

[0038] In this embodiment, a structured knowledge graph is first constructed based on the flowchart to represent the logical relationships, conditional judgments, and execution paths between nodes in the process. Subsequently, a multimodal large model is used to understand and identify the user's input questions, and reasoning is performed in conjunction with the semantic relationships in the knowledge graph, thereby achieving accurate question answering of the flowchart content.

[0039] Based on and Figure 1 Based on the same principle as the method shown, this embodiment of the invention also provides a flowchart understanding system, such as... Figure 3 As shown, it includes: The data preprocessing module is used to acquire raw flowchart data, perform data parsing and data cleaning on the raw flowchart data, and obtain flowchart text data. The model training module is used to construct a training dataset using the flowchart data, and to fine-tune the pre-trained visual multimodal large model using the training dataset to obtain a large model of the process domain. The process knowledge graph module is used to extract process entities and their logical relationships from process graph data using the large model of the process domain, construct a structured process knowledge graph, and store the process knowledge graph in a graph database. The preliminary retrieval module is used to obtain user questions, retrieve relevant entities and relationships in the graph database to obtain preliminary retrieval answers; The enhanced prompt module is used to merge the preliminary search answer with the user's question to obtain a prompt question; The answer generation module is used to input the prompt question into the process domain model and obtain the final flowchart understanding answer generated by the process domain model.

[0040] The flowchart understanding system of the present invention can execute the flowchart understanding method provided in the present invention. The implementation principle is similar. The actions performed by each module and unit in the flowchart understanding system of the present invention correspond to the steps in the flowchart understanding method of the present invention. For detailed functional descriptions of each module of the flowchart understanding system, please refer to the descriptions in the corresponding flowchart understanding methods shown above. They will not be repeated here.

[0041] The flowchart understanding system described above can be a computer program (including program code) running on a computer device, such as an application software; the application software can be used to execute the corresponding steps in the method provided in the embodiments of the present invention.

[0042] The modules described in the embodiments of the present invention can be implemented in software or hardware. The names of the modules are not, in some cases, limiting the scope of the module itself.

[0043] Based on the same principles as the methods shown in the embodiments of the present invention, the embodiments of the present invention also provide an electronic device, which may include, but is not limited to: a processor and a memory; the memory for storing computer programs; and the processor for executing the methods shown in any embodiment of the present invention by invoking the computer programs.

[0044] In one alternative embodiment, an electronic device is provided, such as Figure 4 As shown, Figure 4 The illustrated electronic device includes a processor and a memory. The processor and memory are connected, for example, via a bus. Optionally, the electronic device may also include a transceiver, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver is not limited to one unit, and the structure of this electronic device does not constitute a limitation on the embodiments of the present invention.

[0045] The memory stores application code (computer program) that executes the present invention, and its execution is controlled by a processor. The processor executes the application code stored in the memory to implement the content shown in the foregoing method embodiments.

[0046] Among these, electronic devices can also be terminal devices. Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0047] This invention provides a computer-readable storage medium storing a computer program that, when run on a computer, enables the computer to execute the corresponding content in the aforementioned method embodiments.

[0048] According to another aspect of the present invention, a computer program product or computer program is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various embodiments described above.

[0049] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the above embodiments.

[0050] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

Claims

1. A flowchart understanding method, characterized in that, The method includes: Obtain the original flowchart data, and perform data parsing and data cleaning on the original flowchart data to obtain flowchart text data; A training dataset is constructed using the flowchart data, and the pre-trained visual multimodal large model is fine-tuned using the training dataset to obtain a large model of the process domain. The process domain big model is used to extract process entities and their logical relationships from process graph data, construct a structured process knowledge graph, and store the process knowledge graph in a graph database. The system retrieves user questions, searches for relevant entities and relationships in the graph database to obtain preliminary answers. The preliminary search results are then combined with the user's question to generate a prompt question. The prompt question is input into the process domain model, and the process domain model generates the final flowchart understanding answer.

2. The flowchart understanding method according to claim 1, characterized in that, The original flowchart data is parsed and cleaned to obtain the flowchart text data as follows: Obtain the original flowchart data, perform data cleaning and parsing on the original flowchart data to obtain parsed data; Based on the parsed data, a question-answer pair set is generated; The question-and-answer pair set is subjected to noise reduction and compliance checks to obtain flowchart data.

3. The flowchart understanding method according to claim 1, characterized in that, By fine-tuning the pre-trained visual multimodal large model using the aforementioned training dataset, the resulting process domain large model is as follows: The flowchart data is converted into a training dataset in triplet form; Construct a large visual multimodal model and insert a low-rank adaptation module into the large visual multimodal model; The low-rank adaptation module is trained using the training dataset to achieve instruction fine-tuning and obtain a large model of the process domain.

4. The flowchart understanding method according to claim 1, characterized in that, Constructing a structured process knowledge graph and storing it in a graph database specifically involves: The flowchart data is input into the large process domain model to extract entities and relationships, and the extraction is assisted by a retrieval enhancement generation method to obtain extracted data in the form of triples. The extracted data is aggregated into triples to obtain a process knowledge graph. The process knowledge graph is stored in a graph database.

5. The flowchart understanding method according to claim 1, characterized in that, The user's question is retrieved from the graph database to find relevant entities and relationships, yielding the following preliminary search results: The user question is obtained, and the named entity recognition module and relation extraction module are used to parse the user question to obtain the target entity and relation; Transform the target entities and relationships into query statements; The query statement is used to perform a multi-way search in the graph database to obtain multiple candidate answers; The candidate answers are sorted by relevance and then filtered to obtain the filtered answers; The filtered answers are then structured to obtain preliminary search results.

6. The flowchart understanding method according to claim 1, characterized in that, The preliminary search results are then combined with the user's question to obtain the following prompt: An enhanced prompt template is constructed, and the preliminary search answer is combined with the enhanced prompt template of the user's question input to obtain the prompt question.

7. The flowchart understanding method according to claim 1, characterized in that, The specific answer to the question of how to generate the final flowchart from the large model of the process domain is as follows: The system receives the prompt questions and parses the technical terms and logical structure to obtain process semantic data. Internal reasoning is performed based on the process semantic data to generate a structured response; wherein the structured response is the answer to the flowchart comprehension question.

8. A flowchart understanding system, characterized in that, include: The data preprocessing module is used to acquire raw flowchart data, perform data parsing and data cleaning on the raw flowchart data, and obtain flowchart text data. The model training module is used to construct a training dataset using the flowchart data, and to fine-tune the pre-trained visual multimodal large model using the training dataset to obtain a large model of the process domain. The process knowledge graph module is used to extract process entities and their logical relationships from process graph data using the large model of the process domain, construct a structured process knowledge graph, and store the process knowledge graph in a graph database. The preliminary retrieval module is used to obtain user questions, retrieve relevant entities and relationships in the graph database to obtain preliminary retrieval answers; The enhanced prompt module is used to merge the preliminary search answer with the user's question to obtain a prompt question; The answer generation module is used to input the prompt question into the process domain model and obtain the final flowchart understanding answer generated by the process domain model.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method of any one of claims 1-7.

10. A computer storage medium, characterized in that, The computer storage medium stores a computer program, which, when executed by a processor, implements the method of any one of claims 1-7.

Images