Flow chart intelligent analysis method based on information flow understanding

Abstract

The invention discloses a flow chart intelligent analysis method based on information flow understanding. The method comprises the following steps: step 1, carrying out component identification and segmentation on a flow chart; 2, after component identification and segmentation in the step 1 are completed, OCR text identification is carried out; step 3, completing component identification and segmentation in the step 1 and OCR text identification in the step 2; step 4, performing component identification, layout analysis and text extraction on the flow charts in the step 1 to the step 3; and 5, based on an analysis result in the step 4, automatically generating an accurate and complete structured text by adopting a large language model. According to the flow chart intelligent analysis method based on information flow understanding, a structured text with more flow chart readability is output for a user through a large language model and an RAG retrieval enhancement generation technology.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and more particularly to a method for intelligently analyzing flowcharts based on understanding of information flow. Background Art

[0002] With the continuous advancement of machine learning technology, artificial intelligence has made significant progress in flowchart recognition. Previous flowchart recognition methods have been limited in their ability to recognize diagram symbols such as arrows, often relying solely on the overlap of bounding boxes between different objects. In 2021, the renowned journal IJDAR published an article titled "Arrow R-CNN for Handwritten Diagram Recognition." The authors incorporated an arrow keypoint predictor into the Faster R-CNN deep learning framework and, based on this, employed novel data augmentation and post-processing methods to improve the performance of the AI ​​model.

[0003] In many businesses, drawing flowcharts is an essential step in analyzing business processes. Despite the availability of numerous professional process modeling tools online, most process model development begins with hand-drawn sketches on a blank sheet of paper. In the same year, the renowned ICDAR conference published a paper titled "Sketch2BPMN: Automatic Recognition of Hand-drawn BPMN Models." The authors designed a BPMN model to recognize hand-drawn flowcharts.

[0004] Most hand-drawn flowcharts are too sloppy to be recognized, often confusing arrows with other diagram structures. That same year, the paper "DiagramNet: Hand-Drawn Diagram Recognition Using Visual Arrow-Relation Detection" was published at ICDAR. The authors designed a neural network architecture that includes shape detection, shape degree detection, candidate edge generation, edge prediction, and edge optimization. This new architecture achieved significantly higher recognition accuracy than previous results.

[0005] In addition to conventional flowcharts, a paper specifically addressing finite automata recognition, "Using YOLO Network for Automatic Processing of Finite Automata Images with Application to Bit-Strings Recognition," was presented at the ACM DocEng conference. The authors utilized the YOLO network architecture and several post-processing pipeline methods to achieve finite automata recognition and designed an application to measure the conversion accuracy of finite automata identified by detecting input bit strings.

[0006] In addition to hand-drawn flowcharts, the article "FR-DETR: End-to-End Flowchart Recognition With Precision and Robustness" published on IEEE Access uses machine-generated flowcharts as a training set, integrating object detection and line segment detection to build an end-to-end recognition model, achieving higher accuracy and better results in the field of machine-generated image recognition.

[0007] Currently, existing flowchart recognition technologies have solved some problems in research directions such as arrow recognition, hand-drawn image recognition, and machine image recognition. However, these technologies are often limited to identifying and generating data structures corresponding to flowcharts. This field still faces the challenge of whether artificial intelligence can deeply understand the content logic of flowcharts, and there is a lack of attempts to describe the recognized flowcharts using structured text language. Summary of the Invention

[0008] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide an intelligent flowchart parsing method based on information flow understanding, which optimizes and improves the model structure so that it has the ability to accurately identify, segment and understand the content of the flowchart.

[0009] To achieve the above-mentioned purpose, the present invention provides the following technical solution: a flowchart intelligent parsing method based on information flow understanding, characterized in that it includes the following steps:

[0010] Step 1: Identify and segment components of the flowchart;

[0011] Step 2: After component identification and segmentation in step 1, perform OCR text recognition;

[0012] Step 3: After the component recognition and segmentation in step 1 and the OCR text recognition in step 2, the relative position relationship of each graphic element in the flowchart is further analyzed through layout analysis;

[0013] Step 4: After the flowchart component identification, layout analysis, and text extraction in steps 1 to 3, the text information extracted from the flowchart is further analyzed and semantically understood using natural language processing technology;

[0014] Step 5: Based on the analysis results of step 4, a large language model is used to automatically generate accurate and complete structured text to describe the overall structure of the flowchart, the logical relationship between each process node, and the key information in the flowchart to complete the intelligent analysis of the flowchart.

[0015] As a further improvement of the present invention, the specific steps of identifying and segmenting the components of the flowchart in step 1 are as follows:

[0016] In step one, a mask segmentation algorithm is used to accurately identify various components in the flowchart;

[0017] In steps one and two, the mask segmentation algorithm is used to effectively segment the identified graphic elements from the background image, laying the foundation for subsequent analysis.

[0018] As a further improvement of the present invention, the mask segmentation algorithm trains the model through a large amount of hand-drawn or machine-generated flowchart data in the training set, so that the model can recognize various components in the flowchart and mark their bounding boxes and categories.

[0019] As a further improvement of the present invention, the specific steps of OCR text recognition in step 2 are as follows:

[0020] Step 21: further process each segmented graphic component to determine whether there is text content inside each component;

[0021] Step 22: The text area contained in each component is automatically identified through the deep learning-based OCR model. After determining the text area, the OCR model begins to recognize the characters in the area and converts the characters in the image into corresponding text information.

[0022] Among them, for different components, the OCR model will first identify the text layout and distinguish between text and graphic parts. For characters in complex flowcharts or handwritten charts, pre-processing will be performed to improve detection accuracy. As a further improvement of the present invention, the specific steps for in-depth analysis of the relative position relationship of each graphic element in the flowchart in step three are as follows:

[0023] Step 31: extract the coordinate position and size information of each component and construct an overall geometric layout diagram;

[0024] Step 32: Identify potential logical relationships by calculating the relative distances between components to preliminarily determine the logical order of components;

[0025] Step 33: Detect arrows based on key points. After identifying the arrows, the model further detects the specific positions of the arrow's head and tail. By calculating the direction vectors of the arrow's head and tail, it accurately determines the arrow's directionality and further determines the information flow in the process to complete the layout analysis.

[0026] In steps three and four, after completing the layout analysis, the model will integrate the spatial position, connection relationship, and arrow directionality of each component into a graph data structure.

[0027] As a further improvement of the present invention, the specific method of performing in-depth analysis and semantic understanding of the text information extracted from the flowchart in step four is to perform semantic understanding through a semantic understanding model, so as to analyze the relevant text content such as nodes, arrows, and decision conditions in the flowchart, and explore the business logic and semantic meaning behind them, thereby providing basic support for the automated optimization and intelligent analysis of the process.

[0028] As a further improvement of the present invention, in step four, before semantic understanding, the text information extracted from the flowchart is first preprocessed, and the various formats and language expressions contained in the text in the flowchart are uniformly normalized, including text normalization, word segmentation and part-of-speech tagging, and stop word filtering.

[0029] As a further improvement of the present invention, the semantic understanding model includes:

[0030] Named Entity Recognition module, which is used to identify important entities from text by extracting text features, such as process step names, decision conditions, and time nodes;

[0031] The syntactic analysis module uses a neural network-based parser to break down the grammatical structure of sentences and determine the subject-verb-object structure of the sentence, thereby understanding the overall semantic logic of the text. This module is particularly useful when dealing with complex decision conditions or process step descriptions, clarifying the logical relationships between different sentences.

[0032] The semantic role labeling module is used to determine the semantic role of each word in the text through a deep learning model. As a further improvement of the present invention, the specific steps of automatically generating accurate and complete structured text using a large language model in step 5 are as follows:

[0033] Step 51: Generate text describing each branch path based on the directionality of the arrow;

[0034] Step 52: After the text generated in step 1 is initially generated, it undergoes certain post-processing. The post-processing process includes language fluency checking, logical consistency verification, and format adjustment to complete the automatic generation of the text.

[0035] Beneficial effects of the present invention:

[0036] (1) This invention combines multiple technologies, including flowchart component segmentation and recognition, OCR text recognition, and layout analysis, to accurately identify, segment, and extract various graphical components within flowcharts. Compared to traditional methods, this invention is more accurate in automatically extracting and recognizing the textual information of each component within a flowchart, laying a solid foundation for subsequent text feature extraction and semantic understanding.

[0037] (2) The AI ​​model constructed by this invention can more effectively understand flowchart logic and automatically generate structured text descriptions of key flowchart content based on user needs. This innovation not only improves the accuracy of text recognition but also efficiently completes flowchart logic parsing and information output, meeting the needs of intelligent flowchart processing in different application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a flow chart of the intelligent analysis method of flowchart based on information flow understanding of the present invention. DETAILED DESCRIPTION

[0039] The present invention will be further described below with reference to the embodiments shown in the accompanying drawings.

[0040] Reference Figure 1 As shown, a flowchart intelligent parsing method based on information flow understanding of this embodiment includes the following steps:

[0041] 1) Component identification and segmentation:

[0042] First, the present invention uses a mask segmentation algorithm (such as the YOLO series of deep learning models) to accurately identify various components in the flowchart (including but not limited to rectangles, arrows, ellipses and other graphic elements). The YOLO (You Only LookOnce) model is very suitable for the component recognition task of the present invention because it has the ability to achieve real-time detection and recognition in a single neural network structure. The YOLO network divides the input image into multiple grids, each grid is responsible for predicting the position and category of a part of the graphic elements. In our application, the network is trained to recognize elements such as rectangles, arrows, ellipses, etc. in the flowchart. Through a large amount of hand-drawn or machine-generated flowchart data in the training set, the model can recognize various components in the flowchart and mark their bounding boxes and categories.

[0043] Subsequently, an algorithm is used to effectively segment the identified graphic elements from the background image, laying the foundation for subsequent analysis. The goal of using mask segmentation technology is to segment these graphic elements from the original background image by generating a binary mask for each component. The 1 in the mask represents the component part, and the 0 represents the background part. This segmentation process can lay the foundation for subsequent analysis and structuring work, because in the flowchart, the clearly segmented graphic elements will be used for further connection relationship analysis. After the segmentation is completed, the image processing algorithm is used to identify the connecting line segments between the graphic elements in the flowchart. For the arrow, an important connection symbol in the flowchart, the present invention draws on the idea of ​​Arrow R-CNN and adds a key point detection module for the arrow head and tail. After detecting the arrow head and tail, by calculating the direction vector between the two points, the direction of the arrow can be accurately analyzed, thereby determining the direction of the information flow in the flowchart, laying the foundation for subsequent work.

[0044] 2)OCR text recognition:

[0045] After component identification and segmentation are completed, the present invention uses optical character recognition (OCR) technology to extract text content from the segmented components to achieve the conversion of graphic information into text information. OCR technology is a technology used to recognize and extract characters from images. Its core task is to convert unstructured image data into editable and searchable text information. In a flowchart, various components usually contain important text content, such as descriptions of process nodes, decision conditions, etc. These texts are crucial for the parsing and understanding of the flowchart. Therefore, the extraction of these texts through OCR technology can greatly improve the recognition accuracy and application value of the flowchart.

[0046] The present invention further processes each segmented graphic component to determine whether there is text content within each component. Through deep learning-based OCR technology, the model can automatically identify the text area contained in each component and convert the characters in the image into corresponding text information. For different components, the OCR model will first identify the text layout and distinguish between text and graphic parts. For characters in complex flowcharts or handwritten charts, the system will perform preprocessing to improve detection accuracy, such as enhancing character contrast and eliminating noise through image processing techniques such as grayscale and binarization. After determining the text area, the OCR model begins to recognize the characters in that area. Deep learning algorithms can convert complex fonts and handwritten characters into corresponding text information. By using a trained large-scale character recognition model, the system can accurately recognize characters in various languages ​​and digital symbols. At the same time, the text information recognized by OCR is automatically associated with the corresponding node in the graphical data structure, and the model will bind the recognized text to the position and category information corresponding to each node.

[0047] Because characters in flowcharts may contain specific business terms or professional symbols, the OCR system performs text correction and post-processing after recognizing the text to ensure that the converted text has correct semantics. The system can use a dictionary or business terminology database to correct and optimize the OCR-recognized text, ensuring the final extracted text is highly accurate.

[0048] 3) Layout analysis:

[0049] After identifying and segmenting flowchart components and performing OCR text recognition, the present invention further analyzes the relative positional relationships of the various graphical elements in the flowchart through layout analysis. The core goal of layout analysis is to understand and determine the logical order between components by analyzing the spatial layout of graphical elements, thereby accurately revealing the flow of information in the flowchart. In particular, the arrows in the flowchart are key symbols connecting the components. Their connection relationship and directionality are crucial for understanding the entire flowchart. Therefore, the present invention provides detailed processing for the analysis and direction determination of arrows.

[0050] In a flowchart, different graphic elements convey a specific logical order through relative spatial layout. In order to understand the arrangement and connection relationship of these elements, the present invention first extracts the coordinate position and size information of each component and constructs an overall geometric layout diagram. By analyzing this information, the arrangement order of each component can be preliminarily determined. For example, the starting point of a flowchart is usually located at the top or left, and the information flow gradually flows to the bottom or right. The system will make a preliminary layout inference based on these spatial features. In addition, the present invention identifies potential logical relationships by calculating the relative distance between each component. Components that are relatively close are more likely to have a logical connection relationship, while components that are relatively far away are considered to belong to independent process modules. Through this layout analysis based on geometric information, the logical order of each component can be preliminarily determined.

[0051] The existence and direction of arrows in a flowchart are the core elements that indicate the direction of information flow. The present invention adopts a key point-based arrow detection technology. After identifying the arrow, the model will further detect the specific position of the head and tail of the arrow. By calculating the direction vector of the arrow head and tail, the directionality of the arrow can be accurately judged, and then the direction of information flow in the process can be determined. For example, if the tail of the arrow is connected to a rectangular box, the rectangular box is considered to be the source of the information, and the component pointed to by the head of the arrow is considered to be the target component of the information. Based on the results of the arrow directionality analysis, the present invention can accurately infer the logical relationship between the components in the flowchart. For each process node, the system will determine whether it is an input or output of information through layout analysis. For example, when an arrow points from a decision node to multiple options, the system can recognize that this is a branch decision and determine the execution path of each branch by parsing the arrow.

[0052] After completing the layout analysis, the model integrates the spatial positions of each component, the connection relationships, and the directionality of arrows into a graph data structure. This data structure represents the logical structure of the flowchart in the form of nodes and edges. Nodes represent graphical components (such as rectangles or ellipses), while edges represent the connections between components (determined by the direction of the arrows). This data structure provides a complete and structured representation of the flowchart's logical sequence and information flow.

[0053] 4) Semantic understanding:

[0054] After completing flowchart component identification, layout analysis, and text extraction, the present invention further uses natural language processing (NLP) technology to conduct in-depth analysis and semantic understanding of the text information extracted from the flowchart. Semantic understanding is an important component of the model. Its core goal is to analyze the relevant text content in the flowchart, such as nodes, arrows, and decision conditions, to uncover the business logic and semantic meaning behind them, thereby providing basic support for automated process optimization and intelligent analysis.

[0055] Before performing semantic understanding, the system first preprocesses the text information extracted from the flowchart. Because the text in the flowchart may contain a variety of formats and language expressions, such as natural language descriptions, phrases, and symbols, it requires unified normalization of this content, including text normalization, word segmentation and part-of-speech tagging, and stop word filtering.

[0056] The natural language processing techniques of the semantic understanding model include named entity recognition, syntactic analysis, and semantic role labeling. Named entity recognition refers to the identification of important entities in text, such as process step names, decision conditions, and time nodes, through text feature extraction. Syntactic analysis involves the model using a neural network-based parser to break down the grammatical structure of sentences and determine the subject-verb-object structure, thereby understanding the overall semantic logic of the text. This is particularly important when dealing with complex decision conditions or process step descriptions, clarifying the logical relationships between different sentences. Semantic role labeling involves the model using a deep learning model to determine the semantic role of each word in the text, such as "goal" and "action." These three tasks are independent but complementary, and they work together to achieve in-depth analysis and understanding of text.

[0057] 5) Text Generation:

[0058] Based on these analysis results, this paper uses a large language model (LLM) to automatically generate accurate and complete structured text that describes the overall structure of the flowchart, the logical relationships between each process node, and the key information in the flowchart. The goal of text generation is to convert the graphical information into natural language text, making it easier for users to read, understand, and further analyze the flowchart content.

[0059] These large language models are pre-trained on large amounts of natural language data and possess powerful text generation capabilities, capable of generating coherent, natural text descriptions based on the input structured data. The selection of a language model should consider its performance in handling multiple languages, specialized terminology, and complex process descriptions. Specifically, LLMs embed and contextualize the input structured data to generate natural language descriptions that conform to the process logic. For example, when processing a flowchart containing multiple branches or conditional judgments, the language model can generate text describing each branch path based on the directionality of the arrows. Furthermore, the model can flexibly generate different text styles and levels of detail to meet the needs of different application scenarios.

[0060] After initial generation, the generated text undergoes post-processing. This includes checking for linguistic fluency, verifying logical consistency, and adjusting formatting. The model uses automated tools or manual verification to ensure that the generated text is not only grammatically correct but also clearly structured and logically sound. Furthermore, if the generated text contains specialized terminology or domain-specific language, the model further optimizes its use to ensure that the text complies with the standards and requirements of the target domain.

[0061] Compared to traditional flowchart description methods, the text generation technology of this invention leverages the powerful generation capabilities of LLM to achieve automated, highly accurate flowchart text description. Furthermore, compared to the traditional LLM large language model, the flowchart-oriented data preprocessing of this invention effectively improves the relevance and accuracy of the generated text. Whether it's a hand-drawn or machine-generated complex flowchart, the model can accurately generate structured text based on the parsed information flow.

[0062] In summary, the flowchart intelligent parsing method based on information flow understanding of this embodiment has several significant advantages over the existing best technologies. First, through the powerful generation capability of the LLM large language model, the present invention achieves efficient and accurate flowchart text recognition, extraction and understanding functions that are difficult to achieve with existing technologies. At the same time, by integrating natural language processing (NLP) technology, the present invention can not only parse text content, but also understand and infer the logical relationships and business meanings in flowcharts, and can automatically generate structured text descriptions, thereby providing users with more intelligent flowchart interpretation. Compared with existing technologies, the present invention has significantly improved the degree of automation, recognition accuracy and information output efficiency, and is particularly suitable for processing diverse and complex flowcharts.

[0063] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for intelligently parsing flowcharts based on understanding information flow, characterized by: The steps include: Step 1: Identify and segment components of the flowchart; Step 2: After component identification and segmentation in step 1, perform OCR text recognition; Step 3: After the component recognition and segmentation in step 1 and the OCR text recognition in step 2, the relative position relationship of each graphic element in the flowchart is further analyzed through layout analysis; Step 4: After the flowchart component identification, layout analysis, and text extraction in steps 1 to 3, the text information extracted from the flowchart is further analyzed and semantically understood using natural language processing technology; Step 5: Based on the analysis results of step 4, a large language model is used to automatically generate accurate and complete structured text to describe the overall structure of the flowchart, the logical relationship between each process node, and the key information in the flowchart to complete the intelligent analysis of the flowchart.

2. The intelligent flow chart analysis method based on information flow understanding according to claim 1 is characterized by: The specific steps for component identification and segmentation of the flowchart in step 1 are as follows: In step one, a mask segmentation algorithm is used to accurately identify various components in the flowchart; In steps one and two, the mask segmentation algorithm is used to effectively segment the identified graphic elements from the background image, laying the foundation for subsequent analysis.

3. The intelligent flow chart analysis method based on information flow understanding according to claim 2 is characterized by: The mask segmentation algorithm trains the model using a large amount of hand-drawn or machine-generated flowchart data in the training set, thereby enabling the model to identify various components in the flowchart and mark their bounding boxes and categories.

4. The intelligent flow chart analysis method based on information flow understanding according to claim 1, 2 or 3, characterized in that: The specific steps of OCR text recognition in step 2 are as follows: Step 21: further process each segmented graphic component to determine whether there is text content inside each component; Step 22: The text area contained in each component is automatically identified through the deep learning-based OCR model. After determining the text area, the OCR model begins to recognize the characters in the area and converts the characters in the image into corresponding text information. For different components, the OCR model will first identify the text layout and distinguish between text and graphic parts. For characters in complex flowcharts or handwritten charts, preprocessing will be performed to improve detection accuracy.

5. The intelligent flow chart analysis method based on information flow understanding according to claim 1, 2 or 3, characterized in that: The specific steps for in-depth analysis of the relative position relationship of each graphic element in the flowchart in step 3 are as follows: Step 31: extract the coordinate position and size information of each component and construct an overall geometric layout diagram; Step 32: Identify potential logical relationships by calculating the relative distances between components to preliminarily determine the logical order of components; Step 33: Detect arrows based on key points. After identifying the arrows, the model further detects the specific positions of the arrow's head and tail. By calculating the direction vectors of the arrow's head and tail, it accurately determines the arrow's directionality and further determines the information flow in the process to complete the layout analysis. In steps three and four, after completing the layout analysis, the model will integrate the spatial position, connection relationship, and arrow directionality of each component into a graph data structure.

6. The intelligent flow chart analysis method based on information flow understanding according to claim 1, 2 or 3, characterized in that: The specific method of performing in-depth analysis and semantic understanding of the text information extracted from the flowchart in step 4 is to perform semantic understanding through a semantic understanding model to analyze the relevant text content such as nodes, arrows, and decision conditions in the flowchart, and to explore the business logic and semantic meaning behind them, thereby providing basic support for the automated optimization and intelligent analysis of the process.

7. The intelligent flow chart analysis method based on information flow understanding according to claim 6 is characterized by: In step 4, before semantic understanding, the text information extracted from the flowchart is first preprocessed, and the various formats and language expressions contained in the text in the flowchart are uniformly normalized, including text normalization, word segmentation and part-of-speech tagging, and stop word filtering.

8. The intelligent flow chart analysis method based on information flow understanding according to claim 7 is characterized by: The semantic understanding model includes: Named Entity Recognition module, which is used to identify important entities from text by extracting text features, such as process step names, decision conditions, and time nodes; The syntactic analysis module uses a neural network-based parser to break down the grammatical structure of sentences and determine the subject-verb-object structure of the sentence, thereby understanding the overall semantic logic of the text. This module is particularly useful when dealing with complex decision conditions or process step descriptions, clarifying the logical relationships between different sentences. The semantic role labeling module is used to determine the semantic role of each word in the text through a deep learning model.

9. The intelligent flow chart analysis method based on information flow understanding according to claim 1, 2 or 3, characterized in that: The specific steps for automatically generating accurate and complete structured text using a large language model in step 5 are as follows: Step 51: Generate text describing each branch path based on the directionality of the arrow; Step 52: After the text generated in step 1 is initially generated, it undergoes certain post-processing. The post-processing process includes language fluency checking, logical consistency verification, and format adjustment to complete the automatic generation of the text.

Images