The flowchart describes the text generation method, terminal device, and storage medium.

Abstract

This invention discloses a method for generating flowchart description text, a terminal device, and a storage medium. Addressing the difficulty in recognizing complex flowcharts, it proposes a flowchart recognition method based on segmentation and designs a combined framework for flowchart recognition and text generation, transforming flowcharts into semantic information. This helps process data and regulatory rules described only by flowcharts in intelligent supervision services, thereby achieving automated supervision of service processes.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a method for generating flowchart description text, a terminal device, and a storage medium. Background Technology

[0002] The advent of the big data era has freed users from geographical limitations, allowing them to apply for services from anywhere via the internet, while service providers also utilize digital means for processing and feedback. While the widespread adoption of digital services has brought convenience, it has also created several problems. These include whether service providers possess the relevant qualifications, whether the application process for permits is correct, and whether the services are legal and compliant. These issues can harm user rights and hinder social development. Therefore, regulating online services is crucial.

[0003] However, traditional methods relying on manual oversight are no longer adequate to meet the needs and challenges of digital service development. Against this backdrop, intelligent service oversight has emerged. Service oversight is industry-wide and platform-wide, encompassing laws and regulations, specific domain-specific norms, and the behavior of users and service providers. Many of these process specifications are described using flowcharts, such as license applications in the ride-hailing industry, and the scope of medical practice, correct medical procedures, and treatment protocols in the healthcare industry. Therefore, designing methods to translate flowcharts into natural language is crucial for overseeing these process-related behaviors.

[0004] Before deep learning, flowchart recognition relied on digital image processing techniques. It primarily involved manually identifying the features of flowchart elements, separating the text and graphic layers using the concept of connected components, and then classifying the graphic layer based on features to obtain category and location information. These methods depended on manually set features, meaning they became unusable when the dataset changed, exhibiting poor robustness and generalization ability.

[0005] With the development of deep learning technology, research has found that deep learning algorithms can self-learn to extract features from flowchart images, solving the problems of high cost and low accuracy associated with traditional methods that rely on manual feature design. This improves the accuracy and generalization ability of flowchart recognition. However, although deep learning-based flowchart recognition methods outperform traditional methods in various aspects and perform well on various visual tasks, they are not without their limitations and still have some problems. These include insufficient data, high annotation costs, high model training and inference time costs, and poor model interpretability. These issues can lead to the omission or recognition of incorrect information when recognizing complex flowcharts. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a method for generating flowchart description text, a terminal device and a storage medium that can better recognize complex flowcharts, in order to address the shortcomings of the prior art.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a method for generating flowchart description text, comprising the following steps:

[0008] S1. Train the object detection model using the Flowchart dataset;

[0009] S2. Separate the text from the flowchart to obtain the graphic layer and the flowchart text list;

[0010] Statistical analysis of the connected components of complex flowchart images in the Flowchart dataset yields a lower limit value, which is the threshold for evaluating flowchart complexity.

[0011] S3. Based on the flowchart complexity assessment threshold, divide the complex flowchart into two simple parts; the complex flowchart refers to a flowchart in which the number of connected components exceeds the flowchart complexity assessment threshold.

[0012] S4. Input the graphic layer generated in S3 into the trained target detection model to identify primitives and arrows. Combine the flowchart text list to obtain a set of primitives, a set of arrows, and a set of annotations. Separate the output of the target detection model from the graphic layer generated in step S2 to obtain a connecting line layer.

[0013] S5. Use the connecting line layer to obtain a set of straight line segments, match the straight line segments, and obtain a set of connecting lines;

[0014] S6. Match the set of graphic elements, the set of arrows, the set of annotations, and the set of connectors to obtain different paths in the flowchart;

[0015] S7. Generate the final flowchart descriptive text using the different path results.

[0016] This invention divides flowchart description generation into two main parts: flowchart image recognition and text generation. It aims to comprehensively include all information in the flowchart image, first detecting content and structural information from the image and then summarizing it into a text description. Addressing the difficulty in recognizing complex flowcharts, this invention proposes a flowchart recognition method based on segmentation principles and designs a combined framework of flowchart recognition and text generation. This transforms the flowchart into semantic information, aiding in the processing of data and regulatory rules described only by flowcharts in intelligent monitoring services, thus achieving automated monitoring of service processes.

[0017] Experiments revealed that existing flowchart recognition methods suffer from issues such as missed primitive identification and difficulty in recognizing connecting lines when dealing with complex flowcharts with numerous elements and intricate connections. Analysis showed that text and other elements negatively impact connecting line recognition. Furthermore, when using object detection models, flowcharts with many primitives generally exhibit missed primitive identification, while simpler flowcharts with fewer primitives show fewer such issues. Therefore, this invention designs a complex flowchart segmentation algorithm that divides a complex flowchart into two simpler flowcharts, improving the accuracy of primitive identification and resolving the problem of missed primitive identification. Simultaneously, a layer-by-layer flowchart separation process is designed: first, a text-image layer is created, separating the text from the flowchart to obtain the graphic layer; then, an object detection model is used to identify primitives and arrows in the graphic layer, separating primitives and arrows from the graphic layer to obtain the connecting line layer. At this point, only connecting lines remain in the image, making identification easier and resolving the problem of difficult connecting line recognition. Thus, this invention achieves the recognition of complex flowcharts.

[0018] In step S2, the specific implementation process of obtaining the flowchart complexity evaluation threshold includes: counting the number of connected components in all complex flowchart images in the Flowchart dataset, and selecting the minimum value as the flowchart complexity evaluation threshold. Obtaining the threshold facilitates subsequent judgments on flowchart segmentation, reduces unnecessary operations, and improves the efficiency of the entire method.

[0019] Step S3, specifically the process of dividing the complex flowchart into two simpler parts, includes: setting a scanning frame; scanning the flowchart from top to bottom using the scanning frame; and segmenting the complex flowchart to obtain two simpler images when the number of pixels in the current step within the scanning frame is the same as the number of pixels in the previous step and both are less than a set threshold r, the scanning frame is located in the middle of the flowchart, and the scanning frame does not contain any closed and symmetrical shapes. Complex flowchart segmentation solves the problem of missing recognition of complex flowcharts by dividing and conquering them, thus improving the accuracy of flowchart recognition.

[0020] In step S4, if the graphical layer of the input object detection model corresponds to two simple parts of a complex flowchart, then the two outputs of the object detection model need to be merged to obtain all the primitives and arrows of the complex flowchart. The primitives can provide the positional information of the main text of the flowchart, and the arrows can provide the directional information of the connecting lines of the flowchart, serving as an important basis for subsequent determination of connection relationships.

[0021] In step S4, the primitive set, arrow set, and annotation set are obtained. The specific implementation process of separating the output of the target detection model from the graphic layer generated in step S2 to obtain the connector layer includes: performing coordinate transformation on the output of the target detection model to obtain the arrow set; matching the coordinates of the primitive results after coordinate transformation with the flowchart text list to obtain the primitive set; the remaining text list after matching is the annotation set; and removing pixels at corresponding positions in the graphic layer generated in step S2 based on the detected primitives and arrow coordinates to obtain the connector layer. The primitive set, arrow set, and annotation set provide the content information of the flowchart, serving as the basis for subsequent connection relationship judgment; the connector layer only contains the connectors of the flowchart, solving the problem of difficult connector recognition and improving the accuracy of complex flowchart relationship recognition.

[0022] The specific implementation process of step S5 includes:

[0023] Identify all straight line segments in the connector layer to obtain a list of straight line segments;

[0024] Traverse the list of line segments, and determine whether the line segments intersect based on the fast repulsion experiment and the straddle experiment. Treat all intersecting lines as a connecting line to obtain the set of connecting lines.

[0025] Step S5 yields a set of connecting lines, providing flowchart structure information and serving as a crucial basis for subsequent determination of element connection relationships.

[0026] The specific implementation process of step S6 includes:

[0027] 1) Initialize Result, determine the starting element of the flowchart. Based on the coordinates, the top leftmost element with no predecessor node is the starting node of the flowchart, start. Execute step 2).

[0028] 2) Determine if all flag bits of SL, SE, SA, and SF are 1. If not, initialize the temporary result set TR, store the number of the element start in TR, and execute step 3); if yes, end; SL: Connector set;

[0029] SE: Primitive Set; SA: Arrow Set; SF: Annotation Set;

[0030] 3) Traverse SL, find the line connected to start, change the flag bit corresponding to the line in SL to 1, and execute step 4);

[0031] 4) Traverse the SF. If any note is similar to a line, change the flag corresponding to that note in the SF.

[0032] 1. Save the note number to TR and proceed to step 5); if it does not exist, proceed directly to step 5).

[0033] 5) Traverse SA. If any arrow is similar to the line, change the flag corresponding to the arrow in SA to 1 and proceed to step 6); otherwise, proceed directly to step 6.

[0034] 6) Traverse SE. If any element is connected to an arrow, change the flag of the element in SE to 1 and store the element's number in TR. Check if the element already exists in TR. If it does, it means that the branch is a cyclic branch. Store TR in Result and return to step 2). If it does not exist, proceed to step 7). If no element is connected to the arrow, store TR in Result and return to step 2.

[0035] 7) Traverse SL. If any line connects to an element, change the flag corresponding to the line in SL to 1 and return to step 4); otherwise, store TR in Result and return to step 2.

[0036] This step yields a set of all paths in the flowchart, containing the flowchart's content and connection information, which serves as input for generating the flowchart's descriptive text.

[0037] As an inventive concept, the present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory; the processor executes the computer program to implement the steps of the above method.

[0038] As an inventive concept, the present invention also provides a computer-readable storage medium having a computer program / instructions stored thereon; when the computer program / instructions are executed by a processor, they implement the steps of the above-described method.

[0039] As an inventive concept, the present invention also provides a computer program product, including a computer program / instructions; when the computer program / instructions are executed by a processor, they implement the steps of the above-described method.

[0040] Compared with the prior art, the beneficial effects of this invention are as follows: This invention addresses the problem of difficult recognition of complex flowcharts by proposing a flowchart recognition method based on the idea of ​​segmentation, and designs a combined framework of flowchart recognition and text generation to transform flowcharts into semantic information. This helps to process some data and regulatory rules that are only described by flowcharts in intelligent supervision services, thereby realizing automated supervision of service processes. Attached Figure Description

[0041] Figure 1 This is a flowchart of a method according to an embodiment of the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] Example 1

[0044] This embodiment provides a method for generating flowchart description text, including the following steps:

[0045] S1. Construct the Flowchart dataset and train the object detection model—RT-DETR;

[0046] S2. Text and graphics layering: Separate the text in the flowchart image to obtain the graphic layer and the flowchart text list;

[0047] S3. Obtain the flowchart complexity evaluation threshold, statistically analyze the connected components of complex flowchart images in the dataset, and obtain the lower limit value, i.e., the threshold, which is used to determine whether the flowchart image needs to be segmented.

[0048] S4. Flowchart segmentation: Based on the threshold generated in S3, a complex flowchart segmentation algorithm is used to divide the complex flowchart into two simple parts, while the simple flowchart does not need to be segmented.

[0049] S5. Flowchart Object Recognition: Input the graphic layer generated in S4 into the object detection model (RT-DETR model) trained in S1 to recognize primitives (i.e., graphic boxes in the flowchart) and arrows. If S4 produces two results, merge the model detection results; if S4 produces one result, no processing is required. Combine the text list generated in S2 to obtain a primitive set, an arrow set, and an annotation set (hint text near the flowchart connectors). Separate the results detected by the model from the graphic layer generated in S2 to obtain the connector layer.

[0050] S6. Flowchart Connector Recognition: The LSD straight line segment detection algorithm is used to detect the connector layer generated in S5 to obtain a set of straight line segments. Then, the connector recognition algorithm is used to match the straight line segments to obtain a set of connectors.

[0051] S7. Processing the flowchart recognition results: The path matching algorithm is used to match the set of graphic elements, arrows, annotations and the set of connecting lines generated in S5 and S6 to obtain different paths in the flowchart and store the results in XML format.

[0052] S8. Text generation: Input the XML result generated in S7 into GPT-4o to generate the final flowchart descriptive text.

[0053] The image-text layering mentioned in step S2 refers to dividing the flowchart image into an image layer and a text layer. PaddleOCR technology is used to identify the text information in the flowchart image, including its coordinates. Based on the identified coordinates, the text at the corresponding positions in the flowchart image is removed, resulting in an image layer containing only the flowchart. This removes the influence of text, allowing for better recognition of the flowchart structure in subsequent steps.

[0054] The segmentation threshold mentioned in step S3 is determined by counting the number of connected components in all complex flowchart images in the Flowchart dataset and selecting the minimum value as the threshold. If the number of connected components in a flowchart exceeds this threshold, the flowchart image is considered complex and needs to be segmented.

[0055] The complex process image segmentation algorithm described in step S4 aims to solve the problem of difficult recognition of complex process images. The specific steps are as follows:

[0056] (1) After the text and graphics are layered, the graphic layer is refined;

[0057] (2) Calculate the number of connected components in the flow chart;

[0058] (3) Based on the threshold t obtained from S3, if the threshold t is exceeded, the flowchart is considered to be complex and needs to be segmented.

[0059] (4) Define a sliding window-like bounding box with a width and height equal to the width of the flowchart image * p (a set threshold, not less than the average height of the primitives). Scan the flowchart from top to bottom using this bounding box with a step size of q. Segment the flowchart when the window position meets the following conditions: the number of pixels in the current step and the previous step are almost the same and both are less than r; the window position is located in the middle left or right of the flowchart; the window does not contain closed and symmetrical graphics. Finally, two simple images are obtained.

[0060] (5) If a suitable position cannot be found, then do not divide.

[0061] The flowchart object recognition described in step S5 mainly involves recognizing primitives and arrows in the image. Detecting primitives is to identify the main text in the flowchart, distinguishing the main body from annotations; detecting arrows is to determine direction, facilitating subsequent flowchart path recognition. The main idea is to input the two segmented images (or the unsegmented graphic layer) generated in S4 into the trained RT-DETR model to obtain detection results. The detection results of the segmented images need to undergo coordinate transformation so that their coordinates are relative to the original image, resulting in an arrow set. The detected primitive coordinates are matched with the text list obtained in S2 to obtain a primitive set containing the text and its position coordinates within the primitives. The remaining text list is the annotation set. Based on the detected primitive and arrow coordinates, pixels at corresponding positions in the graphic layer generated in S2 are removed to obtain the connection layer.

[0062] The connection cable identification described in step S6 is as follows:

[0063] (1) Use the LSD line segment detection algorithm to identify all line segments in the connection layer generated by S5 and obtain a list of line segments;

[0064] (2) Obtain the set of connecting lines according to the connecting line identification algorithm. Traverse the list of line segments, and determine whether the line segments intersect based on the fast repulsion experiment and the straddle experiment. Treat all intersecting lines as a connecting line to obtain the set of connecting lines.

[0065] The path matching algorithm described in step S7 mainly utilizes the primitive set SE, arrow set SA, annotation set SF obtained in S5, and connector set SL obtained in S6 to obtain all paths in the flowchart. The specific steps are as follows:

[0066] 1) Initialize Result, determine the starting element of the flowchart. Based on the coordinates, the topmost and leftmost element with no predecessor node is the starting node of the flowchart. Execute step 2).

[0067] 2) Determine if all flag bits of SL, SE, SA, and SF are 1. If not, initialize the temporary result set TR, store the number of the primitive start in TR, and execute step 3); if yes, the algorithm ends.

[0068] 3) Traverse SL, find the line connected to start, change the flag bit corresponding to the line in SL to 1 (initially all are 0), and execute step 4);

[0069] 4) Traverse SF. If any note is similar to a line, change the corresponding flag in SF to 1 (initially all are 0), and store the note's number in TR. Then execute step 5). If no note exists, execute step 5.

[0070] 5) Traverse SA. If any arrow is similar to the line, change the flag corresponding to the arrow in SA to 1 (initially all are 0), and execute step 6); if no arrow exists, execute step 6).

[0071] 6) Traverse SE. If any element is connected to an arrow, change the flag of the element in SE to 1 (initially all are 0), and store the element's number in TR. Then check if the element already exists in TR. If it does, it means the branch is a circular branch, so store TR in Result and execute step 2); if it does not exist, execute step 7); if no element is connected to the arrow, store TR in Result and execute step 2).

[0072] 7) Traverse SL. If any line connects to an element, change the flag corresponding to the line in SL to 1 (initially all are 0), and execute step 4); if no line exists, store TR in Result and execute step 2).

[0073] The XML format described in step S7 stores the obtained results in a structured XML format, mainly including graphic elements, annotation content, and paths. Based on the flowchart analysis, the following logical elements are abstracted, as shown in Table 1:

[0074] Table 1 Introduction to XML Element Tags

[0075]

[0076] Detailed explanations of related elements are shown below:

[0077] (1) Flowchart elements

[0078] Function: Root element, marking the start of the flowchart structure.

[0079] Rules: A flowchart has one and only one element; it can have multiple elements, notes, and roads.

[0080] Related examples:

[0081] <flowchart>

[0082] <element id='graphical element'>

[0083]

[0084] <note id='annotation'>

[0085]

[0086] <road id='path'>

[0087]

[0088] < / flowchart>

[0089] (2) attr element

[0090] Function: Stores the text in the flowchart.

[0091] Rule: The 'name' attribute with the value 'text' represents text. There must be a'string' tag representing the text content. The 'attr' element is usually a subordinate element of the 'element' or 'note' element, storing the text content of the graphic primitive or annotation.

[0092] Related example:

[0093] <attr name=’text’>

[0094] <string> Text content or annotations in graphic elements< / string>

[0095]

[0096] (3) 'element' element

[0097] Function: Represents a graphic primitive node in the flowchart.

[0098] Rule: The 'id' uniquely identifies; there must be 1 'attr' element.

[0099] Related example:

[0100] <element id=’Graphic primitive’>

[0101] <attr name=’text’>

[0102] <string> Text content in graphic elements< / string>

[0103]

[0104]

[0105] (4) 'note' element

[0106] Function: Represents an annotation near the connection line in the flowchart.

[0107] Rule: The 'id' uniquely identifies; there must be 1 'attr' element.

[0108] Related example:

[0109] <note id=’Graphic primitive’>

[0110] <attr name=’text’>

[0111] <string> Notes< / string>

[0112]

[0113]

[0114] (5) 'node' element

[0115] Function: Represents a node in the flowchart path.

[0116] Rule: The id is a unique identifier; the value of the content attribute is the id value of an element or a note, referring to a graphic element node or a note in the flow chart; in XML, the order of nodes by default represents the order of nodes in the path, from the start to the end of the path.

[0117] Related examples:

[0118] <node id=”content=’id value of element or note 1’>

[0119] <node id=”content=’id value of element or note 2’>

[0120] (6) road element

[0121] Function: Represents a single path in the flow chart.

[0122] Rule: The id is a unique identifier; if the path does not contain a loop structure, the path starts from the beginning of the flow chart to the end of the flow chart, describing a complete path; if the path contains a loop structure, the path ends at the node where the loop ends; contains the sub-tag node.

[0123] Related examples:

[0124] <road id=’non-loop path’>

[0125] <node id=”content=’start of flow chart’>

[0126] <node id=”content=’end of flow chart’>

[0127]

[0128] <road id=’loop path’>

[0129] <node id=”content=’start of flow chart’>

[0130] <node id=”content=’end of loop’>

[0131]

[0132] The text generation described in step S8 is to input the XML result obtained in S7 into GPT-4o to generate the final text result.

[0133] The flowchart description generation method based on deep learning designed in this invention uses PaddleOCR for image-text separation to segment complex flowcharts, followed by flowchart recognition, including flowchart object recognition, connector recognition, and structural relationship recognition. The generated results are stored in XML format, and finally, GPT-4o is used to generate natural language. The method of this invention can effectively recognize complex flowcharts and is highly practical.

[0134] The following detailed description of the embodiments of the present invention is based on a practical example:

[0135] First, input any flowchart. Then, use PaddleOCR to recognize the text and obtain a list of text results. After separating the text, obtain the graphic layer image.

[0136] Then, it is determined whether the flowchart image needs to be segmented. If segmentation is required, the graphic layer is divided into two parts by a segmentation algorithm and input into the RT-DETR model for detection. The coordinates of the detection results are further transformed relative to the original flowchart to obtain primitives, arrows, and annotation lists. The detection results are then separated from the graphic layer to obtain the connector layer. The LSD algorithm and connector recognition algorithm are used to obtain the connector list.

[0137] Next, based on the obtained result list, a path matching algorithm is used to obtain the flowchart structure and relationships, and the results are stored in XML format.

[0138] Finally, input the XML file into GPT-4o to obtain the final text result.

[0139] Regarding the deep learning-based flowchart description generation method proposed in this embodiment of the invention, the method of this embodiment of the invention was finally evaluated on the BERTS score on the Flowchart dataset. As shown in Table 1, the precision reached 93.53%, the recall reached 91.42%, and the F1 score (or balanced F score) was 92.15%.

[0140] Table 1 shows the evaluation results of the method of this invention on the BERT Score dataset in the embodiments of the present invention.

[0141]

[0142] Example 2

[0143] Embodiment 2 of the present invention provides a terminal device corresponding to Embodiment 1 above. The terminal device can be a processing device for a client, such as a mobile phone, a laptop, a tablet computer, a desktop computer, etc., to execute the method of the above embodiments.

[0144] The terminal device in this embodiment includes a memory, a processor, and a computer program stored in the memory; the processor executes the computer program in the memory to implement the steps of the method in Embodiment 1 described above.

[0145] In some implementations, the memory may be high-speed random access memory (RAM), and may also include non-volatile memory, such as at least one disk storage device.

[0146] In other implementations, the processor can be any type of general-purpose processor, such as a central processing unit (CPU) or a digital signal processor (DSP), and there is no limitation here.

[0147] Example 3

[0148] Embodiment 3 of the present invention provides a computer-readable storage medium corresponding to Embodiment 1 above, on which a computer program / instructions are stored. When the computer program / instructions are executed by a processor, they implement the steps of the method of Embodiment 1 above.

[0149] A computer-readable storage medium can be a tangible device that holds and stores instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination thereof.

[0150] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0151] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0152] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0153] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0154] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for generating flowchart description text, characterized in that, Includes the following steps: S1. Train the object detection model using the Flowchart dataset; S2. Separate the text from the flowchart to obtain the graphic layer and the flowchart text list; Statistical analysis of the connected components of complex flowchart images in the Flowchart dataset yields a lower limit value, which is the threshold for evaluating flowchart complexity. S3. Based on the flowchart complexity assessment threshold, divide the complex flowchart into two simple parts; the complex flowchart refers to a flowchart in which the number of connected components exceeds the flowchart complexity assessment threshold. S4. Input the graphic layer generated in S3 into the trained target detection model to identify primitives and arrows. Combine the flowchart text list to obtain a set of primitives, a set of arrows, and a set of annotations. Separate the output of the target detection model from the graphic layer generated in step S2 to obtain a connecting line layer. S5. Use the connecting line layer to obtain a set of straight line segments, match the straight line segments, and obtain a set of connecting lines; S6. Match the set of graphic elements, the set of arrows, the set of annotations, and the set of connectors to obtain different paths in the flowchart; S7. Generate the final flowchart descriptive text using different path results; The specific implementation process of step S6 includes: 1) Initialize Result, determine the starting element of the flowchart. Based on the coordinates, the topmost and leftmost element without a predecessor node is the starting node of the flowchart (start). Execute step 2). 2) Determine if all flag bits of SL, SE, SA, and SF are 1. If not, initialize the temporary result set TR, store the number of the element start in TR, and execute step 3); if yes, end; SL is the set of connectors, SE is the set of elements, SA is the set of arrows, and SF is the set of annotations; 3) Traverse SL, find the line connected to start, change the flag bit of the line in SL to 1, and execute step 4). 4) Traverse SF. If any note is similar to a line, change the corresponding flag in SF to 1 and store the note's number in TR. Then proceed to step 5. If no note exists, proceed directly to step 5. 5) Traverse SA. If any arrow is similar to a line, change the flag corresponding to the arrow in SA to 1 and proceed to step 6); otherwise, proceed directly to step 6). 6) Traverse SE. If any element is connected to an arrow, change the flag of the element in SE to 1 and store the element's number in TR. Check if the element already exists in TR. If it does, it means the current branch is a loop branch, so store TR in Result and return to step 2). If it does not exist, proceed to step 7). If no element is connected to the arrow, store TR in Result and return to step 2). 7) Traverse SL. If any line connects to an element, change the flag corresponding to the line in SL to 1 and return to step 4. If no line exists, store TR in Result and return to step 2.

2. The method for generating flowchart description text according to claim 1, characterized in that, In step S2, the specific implementation process of obtaining the flowchart complexity evaluation threshold includes: counting the number of connected components of all complex flowchart images in the Flowchart dataset, and selecting the minimum value as the flowchart complexity evaluation threshold.

3. The method for generating flowchart description text according to claim 1, characterized in that, In step S3, the specific implementation process of dividing the complex flowchart into two simple parts includes: setting a scanning frame, using the scanning frame to scan the flowchart from top to bottom, and when the number of pixels in the current step within the scanning frame is the same as the number of pixels in the previous step and both are less than the set threshold r, and the scanning frame is located in the middle of the flowchart, and the scanning frame does not contain closed and symmetrical graphics, the complex flowchart is divided to obtain two simple images.

4. The method for generating flowchart description text according to claim 1, characterized in that, In step S4, if the graphical layer of the input target detection model corresponds to two simple parts of the complex flowchart, then the two output results of the target detection model are merged to obtain all the primitives and arrows of the complex flowchart.

5. The method for generating flowchart description text according to claim 1, characterized in that, In step S4, the primitive set, arrow set, and annotation set are obtained. The specific implementation process of separating the output of the target detection model from the graphic layer generated in step S2 to obtain the connecting line layer includes: performing coordinate transformation on the output of the target detection model to obtain the arrow set; matching the coordinates of the primitive results after coordinate transformation with the flowchart text list to obtain the primitive set; the remaining text list after matching is the annotation set; and removing the pixels at the corresponding positions in the graphic layer generated in step S2 according to the detected primitives and arrow coordinates to obtain the connecting line layer.

6. The method for generating flowchart description text according to claim 1, characterized in that, The specific implementation process of step S5 includes: Identify all straight line segments in the connector layer to obtain a list of straight line segments; Traverse the list of line segments, and determine whether the line segments intersect based on the fast repulsion experiment and the straddle experiment. Treat all intersecting lines as a connecting line to obtain the set of connecting lines.

7. A terminal device, comprising a memory, a processor, and a computer program stored in the memory; characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program / instructions stored thereon; characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1 to 6.

9. A computer program product, comprising a computer program / instructions; characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the method of any one of claims 1 to 6.

Images