Flow chart description text generation method, terminal equipment and storage medium

Abstract

The invention discloses a flow chart description text generation method, a terminal device and a storage medium, provides a flow chart recognition method based on a segmentation thought aiming at the problem that a complex flow chart is difficult to recognize, designs a combined framework of flow chart recognition and text generation, converts the flow chart into semantic information, and improves the recognition efficiency of the flow chart. The method helps to process some data and supervision rules which are only described by a flow chart during intelligent supervision service, and achieves the automatic supervision of the service flow.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method for generating flow chart description text, a terminal device and a storage medium. Background Art

[0002] With the advent of the big data era, users are no longer restricted by geographical location and can apply for services through the Internet at any location. Service providers also use digital means to accept and respond. Although the popularization of digital services has provided convenience for people, there are also some problems. For example, whether the service provider has relevant qualifications, whether the application process is correct, whether the service is legal and compliant, etc. These problems will damage the rights and interests of users and affect social development. Therefore, it is very important to regulate online services.

[0003] However, the traditional approach that relies on manual supervision can no longer adapt to the needs and challenges of the development of digital services. In this context, intelligent service supervision has emerged. Service supervision is conducted across the entire industry and platform. The objects that need to be supervised include laws and regulations, specific field and specific things, and the behavior of users and service providers. A large number of process specifications are described in the form of process images, such as the application for qualification certificates in the online car-hailing industry, the scope of practice in the medical industry, the correct medical treatment process, and the treatment process. To supervise such process behaviors, it is crucial to design a method for converting process images into natural language.

[0004] Before deep learning, flowchart recognition relied on digital image processing technology, which mainly relied on manual statistics of flowchart element features, first separated the text layer and the graphic layer through the idea of ​​connected domain, and then classified the graphic layer according to features to obtain category and location information. This type of method relies on manually set features. When the data set changes, the same method cannot be applied, and the robustness and generalization ability are poor.

[0005] With the development of deep learning technology, research has found that deep learning algorithms can be used to self-learn and extract process image features, solving the problem of high cost and low accuracy of manually designed features in traditional methods, and improving the accuracy and generalization ability of flowchart recognition. However, although the flowchart recognition method based on deep learning outperforms traditional methods in all aspects and performs well in various visual tasks, it is not perfect and still has some problems. For example, insufficient data volume, high annotation cost, high model training and reasoning time cost, poor model interpretability, etc., which leads to missing or identifying wrong information when recognizing complex flowcharts. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a method for generating a flowchart description text, a terminal device and a storage medium to better identify complex flowcharts in view of the deficiencies in the prior art.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is: a method for generating a flow chart description text, comprising the following steps:

[0008] S1. Train the target detection model using the Flowchart dataset.

[0009] S2, separating the text in the process image to obtain a graphic layer and a flowchart text list;

[0010] Statistically analyze the connected domains of complex process images in the Flowchart dataset to obtain the lower limit value, which is the threshold for judging the complexity of the flowchart;

[0011] S3. According to the flowchart complexity evaluation threshold, the complex flowchart is divided into two simple parts; the complex flowchart refers to a flowchart whose number of connected domains exceeds the flowchart complexity evaluation threshold;

[0012] S4, inputting the graphic layer generated by S3 into the trained target detection model, identifying the graphic elements and arrows, combining the flowchart text list, obtaining a graphic element set, an arrow set and an annotation set, separating the output result of the target detection model from the graphic layer generated by step S2, and obtaining a connection line layer;

[0013] S5. Obtain a set of straight line segments using the connection line layer, match the straight line segments, and obtain a set of connection lines;

[0014] S6, matching the graphic element set, arrow set, annotation set, and connection line set to obtain different paths in the flowchart;

[0015] S7. Generate the final flowchart descriptive text using different path results.

[0016] The present invention divides the generation of flowchart description into two parts: process image recognition and text generation. All information in the process image is included as comprehensively as possible, that is, the content and structure information are first detected from the image, and then the description is summarized into a text. In view of the problem that complex flowcharts are difficult to recognize, a flowchart recognition method based on the idea of ​​segmentation is proposed, and a combined framework of flowchart recognition and text generation is designed to convert flowcharts into semantic information, which helps to process some data and supervision rules described only by flowcharts during intelligent supervision services, and realizes the automated supervision of service processes.

[0017] It has been found through experiments that the existing flowchart recognition method has the problem of missing primitives and difficult recognition of connecting lines when recognizing complex flowcharts with too many elements and complicated connection relationships. After analysis, it is found that text and other elements will have a certain negative impact on the recognition of connecting lines, and when using the target detection model, flowcharts with many primitives generally have the problem of missing primitives, while simple flowcharts with few primitives have less missed recognition. Therefore, the present invention designs a complex flowchart segmentation algorithm to divide the complex flowchart into two simple flowcharts, improve the accuracy of flowchart primitive recognition, and solve the problem of missing primitives; at the same time, a flowchart layer-by-layer separation process is designed, first the text and graphics are layered, the text is separated from the flowchart to obtain the graphic layer, and then the graphic layer primitives and arrows are identified by the target detection model, and the primitives and arrows are separated from the graphic layer to obtain the connecting line layer. At this time, only the connecting lines are left in the image, which is more convenient to identify and solves the problem of difficult recognition of connecting lines. Thus, the present invention realizes the recognition of complex flowcharts.

[0018] In step S2, the specific implementation process of obtaining the flowchart complexity judgment threshold includes: counting the number of connected domains of all complex process images in the Flowchart data set, and selecting the minimum value as the flowchart complexity judgment threshold. Obtaining the threshold facilitates subsequent judgment on flowchart segmentation, reduces unnecessary operations, and improves the efficiency of the entire method.

[0019] In step S3, the specific implementation process of dividing the complex flowchart into two simple parts includes: setting a scanning frame, using the scanning frame to scan the flowchart from top to bottom, when the number of pixels of the current step in the scanning frame is the same as the number of pixels of the previous step and both are less than the set threshold r, and the scanning frame is located in the middle of the flowchart, and the scanning frame does not contain a closed and symmetrical figure, the complex flowchart is divided to obtain two simple images. Complex flowchart segmentation divides complex problems into two parts, solves the problem of complex flowcharts being missed, and improves the accuracy of flowchart recognition.

[0020] In step S4, if the graphic layer of the input target detection model corresponds to two simple parts of a complex flowchart, the two output results of the target detection model need to be merged to obtain all the graphic elements and arrows of the complex flowchart. The graphic elements can provide the location information of the main text of the flowchart, and the arrows can provide the direction information of the connection lines of the flowchart, which is an important basis for the subsequent connection relationship judgment.

[0021] In step S4, the specific implementation process of obtaining the primitive set, arrow set and annotation set, separating the output result of the target detection model from the graphic layer generated in step S2, and obtaining the connection line layer includes: performing coordinate conversion on the output of the target detection model to obtain the arrow set; matching the coordinates of the primitive result after coordinate conversion with the flowchart text list to obtain the primitive set; the remaining text list after matching is the annotation set; according to the detected primitive and arrow coordinates, removing the pixels at the corresponding position in the graphic layer generated in step S2 to obtain the connection line layer. The primitive set, arrow set and annotation set provide the content information of the flowchart as the basis for the subsequent connection relationship judgment; the connection line layer only contains the connection lines of the flowchart, which solves the problem of difficult identification of connection lines and improves the accuracy of complex flowchart relationship identification.

[0022] The specific implementation process of step S5 includes:

[0023] Identify all straight line segments in the connecting line layer and obtain a list of straight line segments;

[0024] Traverse the list of straight line segments, determine whether the straight line segments intersect based on the rapid repulsion test and the straddle test, regard all intersecting straight lines as a connecting line, and obtain a connecting line set.

[0025] Step S5 obtains a set of connection lines, which provides flow chart structure information and is an important basis for subsequent judgment of the connection relationship of graphic elements.

[0026] The specific implementation process of step S6 includes:

[0027] 1) Initialize Result, determine the starting element of the flowchart, and determine based on the coordinates that the element at the top leftmost without a predecessor node is the starting node of the flowchart, and execute step 2);

[0028] 2) Determine whether all the flag bits of SL, SE, SA, and SF are 1. If not, initialize the temporary result set TR, store the number of the primitive start in TR, and execute step 3); if yes, end; SL: connection line set;

[0029] SE: graphic element set; SA: arrow set; SF: annotation set;

[0030] 3) Traverse SL, find the connecting line line connected to start, change the mark bit corresponding to line in SL to 1, and execute step 4);

[0031] 4) Traverse SF, if there is any note that is close to the line, change the mark corresponding to the note in SF to

[0032] 1, and store the note number in TR and execute step 5); if it does not exist, execute step 5 directly;

[0033] 5) Traverse SA. If there is any arrow that is close to line, change the flag bit corresponding to arrow in SA to 1 and execute step 6); if there is no arrow, execute step 6 directly;

[0034] 6) Traverse SE. If there is any element connected to arrow, change the flag bit corresponding to element in SE to 1, and store the element number in TR. Determine whether the element already exists in TR. If it does, it means that the branch is a loop branch, then store TR in Result and return to step 2); if it does not exist, execute step 7); if there is no element connected to arrow, store TR in Result and return to step 2);

[0035] 7) Traverse SL. If there is any connecting line line connected to element, change the mark bit corresponding to line in SL to 1 and return to step 4); if not, store TR in Result and return to step 2).

[0036] This step obtains the set of all paths of the flowchart, including the content of the flowchart and the connection relationship information, which serves as the input for generating the descriptive text of the flowchart.

[0037] As an inventive concept, the present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory; the processor executes the computer program to implement the steps of the above method.

[0038] As an inventive concept, the present invention also provides a computer-readable storage medium having a computer program / instruction stored thereon; the computer program / instruction implements the steps of the above method when executed by a processor.

[0039] As an inventive concept, the present invention also provides a computer program product, including a computer program / instruction; when the computer program / instruction is executed by a processor, the steps of the above method are implemented.

[0040] Compared with the prior art, the beneficial effects of the present invention are as follows: in order to solve the problem of difficult recognition of complex flowcharts, the present invention proposes a flowchart recognition method based on the idea of ​​segmentation, and designs a combined framework of flowchart recognition and text generation to convert flowcharts into semantic information, thereby helping to process some data and supervision rules that are only described by flowcharts in intelligent supervision services, thereby realizing automated supervision of service processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 The figure is a flow chart of a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] Example 1

[0044] This embodiment provides a method for generating a flowchart description text, comprising the following steps:

[0045] S1. Build the Flowchart dataset and train the target detection model - RT-DETR;

[0046] S2. Graphics and texts are layered, and the text in the process image is separated to obtain a graphic layer and a flowchart text list;

[0047] S3. Obtain the threshold value for judging the complexity of the flowchart, statistically analyze the connected domains of the complex process images in the data set, and obtain the lower limit value, i.e., the threshold value, to determine whether the process image needs to be segmented;

[0048] S4. Flowchart segmentation: Based on the threshold generated in S3, a complex flowchart segmentation algorithm is used to segment the complex flowchart into two simple parts, while the simple flowchart does not need to be segmented;

[0049] S5. Flowchart object recognition: input the graphic layer generated by S4 into the target detection model (RT-DETR model) trained by S1 to identify the graphic elements (i.e., the graphic boxes in the flowchart) and arrows. If S4 generates two results, the model detection results are merged. If S4 generates one result, no processing is required. Combined with the text list generated by S2, the graphic element set, arrow set, and annotation set (the prompt text near the connection line of the flowchart) are obtained. The results detected by the model are separated from the graphic layer generated by S2 to obtain the connection line layer.

[0050] S6. Flowchart connection line recognition, using the LSD straight line segment detection algorithm to detect the connection line layer generated by S5 to obtain a straight line segment set, and then using the connection line recognition algorithm to match the straight line segments to obtain a connection line set;

[0051] S7. Processing the flowchart recognition results, using a path matching algorithm to match the primitive set, arrow set, annotation set generated by S5 and the connection line set generated by S6 to obtain different paths in the flowchart, and storing the results in XML format;

[0052] S8. Text generation: input the XML result generated by S7 into GPT-4o to generate the final flowchart descriptive text.

[0053] The graphic layering described in step S2 refers to dividing the process image into a graphic layer and a text layer. PaddleOCR technology is used to identify the text information in the process image, including its coordinate position. According to the identified position coordinates, the text at the corresponding position in the process image is removed to obtain a graphic layer including only the process flow chart, so as to better identify the flow chart structure in the future and remove the influence of the text.

[0054] The segmentation threshold in step S3 is obtained by counting the number of connected domains of all complex process images in the Flowchart data set and selecting the minimum value as the threshold. If the number of connected domains of the flowchart exceeds the threshold, the process image is considered to be complex and needs to be segmented.

[0055] The complex process image segmentation algorithm described in step S4 aims to solve the problem that complex process images are difficult to identify. The specific steps are as follows:

[0056] (1) After the images and texts are layered, the graphic layer is refined;

[0057] (2) Calculate the number of connected domains in the process graph;

[0058] (3) According to the threshold t obtained in S3, if it exceeds the threshold t, the flowchart is considered complex and needs to be segmented;

[0059] (4) Set a box similar to a sliding window, the width and height of which are the width of the process image * p (a threshold value is set, not less than the average height of the primitive). Use this box to scan the process graph from top to bottom, with a step length of q. When the window position meets the following conditions, segmentation is performed: the number of pixels in the current step and the previous step of the window are almost the same and both are less than r; the window position is located around the middle of the process graph; the window does not contain closed and symmetrical figures. Finally, two simple images are obtained;

[0060] (5) If no suitable location is found, no splitting is done.

[0061] The flowchart object recognition described in step S5 is mainly to identify the primitives and arrows in the image. The purpose of detecting primitives is to determine the main text in the flowchart and distinguish the flowchart body from the annotations; the purpose of detecting arrows is to determine the direction, which is convenient for subsequent flowchart path recognition. The main idea is to input the two-part image or the unsegmented graphic layer generated by S4 into the trained RT-DETR model to obtain the detection result. The detection result of the segmented image needs to be converted to a coordinate so that its coordinate result is relative to the original image to obtain an arrow set; according to the coordinates of the detected primitive results and the text list obtained by S2, a primitive set is obtained, which includes the text and position coordinates in the primitive; the remaining text list is the annotation set; according to the detected primitive and arrow coordinates, the pixels at the corresponding position in the graphic layer generated by S2 are removed to obtain a connecting line layer.

[0062] The connection line identification described in step S6 is specifically performed in the following steps:

[0063] (1) Using the LSD straight line segment detection algorithm to identify all straight line segments in the connecting line layer generated by S5, and obtain a straight line segment list;

[0064] (2) Obtain a set of connecting lines according to the connecting line identification algorithm. Traverse the list of straight line segments, determine whether the straight line segments intersect based on the fast repulsion test and the straddle test, regard all intersecting straight lines as a connecting line, and obtain a set of connecting lines.

[0065] The path matching algorithm described in step S7 mainly uses the graphic element set SE, arrow set SA, annotation set SF obtained in step S5 and the connection line set SL obtained in step S6 to obtain all the paths of the flowchart. The specific steps are as follows:

[0066] 1) Initialize Result, determine the starting element of the flowchart, and judge based on the coordinates that the element at the top and leftmost without a predecessor node is the starting node of the flowchart, and execute step 2);

[0067] 2) Determine whether all the flag bits of SL, SE, SA, and SF are 1. If not, initialize the temporary result set TR, store the number of the primitive start in TR, and execute step 3); if so, the algorithm ends;

[0068] 3) Traverse SL, find the connecting line line connected to start, change the mark bit corresponding to line in SL to 1 (initial all are 0), and execute step 4);

[0069] 4) Traverse SF. If there is any note that is close to the line, change the mark corresponding to the note in SF to 1 (initial value is 0), store the note number in TR, and execute step 5); if there is no note, execute step 5);

[0070] 5) Traverse SA. If there is any arrow that is close to line, change the mark corresponding to arrow in SA to 1 (initial 0), and execute step 6); if there is no arrow, execute step 6);

[0071] 6) Traverse SE. If there is any element connected to arrow, change the flag bit corresponding to the element in SE to 1 (initial 0), and store the element number in TR. Then determine whether the element already exists in TR. If it does, it means that the branch is a loop branch, then store TR in Result and execute step 2); if it does not exist, execute step 7); if there is no element connected to arrow, then store TR in Result and execute step 2);

[0072] 7) Traverse SL. If there is any connecting line line connected to element, change the mark bit corresponding to line in SL to 1 (initial 0), and execute step 4); if there is no such line, store TR in Result, and execute step 2);

[0073] The XML format described in step S7 stores the obtained results in a structured XML format, which mainly includes graphic elements, annotation content and paths. The following logical elements are abstracted according to the flow chart analysis, as shown in Table 1:

[0074] Table 1 XML element tag introduction

[0075]

[0076] The relevant elements are described in detail below:

[0077] (1) Flowchart elements

[0078] Function: Root element, marking the start of the flowchart structure.

[0079] Rules: There is only one flowchart; there are multiple element, note, and road elements.

[0080] Related examples:

[0081] <flowchart>

[0082] <element id='graphical element'>

[0083]

[0084] <note id='annotation'>

[0085]

[0086] <road id='path'>

[0087]

[0088] < / flowchart>

[0089] (2) attr element

[0090] Function: Stores the text in the flowchart.

[0091] Rule: The 'name' attribute with the value 'text' represents text; there must be a'string' tag representing the text content; the 'attr' element is usually a sub - element of 'element' or 'note', storing the text content of the graphic primitive or annotation.

[0092] Related examples:

[0093] <attr name=’text’>

[0094] <string> Text content or annotation content in the graphic element< / string>

[0095]

[0096] (3) 'element' element

[0097] Function: Represents a graphic primitive node in the flowchart.

[0098] Rule: The 'id' uniquely identifies; there must be 1 'attr' element.

[0099] Related examples:

[0100] <element id=’Graphic primitive’>

[0101] <attr name=’text’>

[0102] <string> Text content in the graphic element< / string>

[0103]

[0104]

[0105] (4) 'note' element

[0106] Function: Represents an annotation near the connection line in the flowchart.

[0107] Rule: The 'id' uniquely identifies; there must be 1 'attr' element.

[0108] Related examples:

[0109] <note id=’Graphic primitive’>

[0110] <attr name=’text’>

[0111] <string> Note content< / string>

[0112]

[0113]

[0114] (5) 'node' element

[0115] Function: Represents a node in the flowchart path.

[0116] Rule: The id is a unique identifier; the value of the content attribute is the id value of an element or a note, referring to a graphic element node or a note in the flow chart; in XML, the order of nodes by default represents the order of nodes in the path, from the start to the end of the path.

[0117] Related examples:

[0118] <node id="content='id value 1 of element or note'>

[0119] <node id="content='id value 2 of element or note'>

[0120] (6) road element

[0121] Function: Represents a single path in the flow chart.

[0122] Rule: The id is a unique identifier; if the path does not contain a loop structure, the path goes from the start to the end of the flow chart, describing a complete path; if the path contains a loop structure, the path ends at the node where the loop ends; it contains the sub-tag node.

[0123] Related examples:

[0124] <road id='non-loop path'>

[0125] <node id="content='start of flow chart'>

[0126] <node id="content='end of flow chart'>

[0127]

[0128] <road id='loop path'>

[0129] <node id="content='start of flow chart'>

[0130] <node id="content='end of loop'>

[0131]

[0132] The text generation described in step S8 is to input the XML result obtained in S7 into GPT-4o to generate the final text result.

[0133] The flowchart description generation method based on deep learning designed in the embodiment of the present invention uses PaddleOCR to separate text and images, segment the complex flowchart, and then perform flowchart recognition, which includes flowchart object recognition, connection line recognition, and structural relationship recognition. The generated results are stored in XML format, and finally GPT-4o is called to generate natural language. The method of the embodiment of the present invention can better recognize complex flowcharts and is highly practical.

[0134] The following is a detailed description of the embodiment of the present invention in conjunction with an actual case:

[0135] First, input any flowchart, recognize the text through PaddleOCR, get the text result list, separate the text, and get the graphic layer image.

[0136] Then, determine whether the process image needs to be segmented. If segmentation is required, the graphic layer is segmented into two parts through the segmentation algorithm, and the two parts are input into the RT-DETR model for detection respectively. The coordinates of the detection results are further converted to those relative to the original flow chart to obtain the list of graphics elements, arrows, and annotations. The detection results are separated from the graphic layer to obtain the connection line layer, and the LSD algorithm and connection line recognition algorithm are used to obtain the connection line list.

[0137] Next, a path matching algorithm is used to obtain the flow chart structure content and relationships based on the result list, and the results are stored in XML format.

[0138] Finally, the XML file is input into GPT-4o to get the final text result.

[0139] For the flowchart description generation method based on deep learning proposed in the embodiment of the present invention, the method of the embodiment of the present invention is finally evaluated using BERTScore on the Flowchart dataset. As shown in Table 1, the precision (Precision) can reach 93.53%, the recall rate (Recall) can reach 91.42%, and the F1 score (or balanced F score) is 92.15%.

[0140] Table 1 Evaluation results of the method of the embodiment of the present invention using BERTScore on the Flowchart dataset

[0141]

[0142] Example 2

[0143] Embodiment 2 of the present invention provides a terminal device corresponding to the above-mentioned embodiment 1. The terminal device may be a processing device for a client, such as a mobile phone, a laptop computer, a tablet computer, a desktop computer, etc., to execute the method of the above-mentioned embodiment.

[0144] The terminal device of this embodiment includes a memory, a processor, and a computer program stored in the memory; the processor executes the computer program in the memory to implement the steps of the method in the above-mentioned embodiment 1.

[0145] In some implementations, the memory may be a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory.

[0146] In some other implementations, the processor may be a central processing unit (CPU), a digital signal processor (DSP), or other general-purpose processors of various types, which are not limited herein.

[0147] Example 3

[0148] Embodiment 3 of the present invention provides a computer-readable storage medium corresponding to the above embodiment 1, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the method of the above embodiment 1 are implemented.

[0149] Computer readable storage media can be tangible devices that hold and store instructions used by instruction execution devices. Computer readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any combination thereof.

[0150] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiments of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0151] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0152] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0153] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0154] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for generating a flowchart description text, characterized in that: The following steps are involved: S1. Train the target detection model using the Flowchart dataset. S2, separating the text in the process image to obtain a graphic layer and a flowchart text list; Statistically analyze the connected domains of complex process images in the Flowchart dataset to obtain the lower limit value, which is the threshold for judging the complexity of the flowchart; S3. According to the flowchart complexity evaluation threshold, the complex flowchart is divided into two simple parts; the complex flowchart refers to a flowchart whose number of connected domains exceeds the flowchart complexity evaluation threshold; S4, inputting the graphic layer generated by S3 into the trained target detection model, identifying the graphic elements and arrows, combining the flowchart text list, obtaining a graphic element set, an arrow set and an annotation set, separating the output result of the target detection model from the graphic layer generated by step S2, and obtaining a connection line layer; S5. Obtain a set of straight line segments using the connection line layer, match the straight line segments, and obtain a set of connection lines; S6, matching the graphic element set, arrow set, annotation set, and connection line set to obtain different paths in the flowchart; S7. Generate the final flowchart descriptive text using different path results.

2. The method for generating flowchart description text according to claim 1, characterized in that: In step S2, the specific implementation process of obtaining the flowchart complexity judgment threshold includes: counting the number of connected domains of all complex process images in the Flowchart data set, and selecting the minimum value as the flowchart complexity judgment threshold.

3. The method for generating flowchart description text according to claim 1, characterized in that: In step S3, the specific implementation process of dividing the complex flowchart into two simple parts includes: setting a scanning frame, using the scanning frame to scan the flowchart from top to bottom, when the number of pixels of the current step in the scanning frame is the same as the number of pixels of the previous step and both are less than the set threshold r, and the scanning frame is located in the middle of the flowchart, and the scanning frame does not contain a closed and symmetrical figure, the complex flowchart is divided to obtain two simple images.

4. The method for generating flowchart description text according to claim 1, characterized in that: In step S4, if the graphic layer of the input target detection model corresponds to two simple parts of the complex flowchart, the two output results of the target detection model are merged to obtain all the graphics elements and arrows of the complex flowchart.

5. The method for generating flowchart description text according to claim 1, characterized in that: In step S4, a primitive set, an arrow set and an annotation set are obtained, and the output result of the target detection model is separated from the graphic layer generated in step S2. The specific implementation process of obtaining the connection line layer includes: performing coordinate transformation on the output of the target detection model to obtain an arrow set; matching the coordinates of the primitive results after coordinate transformation with the flowchart text list to obtain a primitive set; the remaining text list after matching is the annotation set; and removing the pixels at the corresponding positions in the graphic layer generated in step S2 according to the detected primitive and arrow coordinates to obtain a connection line layer.

6. The method for generating flowchart description text according to claim 1, characterized in that: The specific implementation process of step S5 includes: Identify all straight line segments in the connecting line layer and obtain a list of straight line segments; Traverse the list of straight line segments, determine whether the straight line segments intersect based on the rapid repulsion test and the straddle test, regard all intersecting straight lines as a connecting line, and obtain a connecting line set.

7. The method for generating flowchart description text according to claim 1, characterized in that: The specific implementation process of step S6 includes: 1) Initialize Result, determine the starting element of the flowchart, and determine based on the coordinates that the element at the top leftmost without a predecessor node is the starting node of the flowchart, and execute step 2); 2) Determine whether all the flag bits of SL, SE, SA, and SF are 1. If not, initialize the temporary result set TR, store the number of the element start in TR, and execute step 3); if yes, end; SL is the connection line set, SE is the element set, SA is the arrow set, and SF is the annotation set; 3) Traverse SL, find the connecting line line connected to start, change the mark bit corresponding to line in SL to 1, and execute step 4); 4) Traverse SF. If there is any note that is close to the line, change the mark corresponding to the note in SF to 1, store the note number in TR, and execute step 5); if there is no note, execute step 5 directly; 5) Traverse SA. If there is any arrow that is close to line, change the flag bit corresponding to arrow in SA to 1 and execute step 6); if there is no arrow, execute step 6 directly; 6) Traverse SE. If there is any element connected to arrow, change the flag bit corresponding to element in SE to 1, and store the element number in TR. Determine whether the element already exists in TR. If it does, it means that the branch is a loop branch, then store TR in Result and return to step 2); if it does not exist, execute step 7); if there is no element connected to arrow, store TR in Result and return to step 2); 7) Traverse SL. If there is any connecting line line connected to element, change the mark bit corresponding to line in SL to 1 and return to step 4); if not, store TR in Result and return to step 2).

8. A terminal device comprising a memory, a processor and a computer program stored in the memory; characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program / instruction stored thereon; characterized in that: When the computer program / instructions are executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program / instructions; characterized in that When the computer program / instructions are executed by a processor, the steps of the method of any one of claims 1 to 7 are implemented.

Images