A document image flowchart recognition method and device based on YOLO v5 and a medium

Abstract

The present application relates to the technical field of image processing, in particular to a document image flowchart recognition method and device based on YOLO v5 and a medium; comprising: step 1: using YOLO v5 to locate the position of the flowchart in the document image; step 2: flowchart structure element detection, obtaining the category and position of the structure element; step 3: text recognition, obtaining the text and conditional text information in the structure element; step 4: analyzing the position of the arrow line segment to obtain the starting position information of the arrow line segment; step 5: analyzing the relationship structure of the flowchart by combining the position of the structure element and the arrow line segment; step 6: saving the flowchart recognition result; the present application effectively locates the position of the flowchart in the document image through the YOLO v5 model, and can detect each structure element in the flowchart, has strong robustness, and avoids the serious dependence problem of the connected domain method on contour continuity; through an end-to-end target detection method, the complex rule design problem involved in the corner point detection method is avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a document image flowchart recognition method based on YOLO v5, equipment and medium. BACKGROUND

[0002] A flowchart is composed of multiple structural elements and the connection relationship between them. Compared with single text description, it can more intuitively reflect the execution process of the matter, and is therefore widely used in documents such as business guide and algorithm flow. Identifying and analyzing the flowchart in the document image is of great significance to document image recognition. The difficulty lies in that there are many structural elements constituting the flowchart, although there are only three relationships between the structural elements, namely sequence, cycle and branch, but they can be combined into various complex layout structures, which brings certain difficulty to the recognition of the flowchart.

[0003] The existing flowchart recognition idea is to process the structural elements and the text respectively. The processing of the structural elements is to identify the shape and connection relationship of each structural element, and the processing of the text is to identify the text in the flowchart through mature OCR technology. The recognition of the structural elements is the key and difficulty, and the existing recognition methods are connected domain analysis method and corner point detection method. Among them, the connected domain method is to first process the structural elements and the text in layers, and then extract and recognize the structural elements through connected domain analysis combined with threshold method. The limitation is that the connected domain analysis judges whether the pixels are of the same category through the continuity of the pixels, so it cannot handle the case of discontinuous contour. The other corner point detection method takes the structural elements and the connection lines between them as the classification target. The corner point is defined as the intersection of straight lines or curves, so the category of the corresponding structural element can be judged by analyzing the combination mode of different types of corner points. The structural elements are divided into straight line type and curve type, and the type of the corner point is detected by different corner point detection algorithms, and then the category of the structural element is judged by the combination rule of the corner point type. The disadvantage is that there are many types of structural elements, so the combination rule design is complex and the algorithm implementation is difficult.

[0004] Therefore, the existing method needs to be improved in terms of recognition accuracy and algorithm efficiency of the flowchart structural elements. SUMMARY

[0005] Therefore, the present application aims to overcome the defects of inaccurate recognition and low algorithm efficiency in the prior art, and to provide a document image flowchart recognition method based on YOLO v5, equipment and medium.

[0006] A document image flowchart recognition method based on YOLO v5, comprising the following steps:

[0007] Step 1: Locate the position of the flowchart in the document image using the YOLO v5 positioning process;

[0008] Step 2: Flowchart structure element detection, obtain the class and position information of each structure element;

[0009] Step 3: Text recognition, get the text information and conditional text information in the structure element;

[0010] Step 4: Analyze the position of the arrow line segment to get the starting position information of each arrow line segment;

[0011] Step 5: Analyze the relationship structure of the flowchart combining the position information of the structure element and the arrow line segment;

[0012] Step 6: Save the flowchart recognition result.

[0013] Further, the specific content of step 1 is:

[0014] 1.1: Use the public document image dataset with flowchart position information, or use the existing document image and mark the position of the flowchart area in the image as the dataset;

[0015] 1.2: Train the YOLO v5 model using the dataset obtained in step 1.1, and save the trained model;

[0016] 1.3: Detect the flowchart area in the image through the trained model, get the top-left and bottom-right coordinate information of the flowchart, and slice the document image to retain the flowchart area in the image.

[0017] Further, the specific content of step 2 is:

[0018] 2.1: Flowchart data generation, set the flowchart structure element class, randomly generate flowchart data, retain the generated flowchart structure and corresponding class and position information as the dataset, or use the existing document image data containing flowchart and mark the class and position information of each structure element in the flowchart as the dataset.

[0019] 2.2: Train the YOLO v5 model using the dataset obtained in step 2.1, and save the trained model.

[0020] 2.3: Identify and locate the structure elements of the flowchart through the trained model, get the class information and position information of each structure element.

[0021] Further, the flowchart structure element class is conditional text, rectangle, ellipse, diamond, or parallelogram.

[0022] Further, in step 3, the text recognition needs to recognize the text in the structural elements by the OCR model.

[0023] Further, step 3 also includes detecting and recognizing the conditional text not located in the structural element box.

[0024] Further, in step 4, the arrow line segment position analysis adopts a method based on connected domains, and the specific process is as follows:

[0025] 4.1: According to the structural element position information obtained in step 2, remove the structural elements of the flowchart area, and retain the arrow line segments between the structural elements;

[0026] 4.2: Perform a binary inversion operation to obtain a binary image, wherein the pixels of the arrow line segment part are 255, and the pixels of the background part are 0;

[0027] 4.3: Detect the connected domains in the image, and each connected domain represents an arrow line segment, and obtain the pixel information constituting the arrow line segment;

[0028] 4.4: Arrow line segment starting position analysis: for each arrow region, a small region is set, and the region centered at each pixel point is traversed;

[0029] If there are two connected domains in the region, the region is a candidate endpoint region of the arrow line segment, and the number of pixels with a value of 255 in the region is counted, and the regions centered at the remaining pixel points in the region are no longer analyzed;

[0030] If there are three connected domains, they are non-endpoint pixels in the arrow line;

[0031] 4.5: In the candidate endpoint region, the region with a larger number of pixels with a value of 255 is the arrow region, and the region with a smaller number of pixels is the arrow tail region. The center point coordinates corresponding to the arrow region and the arrow tail region are taken as the coordinate information of the arrow and the arrow tail.

[0032] Further, in step 5, flowchart relationship structure analysis: each arrow line segment represents a relationship, and by traversing the position information of the structural elements except the conditional text through the position information of the arrow and the arrow tail, the structural elements connected to the arrow and the arrow tail of the arrow line are obtained.

[0033] For the conditional text, find the arrow line segment closest to it in the Euclidean distance, which becomes the condition between the arrow line segment and the structural element.

[0034] An electronic device includes a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the above methods.

[0035] A computer readable storage medium for storing computer instructions, the computer instructions being executed by a processor to implement the steps of any of the above methods.

[0036] Compared with the prior art, the YOLO v5 model can effectively locate the position of the flowchart in the document image, and can detect each structural element in the flowchart, has strong robustness, avoids the serious dependence problem of the contour continuity of the connected domain method, and simultaneously avoids the complex rule design problem involved in the corner point detection method through an end-to-end target detection method. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the specific embodiments or prior art of the present application, the drawings needed in the specific embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0038] Figure 1 Flowchart of the present application;

[0039] Figure 2 Flowchart region detection result of the present application;

[0040] Figure 3 Generated flowchart data of the present application;

[0041] Figure 4 Flowchart structure element recognition positioning result of the present application;

[0042] Figure 5 Arrow line segment analysis result of the present application;

[0043] Figure 6 Flowchart analysis and recognition result of the present application. DETAILED DESCRIPTION

[0044] The technical solutions of the present application will be described in detail below with reference to the drawings. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0045] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second", "third" are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0046] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connecting" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0047] In addition, the technical features involved in the different embodiments of the application described below can be combined with each other as long as there is no conflict between them.

[0048] Please refer to Figure 1 A document image flowchart recognition method based on YOLO v5, comprising the following steps:

[0049] Step 1: locate the position of the flowchart in the document image using YOLO v5;

[0050] Step 2: flowchart structure element detection, get the class and position information of each structure element;

[0051] Step 3: text recognition, get the text information and conditional text information in the structure element;

[0052] Step 4: analyze the position of the arrow line segment to get the starting position information of each arrow line segment;

[0053] Step 5: analyze the relationship structure of the flowchart combining the position information of the structure element and the arrow line segment;

[0054] Step 6: save the flowchart recognition result.

[0055] The specific content of step 1 is:

[0056] 1.1: Use a public document image dataset with flowchart position information, or use an existing document image, and mark the position of the flowchart area in the image as a dataset;

[0057] 1.2: Train the YOLO v5 model using the data set obtained in step 1.1, save the trained model;

[0058] 1.3: Detect the flowchart region in the image by the trained model, obtain the top-left and bottom-right coordinates of the flowchart, slice the document image, and retain the flowchart region in the image. The flowchart region detection result is shown in FIG. 2. Figure 2

[0059] Further, the step 2 specifically includes:

[0060] 2.1: Flowchart data generation, set the flowchart structure element category, randomly generate flowchart data, retain the generated flowchart structure and corresponding category and position information as a data set, or use existing document image data containing flowcharts and mark the category and position information of each structure element in the flowchart as a data set. The generated flowchart data is shown in FIG. 3. Figure 3

[0061] 2.2: Train the YOLO v5 model using the data set obtained in step 2.1, save the trained model.

[0062] 2.3: Identify and locate the structure elements of the flowchart by the trained model, obtain the category information and position information of each structure element. The flowchart structure element identification and positioning result is shown in FIG. 4. Figure 4

[0063] The flowchart structure element categories include conditional words, rectangles, ellipses, diamonds, parallelograms, etc.

[0064] In the step 3, the text recognition needs to identify the text in the structure element by an OCR model.

[0065] The step 3 further includes detecting and identifying the conditional words not located in the structure element box.

[0066] In the step 4, the arrow line segment position analysis adopts a method based on connected domains, and the specific process is as follows:

[0067] 4.1: Remove the structure elements of the flowchart region according to the structure element position information obtained in step 2, and retain the arrow line segments between the structure elements;

[0068] 4.2: Binaryzation and inversion operation, obtain a binaryzation image, wherein the pixels of the arrow line segment part are 255, and the pixels of the background part are 0;

[0069] 4.3: Detect the connected domains in the image, each connected domain represents an arrow line segment, and obtain the pixel information constituting the arrow line segment; ​​​

[0070] 4.4: Analysis of the starting position of the arrow line segment: for each arrow area, a small area is set, and the area centered on each pixel point is traversed;

[0071] If there are two connected domains in the area, the area is a candidate endpoint area of the arrow line segment, and the number of pixels with a value of 255 in the area is counted, and the area centered on the remaining pixel points in the area is no longer analyzed;

[0072] If there are three connected domains, they are non-endpoint pixels in the arrow line;

[0073] 4.5: In the candidate endpoint area, the area with a large number of pixels with a value of 255 is the arrow area, and the area with a small number of pixels is the arrow tail area. The center point coordinates corresponding to the arrow and arrow tail areas are taken as the coordinate information of the arrow and arrow tail;

[0074] The arrow line segment analysis result is shown in FIG. 4. Figure 5

[0075] In step 5, the flowchart relationship structure analysis: each arrow line segment represents a relationship, and the position information of the structure elements other than the conditional text is traversed through the position information of the arrow and arrow tail to obtain the structure elements connected to the arrow and arrow tail of the arrow line;

[0076] For the conditional text, find the arrow line segment closest to it in the Euclidean distance, which becomes the condition between the arrow line segment and the structure element.

[0077] In step 6, the flowchart analysis and recognition result is shown in FIG. 5. Figure 6

[0078] An electronic device, comprising a memory and a processor, the memory stores a computer program, and the processor implements the steps of the method of any one of the above when executing the computer program.

[0079] A computer readable storage medium for storing computer instructions, the computer instructions are executed by a processor to implement the steps of the method of any one of the above.

[0080] Obviously, the above embodiments are only examples for clear illustration, and are not limitations on the embodiments. Based on the above description, other different forms of changes or variations can be made by those of ordinary skill in the art. Here, all embodiments cannot be exhausted, and the obvious changes or variations derived therefrom are still within the protection scope of the present application.​​

Claims

1. A document image flowchart recognition method based on YOLO v5, characterized in that, Includes the following steps: Step 1: Use YOLO v5 to locate the position of the flowchart within the document image; Step 2: Flowchart structural element detection to obtain the category and location information of each structural element; Step 3: Text recognition, obtaining text information and conditional text information within the structural elements; Step 4: Analyze the positions of the arrow segments to obtain the starting position information of each arrow segment; Step 5: Analyze the relational structure of the flowchart by combining the positional information of structural elements and arrow segments; Step 6: Save the flowchart recognition results; The specific content of step 1 is as follows: 1.1: Use publicly available document image datasets with flowchart location information, or use existing document images and mark the locations of flowchart regions in the images as the dataset; 1.2: Train the YOLO v5 model using the dataset obtained in step 1.1, and save the trained model; 1.3: Detect flowchart regions in images using the trained model, obtain the coordinates of the top left and bottom right corners of the flowchart, slice the document image, and retain the flowchart regions in the image; The specific content of step 2 is as follows: 2.1: Flowchart data generation: Set the categories of flowchart structural elements, randomly generate flowchart data, retain the generated flowchart structure and corresponding category and position information as a dataset, or use existing flowchart document image data and mark the category and position information of each structural element in the flowchart as a dataset. 2.2: Use the dataset obtained in step 2.1 to train a YOLO v5 model and save the trained model; 2.3: The trained model is used to identify and locate the structural elements of the flowchart, and the category and location information of each structural element is obtained; In step 4, the arrow segment position analysis uses a connected component-based method, and the specific process is as follows: 4.1: Based on the structural element position information obtained in step 2, remove the structural elements in the flowchart area and retain the arrow segments between the structural elements; 4.2: Binarization and inversion operation to obtain a binarized image, where the arrow segment has 255 pixels and the background has 0 pixels; 4.3: Detect connected components in the image. Each connected component represents an arrow segment, and obtain the pixel information that constitutes the arrow segment. 4.4: Analysis of the starting position of the arrow line segment: For each arrow region, define a smaller region and traverse the region centered on each pixel; If there are two connected components in the region, then the region is selected as a candidate endpoint region for the arrow segment, and the number of pixels with a value of 255 in the region is counted. At the same time, the region centered on the other pixels in the region is no longer analyzed. If there are three connected components, then the non-endpoint pixels in the arrow line are the components. 4.5: In the candidate endpoint region, the region with a larger number of pixels with a value of 255 is the arrow region, and the region with a smaller number is the arrow tail region; the coordinates of the center point corresponding to the arrow and arrow tail regions are used as the coordinate information of the arrow and arrow tail. In step 5, the flowchart relationship structure analysis is performed as follows: each arrow segment represents a relationship. By traversing the position information of structural elements other than conditional text through the position information of the arrow and tail, the structural elements connected to the arrow and tail of the arrow line are obtained. For conditional text, find the arrow segment with the closest Euclidean distance to it, and use it as the condition between the arrow segment and the structural element.

2. The method according to claim 1, characterized in that, The flowchart structure elements are categorized as conditional text, rectangle, ellipse, rhombus, or parallelogram.

3. The method according to claim 1, characterized in that, In step 3, text recognition requires the use of an OCR model to identify the text within structural elements.

4. The method according to claim 3, characterized in that, Step 3 also includes detecting and recognizing conditional text that is not located within the structure element box.

5. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1-4.

6. A computer-readable storage medium for storing computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the method according to any one of claims 1-4.

Images