A flowchart recognition method

Abstract

This invention relates to the field of image processing technology, specifically disclosing a flowchart recognition method. The flowchart recognition method includes: acquiring a target image; identifying components in the target image; identifying text in the target image; removing the identified components and text from the target image, and then identifying connections in the target image; identifying connection relationships in the target image based on the attributes of the connections and components; and identifying connection directions in the target image based on the attributes of the components and the connection relationships. This flowchart recognition method can identify components, text, and connections in a target image, and identify connection relationships in the target image based on the attributes of the connections and components, thereby achieving automatic flowchart recognition.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and specifically relates to a method for inflection point recognition. Background Technology

[0002] The development of a full-scale nuclear power plant simulator requires simulation modeling of hundreds of power plant process systems, hundreds of power distribution systems, thousands of DCS (Distributed Control System) control logic diagrams, and various styles of process flow diagrams, teaching control diagrams, and control panel diagrams. Manual modeling is inefficient and prone to errors, a key factor affecting simulator development efficiency. Therefore, there is an urgent need to implement automated modeling methods to automatically convert design drawings into simulation models, thereby improving modeling efficiency and quality.

[0003] To achieve automatic modeling, intelligent diagram recognition must first be realized. How to achieve automatic recognition of flowcharts is a problem that this invention urgently needs to solve. Summary of the Invention

[0004] The purpose of this invention is to provide a flowchart recognition method to solve the problem of automatic flowchart recognition.

[0005] The technical solution of the present invention is as follows: a flowchart recognition method, the method specifically includes...

[0006] S1. Obtain the target image;

[0007] S2. Identify components in the target image;

[0008] S3. Recognize the text in the target image;

[0009] S4. After removing the identified components and text from the target image, identify the connecting lines in the target image.

[0010] S5. Identify the connection relationships in the target image based on the properties of the lines and components in the target image;

[0011] S6. Based on the attributes of the components and the connection relationships in the target image, identify the connection directions in the target image.

[0012] The specific steps for identifying the connection direction in the target image in step S6 are as follows:

[0013] S6.1 Set parameters related to the flow direction in the component attribute definition;

[0014] S6.2 Identify the flow direction with flow direction elements;

[0015] S6.3 updates the set of primitives whose flow direction cannot be identified;

[0016] S6.4 For components whose flow direction cannot be determined, the flow direction shall be given according to the rule of top to bottom and left to right.

[0017] The identification of elements in the target image in step S2 can be achieved using any one or a combination of the following methods:

[0018] (1) Construct a component recognition model and perform component recognition in target images based on deep learning;

[0019] (2) Component recognition in target images is achieved based on template matching;

[0020] (3) Component recognition in target images is achieved based on keyword matching.

[0021] The specific steps for character recognition in the target image in step S3 are as follows:

[0022] S3.1 Perform image preprocessing on the acquired target image;

[0023] S3.2 Perform target detection on the text region in the target image;

[0024] The YOLO algorithm is used to obtain boundary coordinates and class probabilities from image pixels, and the positions and information of all targets in the target image are confirmed and identified.

[0025] S3.3 Collect and recognize the text in the target image;

[0026] The CRNN text recognition model is used to recognize text in a target image. The text recognition model is obtained by training a pre-trained model and a constructed data sample set.

[0027] The specific steps for recognizing text in the target image are as follows:

[0028] (1) Obtain sample data using data augmentation techniques;

[0029] Noise is randomly added to the sample images to create new sample data;

[0030] By establishing a template set of standard characters, a large number of sample images are randomly generated using sample synthesis.

[0031] (2) Perform sliding recognition on the target image;

[0032] The entire target image is divided into multiple sub-images by cropping.

[0033] A sliding sub-image segmentation method is used, allowing the segmentation box to slide across the entire target image;

[0034] The non-maximum suppression algorithm is used to remove duplicate identified regions and then the images are merged.

[0035] (3) Recognize the direction of text in the target image;

[0036] The detection is performed using a binary classification model, which divides horizontal and vertical text into two categories of samples. The binary classification model is trained to determine the direction of the text, and the text direction is then corrected.

[0037] The system performs rule-based detection, determining the direction of the text based on the length and width of the text region.

[0038] The specific steps in step S5 for identifying the connection relationships in the target image based on the attributes of the lines and components in the target image are as follows:

[0039] S5.1 Based on the obtained connection position information, identify the position information of corner points in the target image;

[0040] S5.2 Based on the position information of the corner points in the identified target image, identify the inflection points of the connecting lines in the target image and the connection points of each component;

[0041] S5.3 Identify possible connectivity relationships in the target image;

[0042] S5.3.1 Identify whether there is a possible connection relationship between two adjacent elements;

[0043] If two adjacent elements are found to have connection points facing opposite directions and there is a connection between the two adjacent elements that satisfies a first predetermined condition, then a possible connection relationship between the two adjacent elements is identified.

[0044] S5.3.2 Identify whether there are possible connections between adjacent elements and inflection points;

[0045] If an adjacent element and an inflection point are connected by a line that satisfies a first predetermined condition and the corresponding element has a connection point facing that line, then it is determined that there is a possible connection relationship between the corresponding element and the inflection point.

[0046] S5.3.3 Identify whether there is a possible connection between two adjacent inflection points;

[0047] If a line that satisfies a first predetermined condition is detected between two adjacent inflection points, then it is determined that there is a possible connection between the two related inflection points.

[0048] S5.4 Based on the properties of the components, deterministic and impossible connections are filtered out from the obtained possible connections.

[0049] The specific steps in step S5.4 for filtering out deterministic and impossible connections from the obtained possible connections are as follows:

[0050] S5.4.1 When there are horizontal or vertical straight line connections between components, use a probability identification method to filter out deterministic connection relationships and impossible connection relationships;

[0051] S5.4.1.1 Calculate the overlap ratio D of the opposite surfaces of two adjacent elements:

[0052]

[0053] Where A and B represent the widths of the opposite faces of two adjacent elements, respectively; and C represents the overlap width of the opposite faces of two adjacent elements.

[0054] S5.4.1.2 Calculate the ratio G of the connection length to the spacing between two adjacent components:

[0055]

[0056] Where F represents the length of the connection between two components, and E represents the length of the connection and the spacing between two adjacent components;

[0057] S5.4.1.3 When the overlap ratio D of two components and the ratio of connection length to spacing G reach the corresponding predetermined probability threshold, it is determined that there is a possible connection relationship between the two adjacent components.

[0058] Based on the D and G values, the possible connections between each element are sorted, and deterministic connections are filtered according to the element's attribute definition and sorting results. The larger the D and G values, the greater the probability that the connection is a deterministic connection.

[0059] S5.4.2 When a diagonal line is detected between two adjacent elements, if both elements have unconnected connection points and there are no other possible connections between them, then a deterministic connection relationship is determined between the two elements. When a diagonal line is detected between an adjacent element and an inflection point, if the element has unconnected connection points and there are no other possible connections between them, then a deterministic connection relationship is determined between the element and the inflection point.

[0060] S5.4.3 Filter out impossible connections from possible connections;

[0061] When the number of deterministic connections of an element in a certain direction is the same as the number of connection points defined by the element's attributes in the method, the uncertain possible connections of the element in that direction are determined as impossible connections; when the total number of deterministic connections of an element is the same as the total number of connection points defined by the element's attributes, the uncertain possible connections of the element are determined as impossible connections.

[0062] After identifying the connection direction in the target image in step S6, the method further includes checking the flowchart recognition result in the target image as follows:

[0063] S7.1 calculates and evaluates the accuracy of target image recognition;

[0064] The predicted probability of each element is calculated using the YOLO model;

[0065] S7.2 Checks the automated recognition results in the target image and provides prompts for uncertain recognition results;

[0066] S7.2.1 Automated recognition and inspection of target images based on attribute characteristics;

[0067] The manually defined attributes are compared with the automatically identified attributes. If two attributes of the same element are inconsistent, it is considered that the element may be misidentified. In the identification result file, each element will have an identification accuracy attribute. This attribute is compared with the set threshold to evaluate whether the result is reliable.

[0068] S7.2.2 Removes the identified components from the target image and fills in blank pixels, leaving unidentified components or those with inaccurately marked identification boxes in the original image, thereby quickly and accurately finding unidentified components.

[0069] The specific steps for identifying the connection method in the target image in step S6 are as follows:

[0070] S6.1 Set parameters related to the flow direction in the component attribute definition;

[0071] In the component property definition, set parameters related to the flow direction, that is, indicate whether the component has directionality;

[0072] S6.2 Identify the flow direction with flow direction elements;

[0073] By using template matching, hash verification, or feature point matching, it is possible to determine which type of template the element to be identified matches, thereby identifying the flow direction of the element;

[0074] S6.3 updates the set of primitives whose flow direction cannot be identified;

[0075] For a line without branching points, its flow direction is consistent. If there is an element with a defined flow direction on the line, the flow direction of other elements on the line can be determined, thereby updating the set of primitives whose flow direction cannot be identified.

[0076] The specific steps for constructing the component recognition model and recognizing components in the target image based on deep learning are as follows:

[0077] Construct a YOLO model by dividing the target image into a grid and assigning each grid to detect if the center point falls within the target object.

[0078] S2.1.1 Preprocess the target image to be identified;

[0079] S2.1.2 Perform data labeling;

[0080] Select and mark the components in the target image, including the type of the component and the position of the mark box;

[0081] S2.1.3 Perform sample data augmentation

[0082] S2.1.3.1 Randomly extract local regions from the same image and generate a first sample image based on the distribution of the extracted local regions;

[0083] S2.1.3.2 Add random noise information to the first sample image and generate a new first sample image;

[0084] S2.2.3.3 Constructing virtual sample data;

[0085] Extract several component templates from a pre-defined standard template library, and stitch the extracted component templates together to form several first sample images;

[0086] After S2.1.4 performs sample training and updates, component detection and identification are carried out.

[0087] A YOLO component recognition model was constructed, and after training the model using the first sample set, component detection and recognition were carried out.

[0088] The specific steps for identifying components in a target image based on template matching are as follows:

[0089] S2.2.1 Establish a standard template library;

[0090] Construct component templates that include process components, electrical components, and panel components;

[0091] S2.2.2 Establish multiple template libraries, with component templates in different template libraries having different resolutions;

[0092] S2.2.2.1 Extract several component templates of different resolutions to form multiple different template libraries;

[0093] S2.2.2.2 Sort the component template libraries of different resolutions from high to low resolution as input for baseline resolution optimization;

[0094] S2.2.2.3 Use component template libraries of different resolutions to perform template matching on different flowcharts. All categories of elements are identified and the component matching values ​​and average values ​​on different flowcharts are output.

[0095] S2.2.2.4 Select the optimal resolution template to obtain the template library with the optimal resolution;

[0096] S2.2.3 Perform template matching and identify the components in the target image based on the matching results;

[0097] Based on the target image, the target is searched in the template library; the image to be detected is traversed from left to right and from top to bottom using a matching algorithm, and then the template is matched with the sub-image blocks from top to bottom; and the components in the target image are identified based on the matching results.

[0098] Matching the component template with the sub-image block specifically includes: feature matching, edge contour matching, segmentation matching, pixel proportion matching, and color matching;

[0099] When identifying components in a target image based on matching structures, the specific steps for setting multiple threshold parameters for various template matching identifications are as follows:

[0100] First, set a relatively high matching parameter to match the easily matched elements, and then remove those elements.

[0101] Set lower matching parameters for the remaining components and use multiple matching methods simultaneously to match the remaining components;

[0102] After repeating the above two steps, provide a prompt to the user if there are still unrecognized entries.

[0103] The significant advantage of this invention is that the flowchart recognition method described in this invention can identify components, text, and connections in a target image, and identify the connection relationships in the target image based on the attribute definitions of the connections and components, thereby enabling automatic flowchart recognition. Attached Figure Description

[0104] Figure 1 This is a flowchart of a flowchart recognition method according to the present invention. Detailed Implementation

[0105] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0106] like Figure 1 As shown, a flowchart recognition method includes the following steps:

[0107] S1. Obtain the target image;

[0108] Acquire target images related to system flowcharts such as electrical diagrams, process diagrams, and panel diagrams. The acquired target images include several components, text corresponding to the components, and lines connecting these components.

[0109] S2. Identify components in the target image;

[0110] S2.1 Construct a component recognition model to identify components in target images based on deep learning;

[0111] Construct a YOLO model by dividing the target image into a grid and assigning each grid to detect if the center point falls within the target object.

[0112] S2.1.1 Preprocess the target image to be identified;

[0113] S2.1.2 Perform data labeling;

[0114] Select and mark the component bounding boxes in the target image, including the type of the component and the position of the marked box. The marked box should contain the entire component and have the smallest possible area of ​​blank space.

[0115] S2.1.3 Perform sample data augmentation

[0116] S2.1.3.1 Randomly extract local regions from the same image and generate a first sample image based on the distribution of the extracted local regions;

[0117] The type and location information of each element in the sample image are labeled. A rectangular area is randomly extracted from the image. Based on the position of the rectangular area in the original image and the position of each element in the original image, the position coordinates of each element in the rectangular area relative to the rectangular area can be calculated. Thus, the labeling information of the elements in the rectangular area can be obtained, thereby obtaining a large number of new training samples.

[0118] S2.1.3.2 Add random noise information to the first sample image and generate a new first sample image;

[0119] Random noise, including Gaussian noise, salt-and-pepper noise, image blurring, and image brightness variations, is added to the first sample image to make each element in each training sample different, increase the features of the training samples, and improve the generalization ability of the element recognition model.

[0120] S2.2.3.3 Constructing virtual sample data;

[0121] Extract several component templates from a pre-defined standard template library, and stitch the extracted component templates together to form several first sample images;

[0122] After S2.1.4 performs sample training and updates, component detection and identification are carried out.

[0123] A YOLO component recognition model was constructed, and after training the model using the first sample set, component detection and recognition were carried out.

[0124] The input target image is divided into S×S grids, each grid is responsible for detecting target objects whose center point falls within it. Each grid contains B target bounding boxes, each bounding box consisting of a five-dimensional prediction parameter, including the center point coordinates (x, y), width and height (w, h), and a confidence score S. i Among them, the confidence score S i It is calculated by the following formula:

[0125] s i =Pr(o)×IoU

[0126] Where Pr(o) represents the probability that an object exists within the current mesh bounding box, and o represents the target object; IoU (Intersection over Union) demonstrates the accuracy of the target bounding box position predicted by the current model; assuming the predicted target bounding box is P, and the true target bounding box is t, box... t The box represents the bounding box of the real object in the image. p The target bounding box is represented; then the IoU is calculated by the following formula:

[0127]

[0128] Pr(C i |O) represents the posterior probability that the target belongs to a certain type of object i given that the bounding box contains the target; assuming there are K types of objects in the object detection task, then each grid predicts the object C of type i. i The conditional probability is Pr(C) i |O), i=1,2,…,K;

[0129] After calculating Pr(C) i After |O), during testing, the confidence level that an object exists within a target bounding box can be calculated, as shown in the following formula:

[0130]

[0131] Component identification is performed based on the confidence level that an object exists within the target bounding box;

[0132] S2.2 Component recognition in target images is achieved based on template matching;

[0133] S2.2.1 Establish a standard template library;

[0134] Construct component templates corresponding to process components, electrical components, and panel components. The template library for process components includes schematic diagrams of component icons, icon names, numbers, English names, icon file paths, auxiliary identification keywords, flow direction, interface direction, number of interfaces, and local identification elements. The template library for electrical components has similar elements to that for process components. The template library for panel components includes knobs, dials, etc. The panel diagram template library only needs to extract the key components at a uniform resolution.

[0135] S2.2.2 Establish multiple template libraries, with component templates in different template libraries having different resolutions;

[0136] S2.2.2.1 Extract several component templates of different resolutions to form multiple different template libraries;

[0137] S2.2.2.2 Sort the component template libraries of different resolutions from high to low resolution as input for baseline resolution optimization;

[0138] S2.2.2.3 Use component template libraries of different resolutions to perform template matching on different flowcharts. All categories of elements are identified and the component matching values ​​and average values ​​on different flowcharts are output.

[0139] S2.2.2.4 Select the optimal resolution template to obtain the template library with the optimal resolution;

[0140] Templates with a matching mean greater than 0.95 accounting for more than 90% of the total number of component categories are used as the optimal resolution templates;

[0141] S2.2.3 Perform template matching and identify the components in the target image based on the matching results;

[0142] Based on the target image, the target is searched in the template library; the image to be detected is traversed from left to right and from top to bottom using a matching algorithm, and then the template is matched with the sub-image blocks from top to bottom; and the components in the target image are identified based on the matching results.

[0143] Matching the component template with the sub-image block specifically includes: feature matching, edge contour matching, segmentation matching, pixel proportion matching, and color matching;

[0144] Feature matching involves extracting image features from a component template and a sub-image patch, and determining whether the component template and the sub-image patch match based on the matching degree of their image features. This is achieved by selecting multiple features for comprehensive matching and by adjusting parameters to obtain the optimal feature matching parameters. The feature extraction algorithms that can be used include, but are not limited to, LBP local feature matching, SIFT feature matching, ORB feature matching, and SURF feature matching.

[0145] Edge contour matching is achieved by extracting the image contours of the component template and the sub-image block, and determining whether the component template and the sub-image block match based on the degree of matching between the component template and the image contours of the sub-image block.

[0146] Segmentation matching involves segmenting the component template and sub-image blocks, and then matching the corresponding segmented regions of the component template and sub-image blocks.

[0147] Pixel ratio matching is based on the ratio of the number of pixels with gray values ​​greater than a predetermined threshold in the element template and the sub-image block to determine whether the element template and the sub-image block match.

[0148] Color matching determines whether a component template and a sub-image block match based on the color information of a specified area in the component template and the sub-image block.

[0149] When identifying components in a target image based on matching structures, the specific steps for setting multiple threshold parameters for various template matching identifications are as follows:

[0150] First, set a relatively high matching parameter to match the easily matched elements, and then remove those elements.

[0151] Set lower matching parameters for the remaining components and use multiple matching methods simultaneously to match the remaining components;

[0152] After repeating the above two steps, provide a prompt to the user if there are still unrecognized entries.

[0153] S2.3 Component recognition in the target image is achieved based on keyword matching;

[0154] Identify the keywords corresponding to each element in the target image, match the auxiliary keywords corresponding to the element templates with the identified keywords, and identify the elements in the target image based on the matching results;

[0155] S3. Recognize the text in the target image;

[0156] S3.1 Perform image preprocessing on the acquired target image;

[0157] S3.1.1 Perform image grayscale processing on the acquired image to be identified;

[0158] S3.1.2 Filter the grayscale image to reduce image noise;

[0159] S3.1.3 Binarize the filtered image;

[0160] S3.2 Perform target detection on the text region in the target image;

[0161] The YOLO algorithm is used to obtain boundary coordinates and class probabilities from image pixels, and the positions and information of all targets in the target image are confirmed and identified.

[0162] S3.3 Collect and recognize the text in the target image;

[0163] The CRNN text recognition model is used to recognize text in a target image. The text recognition model is obtained by training a pre-trained model and a constructed data sample set.

[0164] S3.3.1 Obtain sample data using data augmentation techniques;

[0165] S3.3.1.1 Add random noise: Randomly add salt and pepper noise, Gaussian noise, image blur noise, and image erosion noise to the sample images to form new sample data;

[0166] S3.3.1.2 For numbers, letters, and standard symbols that are originally image data samples, a template set of standard characters is established, and a large number of sample images are randomly generated using sample synthesis.

[0167] S3.3.2 Perform sliding recognition on the target image;

[0168] Sub-image partitioning: The entire target image is divided into multiple sub-images by cropping.

[0169] Image sliding: Use sliding sub-images to make the segmentation box slide across the entire target image;

[0170] Image merging: The non-maximum suppression (NMS) algorithm is used to remove duplicate regions before image merging.

[0171] S3.3.3 Recognize the direction of text in the target image;

[0172] S3.3.3.1 uses a binary classification model for detection, dividing horizontal and vertical text into two categories of samples. The binary classification model is trained to determine the direction of the text, and the text direction is then corrected.

[0173] S3.3.3.2 Rule-based detection: Since text is composed of multiple characters, the direction of the text can be determined based on the length and width of the text area.

[0174] S4. After removing the identified components and text from the target image, the lines in the target image are then identified.

[0175] The connection in the target image is identified as a separate element, and the position information of the connection is finally output. After binarizing the target image, the number of consecutive pixels is determined to be continuous by detecting whether there is a number of consecutive pixels whose average pixel value is less than a predetermined pixel threshold. If the number of consecutive pixels exceeds a predetermined threshold and the average pixel value is close to 0, the number of consecutive pixels is determined to be continuous.

[0176] S5. Identify the connection relationships in the target image based on the properties of the lines and components in the target image;

[0177] S5.1 Based on the obtained connection position information, identify the position information of corner points in the target image;

[0178] Corner points include the intersections between lines in an image, as well as the intersections between lines and elements. The positional information of corner points can be identified using the Shi-Tomasi detection algorithm.

[0179] S5.2 Based on the position information of the corner points in the identified target image, identify the inflection points of the connecting lines in the target image and the connection points of each component;

[0180] Based on the obtained connection positions and corner point positions, determine whether there are at least two lines connecting each corner point in the target image and both of them are straight lines; for any corner point, if there are at least two lines connecting it and both of them are straight lines, then the corner point is identified as an inflection point.

[0181] For cases where the connecting lines are horizontal or vertical, the inflection point can be determined by detecting the right-angled sides. The specific steps are as follows:

[0182] Based on the identified connection and corner position information, determine whether the connecting lines to each corner in the target image form a right angle. If the connecting lines to the corner form a right angle, the corner is identified as an inflection point.

[0183] The specific steps for the right angle detection are as follows:

[0184] For any corner point, a rectangular target area is defined centered on that corner point. A central rectangular area is then defined at the center of the target area. The areas above and below the central area within the target area are defined as the vertical detection area, and the areas to the left and right of the central area within the target area are defined as the horizontal detection area. When there is a vertical line within the vertical detection area and the length of the vertical line is approximately equal to the height of the vertical detection area, and there is a horizontal line within the horizontal detection area and the length of the horizontal line is approximately equal to the width of the horizontal detection area, it is determined that the line connecting to the corresponding corner point forms a right angle.

[0185] After identifying the inflection point, exclude the inflection point and the intersection point formed by the curve from the identified corner points. The remaining corner points are the connection points of the components.

[0186] S5.3 Identify possible connectivity relationships in the target image;

[0187] S5.3.1 Identify whether there is a possible connection relationship between two adjacent elements;

[0188] If two adjacent elements are detected to have a connection point facing opposite directions and there is a connection line between the two adjacent elements that satisfies a first predetermined condition, then a possible connection relationship is identified between the two adjacent elements. The first predetermined condition is, for example, that the length of the connection line is greater than or equal to a first multiple threshold of the distance between the two elements, such as 0.95 times. The connection line here can be a solid line, a dashed line, or a diagonal line.

[0189] S5.3.2 Identify whether there are possible connections between adjacent elements and inflection points;

[0190] If an adjacent element and an inflection point are connected by a line that satisfies a first predetermined condition and the corresponding element has a connection point facing that line, then it is determined that there is a possible connection relationship between the corresponding element and the inflection point.

[0191] S5.3.3 Identify whether there is a possible connection between two adjacent inflection points;

[0192] If a line that satisfies a first predetermined condition is detected between two adjacent inflection points, then it is determined that there is a possible connection between the two related inflection points.

[0193] S5.4 Based on the properties of the components, deterministic and impossible connection relationships are filtered out from the obtained possible connection relationships;

[0194] S5.4.1 When there are horizontal or vertical straight line connections between components, use a probability identification method to filter out deterministic connection relationships and impossible connection relationships;

[0195] S5.4.1.1 Calculate the overlap ratio D of the opposite surfaces of two adjacent elements:

[0196]

[0197] Where A and B represent the widths of the opposite faces of two adjacent elements, respectively; and C represents the overlap width of the opposite faces of two adjacent elements.

[0198] S5.4.1.2 Calculate the ratio G of the connection length to the spacing between two adjacent components:

[0199]

[0200] Where F represents the length of the connection between two components, and E represents the length of the connection and the spacing between two adjacent components;

[0201] S5.4.1.3 When the overlap ratio D of two components and the ratio of connection length to spacing G reach the corresponding predetermined probability threshold, it is determined that there is a possible connection relationship between the two adjacent components.

[0202] Based on the D and G values, the possible connections between each element are sorted, and deterministic connections are filtered according to the element's attribute definition and sorting results. The larger the D and G values, the greater the probability that the connection is a deterministic connection.

[0203] S5.4.2 When a diagonal line is detected between two adjacent elements, if both elements have unconnected connection points and there are no other possible connections between them, then a deterministic connection relationship is determined between the two elements. When a diagonal line is detected between an adjacent element and an inflection point, if the element has unconnected connection points and there are no other possible connections between them, then a deterministic connection relationship is determined between the element and the inflection point.

[0204] S5.4.3 Filter out impossible connections from possible connections;

[0205] When the number of deterministic connections of an element in a certain direction is the same as the number of connection points defined by the attribute of the element in the method, the uncertain possible connections of the element in that direction are determined to be impossible connections; when the total number of deterministic connections of an element is the same as the total number of connection points defined by the attribute of the element, the uncertain possible connections of the element are determined to be impossible connections.

[0206] S5.4.4 When the number of connection points of a component is found to be inconsistent with the number of connection points defined in the component's attributes, or when there are unconnected connection points, the corresponding prompt message will be output.

[0207] When the difference between D and G corresponding to different possible connection relationships of the same component is within a predetermined difference range, the corresponding prompt message is output;

[0208] When the number of connection points of a detected element is inconsistent with the number of connection points defined in the element's attribute, a corresponding prompt message will be output.

[0209] When a component has an unconnected connection point, output the corresponding prompt message.

[0210] S6. Based on the attributes of the components and the connection relationships in the target image, identify the connection directions in the target image;

[0211] S6.1 Set parameters related to the flow direction in the component attribute definition;

[0212] In the component attribute definition, parameters related to the flow direction are set, indicating whether the component has directionality. For example, a pump has directionality, and the flow direction of the fluid can be identified through image features. Therefore, the pump attribute parameter is set to True. However, for a valve, its flow direction cannot be identified through an image. Therefore, this attribute of the valve is set to False.

[0213] S6.2 Identify the flow direction with flow direction elements;

[0214] By using template matching, hash verification, or feature point matching, it is possible to determine which type of template the element to be identified matches, thereby identifying the flow direction of the element;

[0215] S6.3 updates the set of primitives whose flow direction cannot be identified;

[0216] For a line without a branching point, its flow direction is consistent. If there is an element with a defined flow direction on the line, the flow direction of other elements on the line can be determined, thereby updating the set of primitives whose flow direction cannot be identified.

[0217] S6.4 For components whose flow direction cannot be determined, the flow direction shall be given according to the rule of top to bottom and left to right;

[0218] S7. Check the flowchart recognition results in the target image;

[0219] S7.1 calculates and evaluates the accuracy of target image recognition;

[0220] The predicted probability of each element is calculated using the YOLO model;

[0221] By setting an appropriate threshold T, some components with a predicted probability lower than the threshold are identified as incorrect, thus obtaining the recognition accuracy α1. The threshold T can be selected based on experiments or the 3σ principle.

[0222] The manually defined attributes are compared with the automatically recognized attributes. When two attributes of the same component are inconsistent, the component is considered to have an identification error, and the identification accuracy α2 is obtained.

[0223] In this embodiment, all lines in the flowchart are assumed to be horizontal and vertical. After identifying the positions of the components in the flowchart, all possible connections for any component are listed based on the component's position coordinates. Then, based on rule reasoning, the possible connections are deleted one by one until only deterministic connections remain. The recognition effect is evaluated by calculating the ratio of the number of deterministic connections to the number of possible connections. The closer the ratio is to 1, the better the recognition effect, that is, all possible connections are deterministic connections, and the recognition accuracy α3 is obtained.

[0224] The recognition accuracy rates obtained from the above three aspects are weighted to obtain the final recognition accuracy rate of the target image:

[0225] α=β1*α1+β2*α2+β3*α3

[0226] The weights β1 to β3 were determined experimentally.

[0227] S7.2 Checks the automated recognition results in the target image and provides prompts for uncertain recognition results;

[0228] S7.2.1 Automated recognition and inspection of target images based on attribute characteristics;

[0229] The manually defined attributes are compared with the automatically identified attributes. If two attributes of the same element are inconsistent, it is considered that the element may be misidentified. In the identification result file, each element will have an identification accuracy attribute. This attribute is compared with the set threshold to evaluate whether the result is reliable.

[0230] S7.2.2 Removes the identified components from the target image and fills in blank pixels, leaving unidentified components or those with inaccurately marked identification boxes in the original image, thereby quickly and accurately finding unidentified components.

Claims

1. A flowchart recognition method, characterized in that, The method specifically includes S1. Obtain the target image; S2. Identify components in the target image; S3. Recognize the text in the target image; S4. After removing the identified components and text from the target image, identify the connecting lines in the target image. S5. Identify the connection relationships in the target image based on the properties of the lines and components in the target image; S5.1 Based on the obtained connection position information, identify the position information of corner points in the target image; S5.2 Based on the position information of the corner points in the identified target image, identify the inflection points of the connecting lines in the target image and the connection points of each component; S5.3 Identify possible connectivity relationships in the target image; S5.3.1 Identify whether there is a possible connection relationship between two adjacent elements; If two adjacent elements are found to have connection points facing opposite directions and there is a connection between the two adjacent elements that satisfies a first predetermined condition, then a possible connection relationship between the two adjacent elements is identified. S5.3.2 Identify whether there are possible connections between adjacent elements and inflection points; If an adjacent element and an inflection point are connected by a line that satisfies a first predetermined condition and the corresponding element has a connection point facing that line, then it is determined that there is a possible connection relationship between the corresponding element and the inflection point. S5.3.3 Identify whether there is a possible connection between two adjacent inflection points; If a line that satisfies a first predetermined condition is detected between two adjacent inflection points, then it is determined that there is a possible connection between the two related inflection points. S5.4 Based on the attributes of the components, deterministic and impossible connection relationships are filtered out from the obtained possible connection relationships. The specific steps are as follows: S5.4.1 When there are horizontal or vertical straight line connections between components, a probabilistic identification method is used to filter deterministic and impossible connection relationships, including: calculating the overlap ratio D of the opposite faces of two adjacent components, and calculating the ratio G of the connection length to the spacing between two adjacent components; sorting the possible connection relationships of each component based on the values ​​of D and G, and filtering deterministic connection relationships according to the component attribute definitions and sorting results; S5.4.2 When a diagonal line is detected between two adjacent elements or between an element and an inflection point, determine whether a deterministic connection relationship exists based on whether there are unconnected connection points and other possible connections between the elements. S5.4.3 Based on whether the number of deterministic connections of an element in a certain direction or in all aspects is consistent with the number of connection points defined in the element's attributes, the uncertain possible connections of the element are determined as impossible connections. S6. Based on the attributes of the components and the connection relationships in the target image, identify the connection directions in the target image; S6.1 Set parameters related to the flow direction in the component attribute definition; S6.2 Identify the flow direction with flow direction elements; S6.3 Update the set of primitives whose flow direction cannot be identified; S6.4 For components whose flow direction cannot be determined, the flow direction shall be given according to the rule of top to bottom and left to right.

2. The flowchart recognition method according to claim 1, characterized in that, The identification of elements in the target image in step S2 can be achieved using any one or a combination of the following methods: (1) Construct a component recognition model and perform component recognition in the target image based on deep learning; (2) Component recognition in target images is achieved based on template matching; (3) Recognize the components in the target image based on keyword matching.

3. The flowchart recognition method according to claim 1, characterized in that, The specific steps for character recognition in the target image in step S3 are as follows: S3.1 Perform image preprocessing on the acquired target image; S3.2 Perform target detection on the text region in the target image; The YOLO algorithm is used to obtain boundary coordinates and class probabilities from image pixels, and the positions and information of all targets in the target image are confirmed and identified. S3.3 Collect and recognize the text in the target image; The CRNN text recognition model is used to recognize text in a target image. The text recognition model is obtained by training a pre-trained model and a constructed data sample set.

4. The flowchart recognition method according to claim 3, characterized in that, The specific steps for recognizing text in the target image are as follows: (1) Obtain sample data using data augmentation techniques; Noise is randomly added to the sample images to create new sample data; By establishing a template set of standard characters, a large number of sample images are randomly generated using sample synthesis. (2) Perform sliding recognition on the target image; The entire target image is divided into multiple sub-images by cropping. A sliding sub-image segmentation method is used, allowing the segmentation box to slide across the entire target image; The non-maximum suppression algorithm is used to remove duplicate identified regions and then the images are merged. (3) Recognize the direction of text in the target image; The detection is performed using a binary classification model, which divides horizontal and vertical text into two categories of samples. The binary classification model is trained to determine the direction of the text, and the text direction is then corrected. The system performs rule-based detection, determining the direction of the text based on the length and width of the text region.

5. The flowchart recognition method according to claim 1, characterized in that, After identifying the connection direction in the target image in step S6, the method further includes checking the flowchart recognition result in the target image as follows: S7.1 calculates and evaluates the accuracy of target image recognition; The predicted probability of each element is calculated using the YOLO model; S7.2 Check the automated recognition results in the target image and provide prompts for uncertain recognition results; S7.2.1 Automated recognition and inspection of target images based on attribute characteristics; The manually defined attributes are compared with the automatically identified attributes. If two attributes of the same element are inconsistent, it is considered that the element may be misidentified. In the identification result file, each element will have an identification accuracy attribute. This attribute is compared with the set threshold to evaluate whether the result is reliable. S7.2.2 Removes the identified components from the target image and fills in blank pixels, leaving unidentified components or those with inaccurately marked identification boxes in the original image, thereby quickly and accurately finding unidentified components.

6. The flowchart recognition method according to claim 1, characterized in that, The specific steps for identifying the connection direction in the target image in step S6 are as follows: S6.1 Set parameters related to the flow direction in the component attribute definition; In the component property definition, set parameters related to the flow direction, that is, indicate whether the component has directionality; S6.2 Identify the flow direction with flow direction elements; By using template matching, hash verification, or feature point matching, it is possible to determine which type of template the element to be identified matches, thereby identifying the flow direction of the element; S6.3 Update the set of primitives whose flow direction cannot be identified; For a line without branching points, its flow direction is consistent. If there is an element with a defined flow direction on the line, the flow direction of other elements on the line can be determined, thereby updating the set of primitives whose flow direction cannot be identified.

7. The flowchart recognition method according to claim 2, characterized in that, The specific steps for constructing the component recognition model and recognizing components in the target image based on deep learning are as follows: Construct a YOLO model by dividing the target image into a grid and assigning each grid to detect if the center point falls within the target object. S2.1.1 Preprocess the target image to be identified; S2.1.2 Perform data labeling; Select and mark the components in the target image, including the type of the component and the position of the mark box; S2.1.3 Perform sample data augmentation S2.1.3.1 Randomly extract local regions from the same image and generate a first sample image based on the distribution of the extracted local regions; S2.1.3.2 Add random noise information to the first sample image and generate a new first sample image; S2.2.3.3 Constructing virtual sample data; Extract several component templates from a pre-defined standard template library, and stitch the extracted component templates together to form several first sample images; After S2.1.4 performs sample training and updates, component detection and identification are carried out. A YOLO component recognition model was constructed, and after training the model using the first sample set, component detection and recognition were carried out. The specific steps for identifying components in a target image based on template matching are as follows: S2.2.1 Establish a standard template library; Construct component templates that include process components, electrical components, and panel components; S2.2.2 Establish multiple template libraries, with component templates in different template libraries having different resolutions; S2.2.2.1 Extract several component templates of different resolutions to form multiple different template libraries; S2.2.2.2 Sort the component template libraries of different resolutions from high to low resolution as input for baseline resolution optimization; S2.2.2.3 Use component template libraries of different resolutions to perform template matching on different flowcharts. All categories of elements are identified and the component matching values ​​and average values ​​on different flowcharts are output. S2.2.2.4 Select the optimal resolution template to obtain the template library with the optimal resolution; S2.2.3 Perform template matching and identify components in the target image based on the matching results; Based on the target image, the target is searched in the template library; a matching algorithm is used to traverse multiple sub-image blocks of the image to be detected from left to right and from top to bottom, and then the template is matched with the sub-image blocks from top to bottom; and based on the matching results, the components in the target image are identified. Matching the component template with the sub-image block specifically includes: feature matching, edge contour matching, segmentation matching, pixel proportion matching, and color matching; When identifying components in a target image based on matching structures, the specific steps for setting multiple threshold parameters for various template matching identifications are as follows: First, set a relatively high matching parameter to match the easily matched elements, and then remove those elements. Set lower matching parameters for the remaining components and use multiple matching methods simultaneously to match the remaining components; After repeating the above two steps, provide a prompt to the user if there are still unrecognized entries.

Images