Flow chart identification method and device, equipment, medium and product

Abstract

The invention discloses a flow chart recognition method, device and equipment, a medium and a product, and relates to the technical field of computer science and information. The method comprises the steps of performing feature extraction on a to-be-identified flow chart to obtain a first feature of each node element and a second feature of each non-node element in the flow chart; determining spatial features of each non-node element according to the geometric features of each node element; and identifying the first feature, the second visual feature and the spatial feature to obtain an identification result of the flow chart. By adopting the technical scheme, the problem that the structure in the flow chart cannot be reliably identified is solved, spatial information among the node elements in the flow chart is further determined by acquiring dimension information of geometry, vision, semantics and the like of the node elements or non-node elements in the flow chart, and automatic identification of structural logic in the flow chart is realized.

Description

Technical Field

[0001] This invention relates to the fields of computer science and information technology, and in particular to a flowchart recognition method, apparatus, device, medium, and product. Background Technology

[0002] In rail transit auxiliary decision-making systems, flowcharts are a commonly used tool to visually display various business processes, control logic, and decision-making rules.

[0003] Currently, when recognizing flowcharts for rail transit auxiliary decision-making, a step-by-step method of extracting text and images is typically used, with common algorithms including convolutional neural networks and recurrent neural network models. However, current recognition methods can only identify individual graphic elements and some simple relationships. They cannot reliably recognize complex logical structures in flowcharts, such as sequential, selection, and loop structures. This can lead to misunderstandings of the flowchart's intent in practical applications, thus affecting the accuracy and reliability of rail transit auxiliary decision-making. Furthermore, actual rail transit flowcharts contain a large number of non-standardized icons, and existing recognition technologies cannot automatically correct or highlight these non-standardized icons, potentially causing bias in the recognition results and reducing the practicality of flowchart recognition.

[0004] Therefore, existing flowchart recognition technologies for rail transit auxiliary decision-making have significant shortcomings in terms of accuracy, comprehensiveness, and reliability. Summary of the Invention

[0005] This invention provides a flowchart recognition method, apparatus, device, medium, and product. By adopting this technical solution, the problem of unreliable recognition of the structure in the flowchart is solved. By acquiring geometric, visual, semantic, and other dimensional information of node elements or non-node elements in the flowchart, the spatial information between node elements in the flowchart is further determined, thereby realizing the automatic recognition of the structural logic in the flowchart.

[0006] According to one aspect of the present invention, a flowchart recognition method is provided, comprising:

[0007] Feature extraction is performed on the flowchart to be identified to obtain the first feature of each node element and the second feature of each non-node element in the flowchart; wherein, the first feature includes geometric features, first visual features and semantic features; the second feature includes second visual features and second contour features; the geometric features include vertex coordinates, first contour features and topological relationships between node elements;

[0008] The spatial characteristics of each non-node element are determined based on the geometric characteristics of each node element.

[0009] The first feature, the second visual feature, and the spatial feature are identified to obtain the recognition result of the flowchart.

[0010] According to another aspect of the present invention, a flowchart recognition device is provided, comprising:

[0011] The feature extraction module is used to extract features from the flowchart to be identified, and obtain the first feature of each node element and the second feature of each non-node element in the flowchart; wherein, the first feature includes geometric features, first visual features and semantic features; the second feature includes second visual features and second contour features; the geometric features include vertex coordinates, first contour features and topological relationships between node elements;

[0012] The spatial feature determination module is used to determine the spatial features of each non-node element based on the geometric features of each node element.

[0013] The recognition module is used to recognize the first feature, the second visual feature, and the spatial feature to obtain the recognition result of the flowchart.

[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0015] At least one processor; and

[0016] A memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the flowchart recognition method according to any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the flowchart recognition method described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the flowchart recognition method according to any embodiment of the present invention.

[0020] The technical solution of this invention, by acquiring geometric, visual and semantic dimensional information of node elements or non-node elements in a flowchart, further determines the spatial information between node elements in the flowchart, solves the problem of unreliable identification of the structure in the flowchart, and realizes automatic identification of the structure in the flowchart.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a flowchart recognition method provided by an embodiment of the present invention;

[0024] Figure 2 This is a design example of a flowchart to be identified according to an embodiment of the present invention;

[0025] Figure 3 This is a flowchart of a method for determining the spatial features of non-node elements according to an embodiment of the present invention;

[0026] Figure 4 This is a flowchart of a method for determining flowchart recognition results according to an embodiment of the present invention;

[0027] Figure 5 This is a flowchart of a flowchart recognition method provided by an embodiment of the present invention.

[0028] Figure 6 This is a schematic diagram of a flowchart recognition device according to an embodiment of the present invention;

[0029] Figure 7 This is a schematic diagram of the structure of an electronic device that implements the flowchart recognition method of the present invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] Furthermore, it should be noted that the collection, storage, use, processing, transmission, provision, and disclosure of data to be processed in the technical solution of this invention all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0033] Figure 1 This invention provides a flowchart recognition method according to an embodiment of the present invention. This embodiment is applicable to the recognition of flowchart structures, especially to the recognition of flowcharts in rail transit auxiliary decision-making. The method can be executed by a flowchart recognition device, which can be implemented in hardware and / or software and can be configured in a server. Figure 1 As shown, the method includes:

[0034] S110. Perform feature extraction on the flowchart to be identified to obtain the first feature of each node element and the second feature of each non-node element in the flowchart; wherein, the first feature includes geometric features, first visual features and semantic features; the second feature includes second visual features and second contour features; the geometric features include vertex coordinates, first contour features and topological relationships between node elements.

[0035] The flowchart to be identified can be a completed flowchart; node elements are the nodes in the flowchart; non-node elements are the arrow segments connecting the nodes in the flowchart; the first visual feature is the line color, fill color, and line type of the node elements; the semantic feature is the text information and label information corresponding to the node elements; the second visual feature is the line color and line type of the non-node elements; the second contour feature is the arrow contour of the non-node elements; the vertex coordinates are the corner points of the bounding box corresponding to each node element; the first contour feature is the shape of the node element, which can be a rectangle, rhombus, or ellipse; the topological relationship between node elements is the connection relationship between two node elements.

[0036] Specifically, computer vision methods can be used to extract features from the flowchart to be identified, obtaining the bounding box corner coordinates, shape, line color, fill color, line type, topological relationship between node elements, text information, and label information of each node element; and obtaining the arrow outline, line color, and line type of each non-node element.

[0037] For example, such as Figure 2 The diagram shows a design example of a flowchart to be identified. The flowchart includes node elements and non-node elements. Non-node elements are arrow segments between two node elements with a topological relationship. The upper left corner of the entire flowchart is taken as the origin, with the X-axis pointing to the right and the Y-axis pointing downwards. The flowchart is extracted to obtain the geometric features of each node. Taking node element C as an example, its geometric features include vertex coordinates [(90,200), (100,190), (110,200), (100,210)], a rhombus shape, and topological relationships including nodes. There are three paths: from node B to node C, from node C to node D, and from node C to node E. Furthermore, the visual features of node C are: a yellow fill color, a solid line for its bounding box, and a black line for its bounding box. The semantic features of node element C include the text "Register or not," with the corresponding tag being "Decision." For the arrow segment of non-node element CE, the flow direction of CE is determined by whether its outline is an arrow; its second visual feature is a black line with a solid line type.

[0038] It is understandable that by extracting features from the flowchart to be identified, three sets of features are obtained: geometric feature layer, visual feature layer, and semantic feature layer. This ensures the accurate identification of each node element or non-node element in the flowchart, and the accurate identification of the flow direction between each node element is ensured through multi-dimensional features.

[0039] In one optional embodiment of the present invention, the flowchart to be identified is an original flowchart obtained from an electronic document, and the original flowchart is preprocessed to obtain the flowchart to be identified. Specifically, vector graphic data, text content and format information can be directly extracted from the electronic document, or images can be captured by a camera or a scanner to extract multiple sets of spatial vector graphic data, text content and format, etc., and the acquired visual images are processed by grayscale conversion, noise reduction, binarization and other processes to enhance the clarity of graphics and lines, which is convenient for subsequent analysis.

[0040] S120. Determine the spatial characteristics of each non-node element based on the geometric characteristics of each node element.

[0041] Among them, the spatial features are the spatial positions of each arrow segment and the relative positional relationship between two node elements, including the starting coordinates, ending coordinates, and relative positional relationship of the arrow segment.

[0042] Specifically, the spatial position of each node element can be determined based on the coordinates of the corner points of the bounding box of each node element. The spatial position of non-node elements can be obtained based on the spatial position of each node element and the topological relationship between each node element, so as to represent the flow direction of the topological relationship between each node element.

[0043] like Figure 3 The method shown is for determining the spatial features of non-node elements. It determines the spatial features of each non-node element based on the geometric features of each node element, including:

[0044] S121. Determine the center point coordinates of each node element based on the vertex coordinates of each node element.

[0045] The center point coordinates are used to represent the position of the node element.

[0046] Specifically, the intermediate values ​​of the vertex coordinates of each node element are calculated to obtain the coordinates of the center point of each node element. Figure 2 Taking node element C as an example, its vertex coordinates are [(90,200), (100,190), (110,200), (100,210)]. Then, by calculating the median value of its vertex coordinates, its center point coordinates are (100,200).

[0047] S122. Calculate the relative positional relationships between node elements based on the center point coordinates of each node element and the topological relationships between each node element.

[0048] The relative positional relationship refers to the positional relationship between a node element with a topological relationship and another node element, that is, the distance between two node elements with a topological relationship.

[0049] Specifically, two node elements with a topological relationship can be obtained, and the coordinates of their center points can be interpolated to obtain the distance offsets in the X and Y axes, respectively.

[0050] by Figure 2 Taking the topological relationship between node element C and node element E as an example, the center point coordinates of node element C are (100, 200) and the center point coordinates of node element E are (100, 250). By performing the difference operation, the distance between node element C and node element E is (Δx = 0, Δy = 50).

[0051] S123. Calculate the arrow angle of each non-node element based on the relative positional relationship between node elements.

[0052] The arrow angle refers to the direction in which the arrow segment points.

[0053] Specifically, the arrow angle is obtained based on the offsets of the node elements on the X-axis and Y-axis in their relative positional relationship.

[0054] by Figure 2 Taking the topological relationship between node element C and node element E as an example, the relative positional relationship between node element C and node element E is (Δx=0, Δy=50). Based on their offsets on the X-axis and Y-axis, the angle is calculated to obtain that the arrow angle is 90°.

[0055] S124. The center point coordinates of each node element, the relative positional relationship between node elements, and the arrow angles of non-node elements are used as the spatial features of each non-node element.

[0056] Specifically, the center point coordinates of each node element, the relative positional relationship of node elements, and the arrow angles of non-node elements are used as the spatial features of non-node elements.

[0057] Taking the non-node element CE as an example, which represents the flow direction of the line segment from node element C to node element E, the starting coordinates of the arrow line segment are the center point coordinates of node element C (100, 200), and the ending coordinates of the arrow line segment are the center point coordinates of node element E (100, 250); the relative positional relationship of the arrow line segment is (Δx=0, Δy=50), and the arrow angle of the arrow line segment is 90°.

[0058] It is understandable that the spatial characteristics of each non-node element are determined by the vertex coordinates of each node element and the topological relationships between node elements, so as to accurately represent the flow direction of the arrow segments of each non-node element, and to accurately identify the flow of the flowchart in the future.

[0059] In an optional embodiment of the present invention, the starting coordinates and ending coordinates of each non-node element can also be represented by the vertex coordinates of each node element. The center point coordinates of the node element can be set according to the requirements of relevant technicians. The spatial characteristics of the non-node elements can be used to represent the flow direction between node elements with topological relationships. The embodiments of the present invention do not specifically limit this.

[0060] S130. Identify the first feature, the second visual feature, and the spatial feature to obtain the recognition result of the flowchart.

[0061] The flowchart identification results include whether each path in the flowchart is valid and whether the structure of the flowchart is accurate.

[0062] Specifically, by identifying the multi-dimensional features of node elements and non-node elements in the flowchart, it is possible to determine whether each path in the flowchart is valid and whether the structure of the flowchart is accurate.

[0063] It is understandable that when processing flowcharts, by acquiring geometric, visual, and semantic information of node elements and determining the geometric, visual, and spatial features of non-node elements, and by recognizing the multi-dimensional features of node and non-node elements, the meaning expressed by the flowchart can be accurately and comprehensively understood.

[0064] like Figure 4 The method for determining the flowchart recognition result shown herein involves recognizing a first feature, a second visual feature, and a spatial feature to obtain the flowchart recognition result, including:

[0065] S131. Vectorize the first feature of the node element to obtain the first feature vector of the node element.

[0066] Vectorization involves converting the textual information in the first feature of a node element into a digital representation.

[0067] Specifically, the first feature of the node element is digitized, that is, the contour feature is digitized. An ellipse is represented by the number "0", a rectangle by "1", a rhombus by "2", and an arrow by "3". In its first visual feature, color is represented by RGB, and the line type is digitized as follows: solid lines are represented by "1", and dashed lines by "2". In its semantic features, text information undergoes dimensionality reduction, and label information is also digitized. For example, a "start" label is represented by "0", an "end" label by "1", a "process" label by "2", a "decision" label by "3", an "input / output" label by "4", and an "error" label by "5". The digitized first feature is used as the first feature vector of the node element.

[0068] For example, taking node C as an example, its geometric features include a rhombus shape, which is represented by "2"; its visual features include a yellow fill color, which is represented by (255,255,0,) and a solid line, which is represented by "0"; the bounding box line color is black, which is represented by (0,0,0,); its semantic features include the text information "whether to register", which is reduced to 2 dimensions and encoded as (0.12,-0.05); its label information is a decision, which is represented by "3".

[0069] In an optional embodiment of the present invention, the digitized first feature is uniformly processed by setting the geometric feature layer to 7 units, including contour features, vertex coordinates, X-axis offset and Y-axis offset; its visual feature layer is set to 7 units, including node fill color, line color and line type; its semantic feature layer is set to 5 units, including label information and text information dimensionality reduction value; if the units in each of the above dimensional layers are insufficient, they are supplemented with 0; taking node C as an example, its geometric feature layer includes the first contour feature and vertex coordinates, which is expressed as [2,(90,200),(100,190),(110,200),(100,210)0,0]; its visual feature layer includes node fill color, line color and line type, which is expressed as [255,255,0,0,0,0,0]; its semantic feature layer includes text information dimensionality reduction value and label information, which is expressed as [3,0.12,-0.05,0,0].

[0070] S132. Based on the spatial features and second features of non-node elements, perform vectorization processing to obtain the second feature vector of non-node elements.

[0071] Vectorization involves converting the textual information in the second feature of non-node elements into a digital representation.

[0072] Specifically, the second feature of non-node elements is digitized, that is, the second contour feature is digitized. The second contour feature of an ellipse is represented by the number "0", the second contour feature of a rectangle is represented by the number "1", the second contour feature of a rhombus is represented by the number "2", and the second contour feature of an arrow is represented by the number "3". In its second visual feature, the color is represented by RGB, and the line type is digitized as follows: solid lines are represented by the number "1" and dashed lines are represented by the number "2". The digitized second feature is used as the second feature vector of non-node elements.

[0073] For example, taking CE, a non-node element, as an example, if the shape of its second contour feature is an arrow, it is represented by "3"; if its line type is a solid line, it is represented by "0"; if the line color is black, it is represented as (0,0,0,).

[0074] In an optional embodiment of the present invention, the digitized second feature is standardized by setting the geometric feature layer to 7 units, including contour features, vertex coordinates, X-axis offset, and Y-axis offset; its visual feature layer is set to 2 units, including line type and arrow angle; if the units in each of the above-mentioned dimensional layers are insufficient, they are supplemented with 0; taking the arrow line segment CE in the non-node element as an example, its geometric feature layer includes the second contour feature, start coordinates, end coordinates, and relative position information, which is expressed as [2,(100,200),(100,250),0,0,0,50]; its visual feature layer includes line color, line type, and angle, which is expressed as [1,90]; it represents that the line segment flows from node element C to node element E.

[0075] S133. Perform logical judgment on the first feature vector of the node element and the second feature vector of the non-node element to obtain the recognition result of the flowchart.

[0076] The logical judgment involves determining the path of node elements in the flowchart and detecting the structure of the flowchart.

[0077] Specifically, the first feature vector of the node element is used to determine the path based on the second feature vector of the non-node element, thus determining whether all paths in the flowchart are valid.

[0078] In an optional embodiment of the present invention, the flowchart recognition result may further include generated code or a concise text description; such as Figure 2 The flowchart shown can be described as follows: Starting from "Entering the software", the user enters the "Login Page"; the user chooses whether to register: if no, the user is redirected to the "Registration Page"; if yes, the user is asked to enter a password; the user enters their account and password to log in, and the system checks the results: if the check result is that the account does not exist, an error message is displayed and the user is returned to the login page; if the check result is that the password is incorrect, the user is prompted to re-enter the password and the process loops; if the check result is that the account and password are correct, the user logs in successfully.

[0079] Optionally, logical judgments are performed on the first feature vector of node elements and the second feature vector of non-node elements to obtain the flowchart recognition result, including:

[0080] Perform loop detection on the second feature vector of non-node elements to obtain the loop detection result;

[0081] Nodes whose label information is the decision label are selected as target node elements.

[0082] Probability determination is performed on the first feature vector corresponding to the target node element to obtain the probability distribution result of the target node element;

[0083] Based on the second feature vector of non-node elements, the first feature vector of node elements is used to identify the flow chart and obtain the state information of the flow chart.

[0084] The results of loop detection, probability distribution, and state information are used as the recognition results of the flowchart.

[0085] Among them, loop detection is to perform loop detection on the structure with loops in the flowchart; the loop detection result is whether the loop structure is accurate; probability judgment is to perform probability detection on each judgment branch of the node element of the decision node; state information is whether the flow direction represented by the second feature vector of the non-node element conforms to the spatial coordinates.

[0086] Specifically, loop detection is performed on the second feature vectors of non-node elements to determine the loop structure of the flowchart and to determine the validity of the loops; based on the semantic feature layer of the node elements, the node elements whose label information is used for decision-making are found and are taken as target node elements, and the probability distribution of the first feature vector corresponding to the target node element is judged; the second feature vectors of non-node elements are used to describe the path of each line segment, and to perform flow recognition on the first feature vectors of each node element to determine whether each path is correct; the obtained loop detection, the probability distribution of the decision node, and the path state of the flowchart are used as the recognition result of the flowchart.

[0087] For example, such as Figure 2 As shown, for a loop structure consisting of an arrow segment from node element J to the corresponding non-node element JE, the vertex coordinates of the geometric feature layer in the first feature vectors of node elements J and E are compared with the arrow direction and offset in the geometric feature layer of the second feature vector of non-node element JE. If the corresponding offset is a decrease in the Y-axis, it is marked as correct; if the representation in node elements J and E is a decrease in the X-axis, it is marked as incorrect; if correct, the corresponding relationship is obtained from the text information in the semantic feature layer of the first feature vectors of node elements J and E as "Prompt for incorrect password - please re-enter password", indicating that it is a valid loop. According to the dynamic Bayesian update probability distribution results, if a probability judgment is made on the decision node element G, the corresponding output is P(account does not exist) = 0.1, P(incorrect password) = 0.4, P(correct input) = 0.5.

[0088] In an optional embodiment of the present invention, a target recognition model can be used for flowchart recognition processing. The target recognition model can be trained using a custom deep learning model for flowchart logic judgment. The training data can be the BPMN (Business Process Model and Notation) 2.0 standard library, vector template library, hand-drawn flowcharts, automatically generated flowcharts, industrial flowcharts, business flowcharts, algorithm flowcharts, etc. The flowchart formats involved include PNG (Portable Network Graphics), JPG (Joint Photographic Experts Group), PDF (Portable Document Format), and vector graphics. The target recognition model is trained by including graphic elements, connecting lines, text annotations, etc.

[0089] Understandably, existing flowcharts from various scenarios or types can be used for data training to improve the applicability of the target recognition model and achieve reliable recognition of flowcharts in multiple scenarios.

[0090] This invention extracts features from the flowchart to be identified, obtaining the first features of each node element and the second features of each non-node element; determines the spatial features of each non-node element based on the geometric features of each node element; and identifies the first features, second visual features, and spatial features to obtain the flowchart identification result. By acquiring geometric, visual, and semantic information of node elements or non-node elements in the flowchart, the spatial information between node elements in the flowchart is further determined, realizing the automatic identification of structural logic in the flowchart and solving the problem of unreliable identification of structure in the flowchart.

[0091] Figure 5 This is a flowchart of a flowchart recognition method provided by an embodiment of the present invention. Based on the above embodiments, the embodiments of the present invention supplement the specific method of feature extraction from the flowchart to be recognized. It should be noted that for parts not described in detail in the embodiments of the present invention, please refer to the relevant descriptions in other embodiments, such as... Figure 5 As shown, the method includes:

[0092] S210. Based on the contour detection algorithm, each node element and each non-node element in the flowchart to be identified are detected respectively, and the vertex coordinates and first contour features of each node element and the second contour features of each non-node element are obtained.

[0093] The first contour feature is the shape of each node element in the flowchart, which can be a rhombus, rectangle, or ellipse; the second contour feature is the arrow type of each non-node element in the flowchart.

[0094] Specifically, based on contour detection, the shapes of each node element and each non-node element are identified; and the vertex coordinates of the shape bounding box are extracted based on the shape of each node element.

[0095] In an optional embodiment of the present invention, the contour detection algorithm can be a computer vision method; the second contour feature of each non-node element is analyzed by creating a Region of Interest (ROI) at the endpoints of the line segments of each non-node element; specifically: an ROI region is created for the two endpoints of each line segment, the maximum contour within the ROI is extracted, and the convex hull defect is calculated; a typical wedge angle is obtained based on the angle distribution between the contour point and the endpoint; the direction of the arrow of each non-node element can be determined based on this angle.

[0096] Understandably, by performing contour detection on each node element and each non-node element, their corresponding geometric features can be obtained, so as to accurately identify the flow direction of the flowchart in the future.

[0097] S220. The topological relationship between node elements is obtained based on the vertex coordinates of each node element and the second contour features of each non-node element.

[0098] Topological relationships are used to characterize the connection relationships between node elements.

[0099] Specifically, the connection relationship between node elements is determined based on the vertex coordinates of each node element and the arrow direction in the second contour feature of each non-node element, and the flow direction between node elements is determined based on the arrow direction in the second contour feature of each non-node element.

[0100] Optionally, the topological relationships between node elements are obtained based on the vertex coordinates of each node element and the second contour features of each non-node element, including:

[0101] Based on the second contour features of each non-node element, feature analysis is performed on each non-node element to obtain the arrow direction of each non-node element.

[0102] The topological relationships between node elements are determined based on the vertex coordinates of each node element and the arrow directions of each non-node element.

[0103] Specifically, based on the second contour features of each non-node element, feature analysis is performed on the arrow segments of each non-node element to obtain the arrow direction of each non-node element. The topological relationship between node elements is determined based on the vertex coordinates of each node element and the arrow direction of each non-node element. For example, for node elements C and E, the flow relationship between node elements C and E is determined to be from node element C to node element E based on the vertex coordinates of the two node elements and the arrow direction of non-node element CE.

[0104] It is understandable that by capturing the geometric features of node elements and non-node elements in the flowchart, the topological relationships between node elements can be obtained, thereby enabling reliable identification of the sequential structure in the flowchart.

[0105] S230. Based on the region segmentation algorithm, visual features are extracted from each node element and each non-node element in the flowchart to be identified, to obtain the first visual feature and the second visual feature.

[0106] The first visual feature is the fill color, wireframe color, and line style of the node elements; the second visual feature is the line color and line style of the non-node elements.

[0107] Specifically, through region segmentation algorithms, visual features are extracted from each node element and each non-node element in the flowchart to be identified, resulting in first visual features and second visual features. For example, node element C has a yellow fill color, a black outline color, and a solid line type; non-node element CE has a black line color and a solid line type.

[0108] It is understandable that extracting the visual features of node elements and non-node elements in the flowchart ensures that the flowchart is visualized after recognition, thereby improving the user experience for relevant technical personnel.

[0109] S240. Extract text from the flowchart to be recognized to obtain the text information of each node element.

[0110] The extraction method can be OCR (Optical Character Recognition, text extraction), and the text information of the node element is the text within the bounding box corresponding to the node element.

[0111] Specifically, the flowchart to be recognized is subjected to OCR recognition to obtain the text information of each node element.

[0112] S250. Based on a deep language representation model, text information is processed to obtain the label information of each node element.

[0113] The label information consists of labels corresponding to the text information, which can be start / end, process, input / output, or error.

[0114] Specifically, the extracted text information of each node element is processed using a deep language representation model to obtain the label information of each node element. For example, such as... Figure 2 The node element C shown has the extracted text information "Register?". According to the deep language representation model, its corresponding label information is the decision label.

[0115] S260. Use text information and label information as semantic features of each node element.

[0116] Specifically, the text information and tag information of each node element are used as the semantic features of each node element; for example, such as Figure 2 The node element C shown has the extracted text information "Register?" and its corresponding label information is the decision label. Therefore, its semantic feature is "[Register? Decision label]".

[0117] S270. Determine the spatial characteristics of each non-node element based on the geometric characteristics of each node element.

[0118] S280. Identify the first feature, the second visual feature, and the spatial feature to obtain the recognition result of the flowchart.

[0119] This invention uses computer vision and OCR technology to extract features from node elements and non-node elements. Feature extraction is performed in three dimensions: geometric features, visual features, and semantic features. The flowchart is processed using a spatial multi-omics method to ensure that the sequence, selection, and loop structures in the flowchart can be reliably identified and the meaning of the flowchart can be accurately and comprehensively understood.

[0120] Figure 6 This invention provides a flowchart recognition device, applicable to situations requiring flowchart structure recognition, particularly in rail transit auxiliary decision-making. The flowchart recognition device can be implemented in hardware and / or software and can be configured on a server. Figure 6 As shown, the flowchart recognition device 300 includes a feature extraction module 310, a spatial feature determination module 320, and a recognition module 330;

[0121] The feature extraction module 310 is used to extract features from the flowchart to be recognized, and obtain the first features of each node element and the second features of each non-node element in the flowchart; wherein, the first features include geometric features, first visual features and semantic features; the second features include second visual features and second contour features; the geometric features include vertex coordinates, first contour features and topological relationships between node elements;

[0122] The spatial feature determination module 320 is used to determine the spatial features of each non-node element based on the geometric features of each node element.

[0123] The recognition module 330 is used to recognize the first feature, the second visual feature and the spatial feature to obtain the recognition result of the flowchart.

[0124] This invention extracts features from the flowchart to be identified, obtaining the first features of each node element and the second features of each non-node element; determines the spatial features of each non-node element based on the geometric features of each node element; and identifies the first features, second visual features, and spatial features to obtain the flowchart identification result. By acquiring geometric, visual, and semantic information of node elements or non-node elements in the flowchart, the spatial information between node elements in the flowchart is further determined, realizing the automatic identification of structural logic in the flowchart and solving the problem of unreliable identification of structure in the flowchart.

[0125] Optionally, the feature extraction module 310 includes a contour detection unit, a topological relationship acquisition unit, a visual feature extraction unit, a text extraction unit, a label information acquisition unit, and a semantic feature acquisition unit.

[0126] The contour detection unit is used to detect each node element and each non-node element in the flowchart to be identified based on the contour detection algorithm, and to obtain the vertex coordinates and first contour features of each node element and the second contour features of each non-node element.

[0127] The topology relationship acquisition unit is used to obtain the topology relationship between node elements based on the vertex coordinates of each node element and the second contour features of each non-node element;

[0128] The visual feature extraction unit is used to extract visual features from each node element and each non-node element in the flowchart to be identified based on the region segmentation algorithm, so as to obtain the first visual feature and the second visual feature.

[0129] The text extraction unit is used to extract text from the flowchart to be recognized, and obtain the text information of each node element;

[0130] The tag information acquisition unit is used to process text information based on a deep language representation model to obtain the tag information of each node element;

[0131] The semantic feature acquisition unit is used to use text information and label information as semantic features of each node element.

[0132] Optionally, the topology relationship acquisition unit is specifically used to perform feature analysis on each non-node element based on the second contour features of each non-node element to obtain the arrow direction of each non-node element; and to determine the topology relationship between node elements based on the vertex coordinates of each node element and the arrow direction of each non-node element.

[0133] Optionally, the spatial feature determination module 320 is specifically used to determine the center point coordinates of each node element based on the vertex coordinates of each node element; calculate the relative positional relationship between node elements based on the center point coordinates of each node element and the topological relationship between each node element; calculate the arrow angle of each non-node element based on the relative positional relationship between node elements; and use the center point coordinates of each node element, the relative positional relationship between node elements, and the arrow angle of each non-node element as the spatial features of each non-node element.

[0134] The recognition module 330 is specifically used to vectorize the first feature of the node element to obtain the first feature vector of the node element; to vectorize the spatial features and second features of the non-node element to obtain the second feature vector of the non-node element; and to perform logical judgment on the first feature vector of the node element and the second feature vector of the non-node element to obtain the recognition result of the flowchart.

[0135] The recognition module 330 is also specifically used to perform cyclic detection on the second feature vector of non-node elements to obtain cyclic detection results; to take the node elements whose label information is the decision label as target node elements; to perform probability judgment on the first feature vector corresponding to the target node elements to obtain the probability distribution result of the target node elements; to perform flow recognition on the first feature vector of node elements based on the second feature vector of non-node elements to obtain the state information of the flowchart; and to take the cyclic detection result, probability distribution result and state information as the recognition result of the flowchart.

[0136] The flowchart recognition device provided in the embodiments of the present invention can execute the flowchart recognition method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0137] According to embodiments of the present invention, the present invention also provides an electronic device, a readable storage medium, and a computer program product.

[0138] Figure 7A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0139] like Figure 7 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0140] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0141] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as flowchart recognition methods.

[0142] In some embodiments, the flowchart recognition method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the flowchart recognition method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the flowchart recognition method by any other suitable means (e.g., by means of firmware).

[0143] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0144] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0145] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0146] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0147] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0148] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0149] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0150] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A flowchart recognition method, characterized in that, include: Feature extraction is performed on the flowchart to be identified to obtain the first feature of each node element and the second feature of each non-node element in the flowchart; wherein, the first feature includes geometric features, first visual features and semantic features; the second feature includes second visual features and second contour features; the geometric features include vertex coordinates, first contour features and topological relationships between node elements; The spatial characteristics of each non-node element are determined based on the geometric characteristics of each node element. The first feature, the second visual feature, and the spatial feature are identified to obtain the recognition result of the flowchart.

2. The method according to claim 1, characterized in that, The process of extracting features from the flowchart to be identified, obtaining the first features of each node element and the second features of each non-node element in the flowchart, includes: Based on the contour detection algorithm, each node element and each non-node element in the flowchart to be identified are detected respectively to obtain the vertex coordinates and first contour features of each node element and the second contour features of each non-node element. The topological relationships between node elements are obtained based on the vertex coordinates of each node element and the second contour features of each non-node element. Based on the region segmentation algorithm, visual features are extracted from each node element and each non-node element in the flowchart to be identified, to obtain the first visual feature and the second visual feature. Text extraction is performed on the flowchart to be identified to obtain the text information of each node element; The text information is processed based on a deep language representation model to obtain the label information of each node element; The text information and the tag information are used as semantic features of each node element.

3. The method according to claim 2, characterized in that, The topological relationships between node elements are obtained based on the vertex coordinates of each node element and the second contour features of each non-node element, including: Based on the second contour features of each non-node element, feature analysis is performed on each non-node element to obtain the arrow direction of each non-node element; The topological relationships between the node elements are determined based on the vertex coordinates of each node element and the arrow directions of each non-node element.

4. The method according to claim 1, characterized in that, The step of determining the spatial features of each non-node element based on the geometric features of each node element includes: The center point coordinates of each node element are determined based on the vertex coordinates of each node element. The relative positional relationships between node elements are calculated based on the center point coordinates of each node element and the topological relationships between the node elements. The arrow angles of each non-node element are calculated based on the relative positional relationships between the node elements. The center point coordinates of each node element, the relative positional relationship between the node elements, and the arrow angle of each non-node element are used as the spatial features of each non-node element.

5. The method according to claim 1, characterized in that, The process of identifying the first feature, the second visual feature, and the spatial feature to obtain the recognition result of the flowchart includes: The first feature of the node element is vectorized to obtain the first feature vector of the node element; Based on the spatial features and second features of the non-node elements, vectorization processing is performed to obtain the second feature vector of the non-node elements. Logical judgment is performed on the first feature vector of the node element and the second feature vector of the non-node element to obtain the recognition result of the flowchart.

6. The method according to claim 5, characterized in that, The step of performing logical judgment on the first feature vector of the node element and the second feature vector of the non-node element to obtain the recognition result of the flowchart includes: The second feature vector of the non-node element is subjected to cyclic detection to obtain the cyclic detection result; The node elements whose label information is the decision label are taken as the target node elements; Probability judgment is performed on the first feature vector corresponding to the target node element to obtain the probability distribution result of the target node element; Based on the second feature vector of the non-node element, the first feature vector of the node element is used for process identification to obtain the state information of the flowchart. The loop detection results, probability distribution results, and state information are used as the recognition results of the flowchart.

7. A flowchart recognition device, characterized in that, include: The feature extraction module is used to extract features from the flowchart to be identified, and obtain the first feature of each node element and the second feature of each non-node element in the flowchart; wherein, the first feature includes geometric features, first visual features and semantic features; the second feature includes second visual features and second contour features; the geometric features include vertex coordinates, first contour features and topological relationships between node elements; The spatial feature determination module is used to determine the spatial features of each non-node element based on the geometric features of each node element. The recognition module is used to recognize the first feature, the second visual feature, and the spatial feature to obtain the recognition result of the flowchart.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the flowchart recognition method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the flowchart recognition method according to any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the flowchart recognition method according to any one of claims 1-6.

Images