Flow chart image analysis and structured reconstruction method and device, and storage medium

Abstract

The invention discloses a flow chart image analysis and structured reconstruction method and device and a storage medium, and relates to the technical field of image processing, and the method comprises the steps: carrying out the element background separation of an input image through a pre-trained convolutional neural network, and obtaining a flow chart image, the elements including text elements and graphic elements; extracting rectangular frame information, arrow information, text information and position coordinates thereof in the preprocessed flow chart image; determining the node type of each node in the flow chart image based on the rectangular frame information and the text information, and determining the connection relationship of each node based on the arrow information in combination with the position coordinates; analyzing the node type and the connection relationship of each node, modeling the logic relationship of each node, and generating a control flow diagram of the flow diagram image; and converting the control flow diagram into structured data, and rendering to generate an interactively editable flow diagram visual interface. According to the method and the device, the technical effect of reconstructing the structured flow chart by identifying the control logic structure of the flow chart is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a flowchart image analysis and structured reconstruction method and device and storage medium. BACKGROUND

[0002] At present, the flowchart analysis method can only identify the basic elements and text content in the image when analyzing the flowchart, and cannot accurately infer the complex logical relationship between nodes. On this basis, the converted flowchart cannot be directly edited based on the identified results, which limits the subsequent modification and reconstruction capabilities.

[0003] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0004] The main purpose of the present application is to provide a flowchart image analysis and structured reconstruction method, device and storage medium, aiming to solve the technical problem of how to accurately identify the control logic structure of the flowchart and reconstruct the structured flowchart.

[0005] To achieve the above purpose, the present application provides a flowchart image analysis and structured reconstruction method, which comprises: Separating the elements and background of the input image by a pre-trained convolutional neural network to obtain a flowchart image, wherein the elements include text elements and graphic elements; Extracting the rectangular frame information, arrow information, text information and their position coordinates in the preprocessed flowchart image; Determining the node type of each node in the flowchart image based on the rectangular frame information and the text information, and determining the connection relationship of each node based on the arrow information combined with the position coordinates; Analyzing the node type and the connection relationship of each node, modeling the logical relationship of each node, and generating a control flow graph of the flowchart image; Converting the control flow graph into structured data and rendering an interactive editable flowchart visualization interface.

[0006] In an embodiment, the step of separating the elements and background of the input image by a pre-trained convolutional neural network to obtain a flowchart image, wherein the elements include text elements and graphic elements, comprises: Performing pixel-level semantic segmentation on the input image by a pre-trained convolutional neural network to separate the text element region and the background region of the input image; extracting a feature map of the input image by the convolutional neural network and applying an edge detection operator to generate an edge feature map to extract a graphic element region of the input image; combining the text element region and the graphic element region of the input image to obtain the flowchart image.

[0007] In an embodiment, the step of extracting the rectangular frame information, the arrow information, the text information and the position coordinates thereof in the preprocessed flowchart image comprises: filtering out a contour conforming to a rectangular frame shape feature as the rectangular frame information based on the graphic element of the flowchart image; determining starting point position information, ending point position information and direction information of an arrow as the arrow information according to a preset arrow shape template and matching the graphic element of the flowchart image; performing character recognition on the text element of the flowchart image to obtain the content of each text element and record the coordinates corresponding to the center point of the text element as the text information and the position coordinates thereof.

[0008] In an embodiment, the step of determining the node type of each node in the flowchart image based on the rectangular frame information and the text information and determining the connection relationship of each node based on the arrow information and the position coordinates comprises: preliminarily determining the node type of each node corresponding to each rectangular frame based on the rectangular frame information of each rectangular frame in the flowchart image and combining the rectangular frame shape specification corresponding to the node type, wherein the node type comprises a start node, an end node, a condition judgment node and a processing node; extracting a keyword in the text information in each rectangular frame and matching the keyword with a mapping keyword in a preset rule library to further determine the node type corresponding to each text information; determining the connection relationship of each node by matching the starting point position information of the arrow information with the position coordinates as the starting node of the connection relationship and matching the ending point position information of the arrow information with the position coordinates as the ending node of the connection relationship.

[0009] In an embodiment, the step of analyzing the node type and the connection relationship of each node, modeling the logical relationship of each node and generating a control flow graph of the flowchart image comprises: mapping each node and the connection relationship into a node object and an edge object in the control flow graph to construct the basic structure of the control flow graph; According to the node type and the connection relationship of each node, a logical relationship of each node is inferred, and a state transition model is established to determine a transition direction of a control flow, the logical relationship including sequential execution, conditional judgment, parallel nodes and loop nodes; In combination with the basic structure and the state transition model of the control flow graph, the control flow graph is formed to describe the overall control logic of the flowchart image.

[0010] In an embodiment, the step of converting the control flow graph into structured data and rendering a flowchart visualization interface that is interactively editable includes: Each node information in the control flow graph and the corresponding logical relationship are converted into structured data; Based on the structured data, the flowchart visualization interface that is interactively editable is constructed.

[0011] In an embodiment, the step of converting each node information in the control flow graph and the corresponding logical relationship into structured data includes: Each node information in the control flow graph is encoded into an HTML container element, including node type, text content and position coordinates; The logical relationship in the control flow graph is encoded into a JavaScript object array, including source node reference, target node reference and connection type identification; The HTML container element and the JavaScript object array are integrated and packaged to generate the structured data.

[0012] In an embodiment, after the step of converting the control flow graph into structured data and rendering a flowchart visualization interface that is interactively editable through a graph modeling tool, the step includes: A version management module is added to the flowchart visualization interface, in response to a version saving instruction, a version snapshot of the flowchart visualization interface is generated and stored as structured data; In response to a version comparison instruction, based on two target versions selected by a user, difference analysis is performed and difference points are displayed through visual markers; In response to a version rollback instruction, the flowchart visualization interface is restored to a specified version state data.

[0013] In addition, to achieve the above-mentioned purpose, the present application also proposes a flowchart image analysis and structured reconstruction device, the device includes a memory, a processor and a computer program stored on the memory and executable on the processor, the computer program is configured to implement the steps of the flowchart image analysis and structured reconstruction method as described above.

[0014] In addition, to achieve the above object, the application further provides a storage medium, which is a computer readable storage medium, and a computer program is stored on the storage medium, and the computer program is executed by a processor to implement the steps of the flowchart image analysis and structured reconstruction method.

[0015] The application provides a flowchart image analysis and structured reconstruction method. The application separates elements from a background of an input image through a pre-trained convolutional neural network to obtain a flowchart image, the elements including text elements and graphic elements. Rectangular box information, arrow information, text information and position coordinates thereof in the preprocessed flowchart image are extracted. Node types of nodes in the flowchart image are determined based on the rectangular box information and the text information, and connection relationships of the nodes are determined based on the arrow information and the position coordinates. The node types and the connection relationships of the nodes are analyzed, a logical relationship of the nodes is modeled, and a control flow graph of the flowchart image is generated. The control flow graph is converted into structured data, and an interactive and editable flowchart visualization interface is rendered and generated. The application first performs adaptive preprocessing on the flowchart image through a convolutional neural network to realize background separation and contour extraction, thereby improving the clarity and contrast of the image. The text content in the preprocessed flowchart image is recognized, and the position coordinates of the text are recorded, so as to ensure the complete acquisition of the text information and the accurate positioning of the position coordinates. The node types in the flowchart image are accurately recognized by deeply analyzing the semantic content of the text information and combining the position coordinate information, thereby laying a foundation for constructing the control flow graph of the flowchart. The nodes and the connection relationships are converted into a graph data structure, and complex flowchart information is converted into a structured representation form. The interactive and editable flowchart visualization interface is rendered and generated, thereby improving the usability of the flowchart. The application realizes the technical effect of accurately recognizing the control logic structure of the flowchart and reconstructing the structured flowchart. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0018] Figure 1 A flowchart diagram is provided for the flowchart image analysis and structured reconstruction method embodiment one of the application; Figure 2A flowchart diagram of the second embodiment of the method for analyzing and reconstructing a flowchart image of the present application is provided; Figure 3 A flowchart diagram of the third embodiment of the method for analyzing and reconstructing a flowchart image of the present application is provided; Figure 4 A flowchart diagram of the fourth embodiment of the method for analyzing and reconstructing a flowchart image of the present application is provided; Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the method for parsing and structured reconstruction of flowchart images in the embodiment of the present application.

[0019] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0020] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0021] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0022] Currently, when parsing flowcharts, methods for parsing flowcharts can only recognize basic elements and text content in the image, and cannot accurately infer the complex logical relationships between nodes. On this basis, the flowchart is converted based on the recognition results. The converted flowchart cannot be directly edited, which limits its subsequent modification and reconstruction capabilities.

[0023] This application uses a convolutional neural network to adaptively preprocess the flowchart image to achieve background separation and contour extraction, thereby improving the clarity and contrast of the image. By identifying the text content in the preprocessed flowchart image and recording the position coordinates of the text, the complete acquisition of text information and the precise positioning of the position coordinates are ensured. By deeply analyzing the semantic content of the text information and combining it with the position coordinate information, the node type in the flowchart image is accurately identified, laying the foundation for constructing the control flow graph of the flowchart. By converting the nodes and connection relationships into a graph data structure, the complex flowchart information is converted into a structured representation. By rendering and generating an interactively editable flowchart visualization interface, the usability of the flowchart is improved.

[0024] It should be noted that the execution subject of this embodiment can be a flowchart image parsing and structured reconstruction system, or a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or a control device for a flowchart image parsing and structured reconstruction system capable of implementing the above functions, etc. This embodiment does not specifically limit this. The following uses the flowchart image parsing and structured reconstruction system as an example to illustrate this embodiment and the following embodiments.

[0025] Example 1 Based on this, this application proposes a method for analyzing and structurally reconstructing a flowchart image of the first embodiment, please refer to Figure 1 , the flowchart image analysis and structured reconstruction method includes: Step S10: performing element-background separation on the input image using a pre-trained convolutional neural network to obtain a flowchart image, where the elements include text elements and graphic elements.

[0026] A pre-trained convolutional neural network is used to separate the elements in the input image from the background, thereby obtaining a flowchart image containing only flowchart-related elements. This removes background information in the image that is irrelevant to the flowchart body, reduces interference, and facilitates subsequent positioning, classification, and relationship analysis of flowchart elements.

[0027] In this embodiment, a convolutional neural network is a deep learning model that excels at processing data with a grid structure. It automatically extracts features from input data through components such as convolutional layers, pooling layers, and fully connected layers. The convolutional layer uses convolution kernels to slide across the image to extract local features; the pooling layer is used to reduce the dimension of the feature map, reduce the amount of computation, and prevent overfitting; the fully connected layer integrates the extracted features for tasks such as classification or regression. The input image is the original image containing the flowchart and the background. Background separation is to distinguish the flowchart elements and background areas from the input flowchart image, and remove or suppress the background part, retaining only the foreground information to reduce the background's interference with subsequent processing. The flowchart image is the image obtained after element-background separation, and only contains the text elements and graphic elements in the flowchart.

[0028] As an optional implementation, the input image is resized and normalized, and the trained convolutional neural network model is applied to the input image to classify each pixel in the image and determine whether each pixel belongs to the background or the foreground. According to the classification result of the pixel, a foreground mask image is generated, in which the foreground region is marked as 1 and the background region is marked as 0. The convolutional layer of the convolutional neural network model extracts a feature map containing semantic information and edge features of the image. Based on the feature map, an edge detection algorithm is applied to extract edge information to further highlight the outline of the flowchart. The foreground mask obtained by separating the background is fused with the edge image obtained by extracting the outline to obtain the final preprocessing result. The flowchart image obtained after preprocessing is output.

[0029] Optionally, a convolutional neural network architecture suitable for image segmentation tasks is selected, and a labeled flowchart image dataset is prepared, including labels for different regions such as background, text, lines, and nodes. The labeled dataset is used to train the convolutional neural network model, and the parameters of the model are optimized to minimize the difference between the predicted results and the true labels.

[0030] Optionally, a U-Net model is used. U-Net is a convolutional neural network commonly used for image segmentation tasks, which has an encoder-decoder structure and can effectively capture local and global features of an image, making it suitable for element-background separation tasks.

[0031] For example, an encoder-decoder structure convolutional neural network is used for element-background separation. The encoder consists of multiple convolutional layers and pooling layers alternately. The input layer of the convolutional neural network receives the original input image, which has a size of HxWxC, , H is the height of the image, W is the width of the image, , C represents the number of channels, and is passed to the convolutional layer of the encoder. The input image from the input layer is received by the convolutional layer, which uses multiple convolutional kernels to perform convolution operations. Each convolutional kernel slides over the image to calculate the dot product of the local region and the convolutional kernel, resulting in an output value. Multiple convolutional kernels generate multiple output channel feature maps. After convolution, the output feature map has a size of HxWxC, , , C is the number of output channels, i.e., the number of convolutional kernels. The pooling layer of the convolutional neural network receives the feature map from the convolutional layer, which has a size of HxWxC. The feature map is downsampled using maximum pooling or average pooling to reduce the size of the feature map while preserving the main features, and the output feature map is output. The size of the feature map is , ×C. Max pooling takes the maximum value in each pooling window as the output, and average pooling takes the average value of all values in the window as the output. The bottleneck layer receives the feature map from the last layer of the encoder, further extracts features and adjusts the number of channels, and outputs the feature map to the decoder. The bottleneck layer is composed of several convolutional layers. The decoder is composed of multiple up-sampling layers, jump connections and convolutional layers. The up-sampling layer receives the feature map from the bottleneck layer, the size of which is h x w x C, and performs up-sampling operation to increase the size of the feature map and output it to the jump connection. The up-sampling method includes transposed convolution and interpolation. Transposed convolution realizes up-sampling by learning the convolution kernel, and interpolation is estimated according to the values of the surrounding pixels. The up-sampling factor is 2, and the output feature map size is 2h x 2w x C. The jump connection receives the output feature map of the up-sampling layer and the feature map of the corresponding layer of the encoder, and splices the feature map of the corresponding layer of the encoder and the output feature map of the up-sampling layer in the channel dimension, so as to combine the shallow features extracted by the encoder with the high-level features of the decoder, improve the details and accuracy of the segmentation, and output the spliced feature map, the size of which is 2h x 2w x + ), is the number of channels of the output feature map of the up-sampling layer, is the number of channels of the feature map of the corresponding layer of the encoder. The convolutional layer of the decoder receives the feature map from the jump connection, uses the convolution kernel to perform convolution operation, further extracts features and adjusts the number of channels, and outputs the feature map, the size of which is 2h x 2w x , is the number of convolution kernels, that is, the number of output channels. The output layer of the convolutional neural network receives the feature map of the last layer of the decoder, and uses a 1 x 1 convolution kernel to adjust the number of channels of the feature map to the number of categories, which is 2, that is, elements and background. Convert the output to the probability of each pixel belonging to each category, and output the probability map of the flowchart image, where the two values at each pixel position represent the probability of the pixel belonging to the element and the background, respectively.

[0032] Optionally, step S10 comprises: Step S11, performing pixel-level semantic segmentation on the input image by the pre-trained convolutional neural network to separate the text element region and the background region of the input image.

[0033] By performing pixel-level semantic segmentation on the flowchart image by the convolutional neural network, the text information in the flowchart is accurately located, the text element region and the background region are separated, and the background interference is removed.

[0034] It should be noted that the pixel-level semantic segmentation is an image processing technology, each pixel in the flowchart image is assigned a class label, so that pixels with the same semantics are classified into the same category, that is, the pixels of each text element and the background pixels are marked respectively, so as to separate the text element region and the background region. The background region is a region in the flowchart image which does not contain any element and does not contain content directly related to the core information of the flowchart, including paper color, irrelevant patterns or blank areas, etc.

[0035] Illustratively, the flowchart image to be processed is input into the trained convolutional neural network model, the model performs forward propagation on the input image, outputs the class label of each pixel, and generates a segmentation mask. According to the segmentation mask, a foreground mask image is generated, in which the text element region is marked as 1 and the background region is marked as 0.

[0036] Step S12, extracting a feature map of the input image through the convolutional neural network, and generating an edge feature map by applying an edge detection operator to extract the graphic element region of the input image.

[0037] It should be noted that the feature map is an image feature representation extracted by the convolutional neural network in the convolutional layer, which contains feature information of the flowchart image at different levels, and each pixel value of the feature map represents the intensity of the response of a certain feature of the input image at the corresponding position. The edge detection operator is a mathematical operator for detecting edges in an image, including Sobel operator, Prewitt operator, Canny operator, etc., which detects edges by calculating the gradient of the pixel gray value in the image.

[0038] Illustratively, the image is input into the trained convolutional neural network model to obtain the feature map output by the convolutional layer. The feature map contains high-level semantic information and low-level texture information of the image. According to the task requirement, an edge detection operator is selected, the extracted feature map is taken as input, and the selected edge detection operator is applied for calculation to obtain an edge feature map. The high-intensity area in the edge feature map represents the detected boundary contour information. According to the edge feature map, the graphic element region of the input image is extracted, and the edge pixels in the edge feature map are identified as the boundary of the graphic element through threshold segmentation, so as to determine the region of the graphic element.

[0039] Optionally, when a relatively clear edge needs to be detected, the Canny operator is selected; if the calculation efficiency is required to be higher, the Sobel operator or the Prewitt operator is selected.

[0040] Step S13, combining the text element region and the graphic element region of the input image to obtain the flowchart image.

[0041] Exemplarily, the obtained text element region and the graphic element region are fused, the text element region and the graphic element region are respectively represented as binary images, pixels of the two regions are combined through logic, and a region containing text elements and graphic elements is obtained. According to the merged region, corresponding pixels are extracted from the original input image, and a flowchart image containing only flowchart text and graphic elements is generated.

[0042] In step S20, rectangular frame information, arrow information, text information and position coordinates thereof in the preprocessed flowchart image are extracted.

[0043] The key element information including rectangular frames, arrows and texts and the position coordinates thereof is extracted from the preprocessed flowchart image, and the text content in the flowchart is converted into editable and processable text data, and the specific position of each text in the flowchart is determined, which is helpful for subsequent semantic analysis, information integration, flowchart logic understanding and other operations on the flowchart text.

[0044] In the embodiment, the rectangular frame information is the feature information contained in the rectangular frame in the flowchart, including the size and type of the rectangular frame, and the rectangular frame represents different operation nodes or steps in the flowchart. The arrow information is the feature information contained in the arrow in the flowchart, including the start point coordinates, end point coordinates and direction information of the arrow, and the arrow represents the connection relationship and execution order between elements in the flowchart. The text information is the text content in the rectangular frame or related to the flowchart, including the description of operation steps, conditional judgment statements and the like. The position coordinates are the specific position of the text in the flowchart image, that is, the center coordinates or boundary coordinates of the text region.

[0045] Optionally, the type of the rectangular frame includes a rectangle, a diamond, a rounded rectangle and the like.

[0046] As an optional implementation, a target detection algorithm is used to detect the rectangular frame in the preprocessed flowchart image, and the size and type of each rectangular frame are recorded. A target detection algorithm is used to identify the arrow based on the shape characteristics of the arrow, to determine the start point coordinates and end point coordinates of the arrow, and to determine the direction information of the arrow by calculating the vector between the start point and end point coordinates. A text detection algorithm is used to locate the text region in the image, to output the position coordinates of the text region, and to determine the position range of the text in the image. The text recognition is performed on the text region to extract the corresponding text content, which is stored in association with the position coordinates of the text region.

[0047] Optionally, all contours in the flowchart image are obtained through a contour detection algorithm, the contours are screened according to the area, perimeter, aspect ratio and other characteristics of the contours, and the contours meeting the shape characteristics are screened out. The rectangular frames are classified according to the characteristics of the rectangular frames, and the types of the rectangular frames are determined.

[0048] Optionally, the pre-processed image is subjected to text recognition by an optical character recognition technique, text content in the image is extracted, and position coordinates of each recognized text in the image are recorded, the position coordinates being based on an actual pixel coordinate system of the flowchart image. The optical character recognition technique (OCR) is a technique of converting characters in an image into computer editable text, recognizing characters in the image by analyzing and processing the image, and converting the characters into corresponding text codes such as ASCII codes or Unicode codes.

[0049] Optionally, step S20 comprises: Step S21, based on the graphical elements of the flowchart image, screening out a contour conforming to a rectangular frame shape feature as the rectangular frame information.

[0050] It should be noted that the rectangular frame does not only refer to a frame with a rectangular shape, but also includes common frames such as rectangular frames, oval frames, diamond frames, and rounded rectangular frames. Among them, the rectangular frame has four right angles and equal and parallel opposite sides; the oval frame is a closed curve composed of two concentric circles; the diamond frame has four equal side lengths and equal opposite angles; and the rounded rectangular frame has four corners with arc transitions.

[0051] Illustratively, an edge detection algorithm is used to detect contours in the image, the matching degrees of the detected contours and various frame shapes are calculated, and the contours conforming to the frame shape features are screened out according to the matching degrees. The position coordinates, sizes, and other information of the screened frame contours are extracted as the rectangular frame information.

[0052] Step S22, according to a preset arrow shape template, matching the graphical elements of the flowchart image, determining the starting point position information, the ending point position information, and the direction information of the arrow as the arrow information.

[0053] It should be noted that the preset arrow shape template is a predefined arrow shape model, which contains features such as the shape of the head of the arrow, the length and width ratio of the arrow shaft, and is used to match the graphical elements in the image to identify the arrow and determine its direction.

[0054] Illustratively, the graphical elements of the flowchart image are traversed, the preset arrow shape template is matched with the graphical elements in the image, and if the matching result exceeds a matching threshold, the graphical element is considered to be an arrow. According to the shape of the template and the matching position, the starting point and ending point coordinates of the arrow are determined. According to the positional relationship of the starting point and ending point, the direction information of the arrow is determined.

[0055] Step S23, character recognition is performed on the text elements of the flowchart image to obtain the content of each text element, and the coordinates corresponding to the center points of the text elements are recorded as the text information and the position coordinates thereof.

[0056] By way of example, the text element regions in the image are determined by a text detection algorithm, character recognition is performed on the text element regions to obtain the text content, the center point coordinates of the text element regions are calculated, the text content and the corresponding center point coordinates are recorded, and a set of text information and position coordinates is formed.

[0057] Step S30, based on the rectangular frame information and the text information, the node types of each node in the flowchart image are determined, and the connection relationship of each node is determined based on the arrow information in combination with the position coordinates.

[0058] By analyzing the extracted rectangular frames and text information, the types of each node in the flowchart image are determined to clearly define the specific operations or logical functions represented by each node in the flow. By analyzing the arrow information to determine the connection relationship between nodes, the logical flow direction between nodes is constructed, thereby completely restoring the execution logic of the flowchart.

[0059] In this embodiment, the text information is the textual content extracted from the flowchart image, which describes the functions, operations or conditions of each node in the flowchart. The position coordinates represent the specific position of the text in the flowchart image, which is represented in the form of pixel coordinates, and are used to determine the spatial distribution of the text in the image, which helps to understand the relative position relationship of the nodes in the flowchart. The node types include start nodes, end nodes, judgment nodes, etc., and different types of nodes represent different functions or operations. The start node represents the starting point of the flow; the end node represents the termination point of the flow; the judgment node is used for branch judgment according to the condition to determine the flow direction. The connection relationship of the nodes describes the physical connection between each node in the flowchart.

[0060] Optionally, the extracted text information is cleaned and normalized, and the nodes are preliminarily positioned and grouped according to the position coordinate information in combination with the layout of the flowchart image.

[0061] As an optional implementation, the text information of the text region corresponding to the rectangular frame region is subjected to semantic analysis, and keywords of the text information are extracted, and the corresponding node type is determined by matching the keywords and a keyword dictionary of node types. The node type is determined by comprehensively judging the type of the extracted rectangular frame and the text semantics. According to the start point coordinates and the end point coordinates of the arrow, the corresponding node is matched, and for each arrow, the node where the start point coordinates are located is found as the starting node, and the node where the end point coordinates are located is found as the target node by comparing the position coordinates corresponding to the text information of each node, so as to determine the connection relationship of the nodes.

[0062] For example, if a frame is a rounded rectangle and the text information is "start", it is determined that the node is a start node; if the frame is a diamond and the text information contains "whether", "condition" and the like, the node is a conditional judgment node.

[0063] Optionally, the node type is directly determined by a natural language processing model, and the node connection relationship is determined in combination with the position coordinates. According to the task requirements and data characteristics, a natural language processing model is selected, a text classification model is used for the node type identification task, and a relation extraction model is used for the node connection relationship identification task. The text information and the position coordinates are converted into feature vectors that can be processed by the model, the feature vectors are input into the trained text classification model for node type identification, and the type label of each node is output by the text classification model. The node pair, the corresponding text information and the position coordinate features are input into the relation extraction model, and the connection relationship type between the nodes is output by the relation extraction model.

[0064] In step S40, the node types and the connection relationships of the nodes are analyzed, the logical relationships of the nodes are modeled, and a control flow graph of the flowchart image is generated.

[0065] The identified node types and node connection relationships in the flowchart image are subjected to in-depth analysis, a logical relationship model between the nodes is constructed, and finally a control flow graph is generated, so as to clearly present the execution order and the logical relationships such as conditional branching between the nodes in the flowchart in a structured manner.

[0066] In this embodiment, the graph structure data is composed of nodes and edges, the nodes represent entities, and the edges represent the relationships between the entities. The graph neural network can learn the feature representation of the nodes and the edges in the graph, capture the complex relationships and dependencies between the nodes, and thus realize the analysis and prediction of the graph structure data. The control flow graph (CFG) is a graph structure used to represent the execution logic of a program or a flow, and the control flow graph represents each operation or step in the flow with a node and represents the execution order and conditional branching relationship between the nodes with an edge, so as to intuitively show the execution flow and logical structure of the flow.

[0067] As an optional implementation, the identified node types and node connection relationships are converted into graph structure data, including a node feature matrix and an adjacency matrix. A feature vector is constructed for each node as the node feature matrix, and an adjacency matrix is constructed according to the connection relationship to represent the connection relationship between nodes. A graph neural network model is used to predict the logical relationship type between nodes, and a graph structure of the control flow graph is generated according to the predicted logical relationship. The control flow graph is output in a structured data format.

[0068] Optionally, step S40 includes: Step S41, mapping each of the nodes and the connection relationships into a node object and an edge object in the control flow graph, and constructing a basic structure of the control flow graph.

[0069] It should be noted that the control flow graph is used to represent program or flow control logic, which is composed of nodes and edges. The nodes represent operations or steps in the program, and the edges represent the execution order of the program and the direction of the control flow. In the control flow graph, the node object represents an operation or step in the flow and contains relevant information of the operation or step, such as node type, name, description, etc. The edge object represents the connection relationship between nodes, represents the execution order of the program or flow and the direction of the control flow, and contains the information of the start node and the end node. It can also contain the weight of the edge, the condition of the edge, etc.

[0070] As an optional implementation, the nodes and connection relationships of the flowchart are obtained and stored in a selected data structure for representing the control flow graph, including an adjacency matrix and an adjacency list. The extracted node information is traversed, and a node object is created for each node. The type, name, description, and other information of the node are stored in the node object. An edge object is created for each connection relationship. The edge object contains the identifiers of the start node and the end node, as well as the direction of the arrow and other information. The created node objects and edge objects are added to the data structure of the control flow graph. For the adjacency list, the identifier of each node object is used as the key, and its adjacent edge object list is used as the value; for the adjacency matrix, each element in the matrix is a node object, and the edge object is marked in the matrix.

[0071] For example, by storing in an adjacency matrix, an n n matrix (n is the number of nodes) is created, and the elements in the matrix represent the connection relationship between node i and node j. If there is a connection between node i and node j, it is assigned a value of 1; if there is no connection, it is assigned a value of 0.

[0072] ​For example, for node i, its adjacency list stores all the nodes directly connected to i.

[0073] At step S42, the logical relationship of each node is inferred according to the node type and the connection relationship of each node, and a state transition model is established to determine the transition direction of the control flow, the logical relationship including sequential execution, conditional judgment, parallel nodes and loop nodes.

[0074] The logical relationship of each node is inferred and the state transition model is established to determine the specific transition direction of the control flow in the flowchart, which more accurately describes the execution logic of the flow, including sequential, branching, loop and other complex structures.

[0075] In this embodiment, the logical relationship describes the execution order and flow control mode between nodes, sequential execution indicating that nodes are executed in linear order; conditional judgment indicating that different execution paths are selected according to specific conditions; parallel nodes indicating that multiple nodes can be executed simultaneously; and loop nodes indicating that the flow will repeatedly execute a certain node or certain nodes when a specific condition is met. The state transition model is a mathematical model used to describe the transition between different states of a system, and each node in the flowchart is regarded as a state, and the edges represent the transition conditions and directions between states. The control flow transition direction refers to the execution order of the flow between nodes, which is determined by the connection relationship and the logical relationship between nodes.

[0076] It should be noted that the connection relationship of each node and the logical relationship of each node are not the same, the connection relationship is at a lower abstraction level and is a direct reflection of the actual line connection between nodes in the flowchart image. The logical relationship is at a higher abstraction level, which not only considers the connection between nodes, but also combines the type and function of the node to infer the specific logical relationship of the node in the flow execution process. In the flowchart, if node A is a judgment node, node B and node C are processing nodes, and node A is connected to node B and node C, the connection relationship only records the fact that node A and node B, node C are connected, and does not involve the specific meaning behind the connection; while the logical relationship will infer that according to the judgment result of node A, the flow will choose to execute node B or node C, which involves the control logic of the flow.

[0077] For example, if there is a direct connection between two nodes, and there is no condition judgment, branch or loop, it is inferred that they are sequentially executed logical relationship. For conditional judgment nodes, check the connected subsequent nodes, the conditional judgment node has multiple subsequent nodes, each corresponding to a conditional branch, by analyzing the conditional expression on the arrow, determine that under certain conditions, the control flow will be transferred to the corresponding subsequent node, and infer the conditional judgment logical relationship. If multiple nodes are connected from a common predecessor node, and there is no obvious execution order dependency, it is inferred that these nodes are parallel logical relationship. Check if there is a loop between nodes, if a node is connected to a certain node in front of it, forming a loop structure, it is inferred as a loop node logical relationship. Create a dictionary to represent the state transition model, traverse each node and edge in the control flow graph, and fill in the state transition model according to the inferred logical relationship.

[0078] Step S43, combining the basic structure of the control flow graph and the state transition model, forming the control flow graph that describes the overall control logic of the flowchart image.

[0079] As an optional implementation, add the transition condition and direction information in the state transition model to the basic structure of the control flow graph, add the transition condition, transition direction and other attributes on the corresponding edge object in the basic structure of the control flow graph according to the logical relationship determined by the state transition model, add the transition condition expression on the edge object and mark the logical type of the node object according to the detailed information in the state transition model, and form a complete control flow graph.

[0080] For example, according to the logical relationship inferred in the state transition model, further optimize the basic structure of the control flow graph, for sequential execution relationship, draw a one-way arrow edge; for conditional judgment relationship, draw multiple edges according to different conditional branches, and mark the conditions on the edges; for parallel nodes, draw special edges to represent parallel execution; for loop nodes, draw loop edges and mark the loop conditions. Generate the complete graph structure of the control flow graph.

[0081] Step S50, convert the control flow graph into structured data, and render an interactive editable flowchart visualization interface.

[0082] The logical structure of the control flow graph is converted into a visual interface that users can directly operate and edit, so as to facilitate users to analyze and modify the flowchart.

[0083] In this embodiment, the interactive editable flowchart visualization interface is a visual interface that users can interact with the flowchart through mouse, keyboard and other input devices, including adding, deleting, moving nodes, modifying the attributes of nodes and edges, adjusting the layout of the flowchart, etc.

[0084] As an optional implementation, all nodes and edges in the control flow graph are traversed, and attribute information of the nodes and edges is extracted. The attribute information of the nodes includes a node ID, a node type, a node name, a node description, and the like. The attribute information of the edges includes a start node ID of the edge, an end node ID of the edge, a type of the edge, a conditional expression of the edge, and the like. The extracted attribute information of the nodes and edges is organized in a JSON (JavaScript Object Notation) format to form control flow graph data in the JSON format. According to a preset conversion rule, each node and edge in the control flow graph data in the JSON format is converted into a corresponding code snippet. The code snippets of all the nodes and edges are combined together to form a complete code data set. The generated code data set is integrated with a front-end framework to dynamically create graphical elements corresponding to the nodes and edges, and an interactive and editable flowchart visualization interface is generated.

[0085] Optionally, a node conversion rule is predefined, and a corresponding code representation form is defined for different types of nodes. An edge conversion rule is predefined, and a corresponding line style and connection manner are defined according to the type of the edge. The edges connected in sequence can adopt a simple straight line style, and the edges connected by conditional branching can adopt a line style with an arrow and a specific mark to intuitively represent different connection relationships.

[0086] For example, the generated control flow graph is converted into structured data, node, text, and relationship data in the control flow graph are converted into HTML and JavaScript code output, a dynamic page is generated by using HTML and JavaScript, a JavaScript file of an mxGraph library is introduced, nodes and corresponding relationships of the flowchart are displayed, and user interactive editing is supported.

[0087] The embodiment provides a flowchart image parsing and structured reconstruction method. The embodiment first performs adaptive preprocessing on the flowchart image by using a convolutional neural network, realizes background separation and contour extraction, and thus improves the clarity and contrast of the image. Text content in the preprocessed flowchart image is recognized, and position coordinates of the text are recorded, so that complete acquisition of the text information and accurate positioning of the position coordinates are ensured. The semantic content of the text information is deeply analyzed, and the position coordinate information is combined to accurately identify the node type in the flowchart image, thereby laying a foundation for constructing a control flow graph of the flowchart. The node and connection relationship are converted into a graph data structure, and complex flowchart information is converted into a structured representation form. An interactive and editable flowchart visualization interface is generated by rendering, and the usability of the flowchart is improved.

[0088] Based on example one, example two of the present application proposes a flowchart image analysis and structured reconstruction method, referring to Figure 2 Step S30 includes: Step S31, based on the rectangular frame information of each rectangular frame in the flowchart image, combining the rectangular frame shape specification corresponding to the node type, initially determining the node type of the node corresponding to each rectangular frame, the node type includes start node, end node, condition judgment node and processing node.

[0089] It should be noted that the rectangular frame shape specification is the standard of the rectangular frame shape corresponding to different node types, the start node and the end node use the rounded rectangular frame; the condition judgment node uses the diamond frame; the processing node uses the rectangular frame.

[0090] Illustratively, the position and shape information of all rectangular frames in the flowchart image are obtained, the shape characteristics of the rectangular frame are matched with the shape specification, and the node type corresponding to each rectangular frame is initially determined according to the matching result.

[0091] The extracted text information is analyzed and extracted by a natural language processing model to determine its semantic meaning, providing a basis for subsequent node type recognition and connection relationship analysis.

[0092] It should be noted that the text information is the text content extracted from the flowchart image. The text semantic content is the actual meaning and concept expressed by the text, including starting the process, performing operations, making conditional judgments, and ending the process.

[0093] Illustratively, the extracted text information is cleaned to remove irrelevant characters, special symbols, and redundant spaces, etc. And the text is processed by word segmentation, splitting the sentence into individual words. The preprocessed text information is input into the natural language processing model, and the semantic feature vector of the text is output by the natural language processing model. Through further processing by classification algorithm, the semantic representation is converted into specific semantic content description.

[0094] Step S32, extracting the keywords in the text information in each rectangular frame, and matching the keywords with the mapping keywords in the preset rule library, further determining the node type corresponding to each text information.

[0095] It should be noted that the preset rule library is a database or rule set containing mapping keywords and their corresponding node types established in advance.

[0096] Optionally, according to a common text semantic pattern, rules corresponding to different node types are predefined. If the text information contains keywords such as "start", "start", etc., it is determined as a start node; if it contains keywords such as "end", "terminate", etc., it is determined as an end node; if it contains condition judgment related words such as "if", "otherwise", etc., it is determined as a condition judgment node; and other texts describing specific operations or tasks are determined as processing nodes.

[0097] For example, the preprocessed text information is extracted, including removing unnecessary spaces, punctuation marks, etc., extracting keywords from the text information, and using a string matching algorithm to match the extracted keywords with the mapping keywords in the preset rule library. According to the matching result, the node type corresponding to each text information is further determined.

[0098] In step S33, the position coordinates matched by the start point position information of the arrow information are determined as the start node of the connection relationship, and the position coordinates matched by the end point position information of the arrow information are determined as the end node of the connection relationship, and the connection relationship of each node is determined.

[0099] For example, the start point position information and the end point position information of the arrow are obtained, and according to the position coordinates of the rectangular frame, the start point and the end point of the arrow are matched to the corresponding node by calculating the distance between the start point and the end point of the arrow and the geometric center of the rectangular frame. The start node and the end node connected by each arrow are recorded to form a connection relationship list between nodes.

[0100] Optionally, a dictionary is used to store the connection relationship, where the key is the start node and the value is the end node list.

[0101] The embodiment provides a flowchart image analysis and structured reconstruction method. The embodiment first matches the rectangular frame information with the rectangular frame shape specification corresponding to the node type, which can preliminarily determine the node type corresponding to each rectangular frame. The keywords in the text information in the rectangular frame are extracted and matched with the mapping keywords in the preset rule library, which further determines the node type corresponding to each text information. Combined with text semantic analysis, the accuracy of node type judgment is effectively improved. By matching the start point and end point position information of the arrow information with the position coordinates of the node, the connection relationship between each node is accurately determined, which provides reliable connection information for constructing a complete control flow graph. By determining the connection relationship of each node, the scattered nodes can be organized into a complete flowchart structure.

[0102] Based on embodiment one, the third embodiment of the application proposes a flowchart image analysis and structured reconstruction method, which refers to Figure 3 , step S50 includes: Step A10, converting each node information and corresponding logical relationship in the control flow graph into structured data.

[0103] For example, the data of node information and logical relationship obtained from the control flow graph. The node information such as node type, text content and position coordinates is converted into JSON format, and the logical relationship is encoded into JavaScript object array. The converted node information and logical relationship are integrated into complete structured data, ensuring that the converted data meets the data format requirements.

[0104] Optionally, step A10 includes: Step A11, encoding each node information in the control flow graph into an HTML container element, including node type, text content and position coordinates.

[0105] It should be noted that the HTML container element is an HTML element for wrapping and presenting content, including 、 etc.

[0106] For example, parse the control flow graph data, extract the type identifier of the node type, text content and position coordinates of each node, create a corresponding HTML container element for each node, set the type identifier, text content and position coordinates as the properties of the HTML element respectively, integrate all the HTML elements created for the nodes into a container to form a complete HTML structure.

[0107] Step A12, encode the logical relationship in the control flow graph as a JavaScript object array, including source node reference, target node reference and connection type identifier.

[0108] The logical relationship in the control flow graph is encoded as a JavaScript object array to dynamically process and display the logical relationship between nodes in the webpage, providing data support for interactive editing and visualization of the flowchart.

[0109] It should be noted that the JavaScript object array is an array composed of multiple JavaScript objects, each object representing a logical relationship and containing attributes such as source node reference, target node reference and connection type identifier.

[0110] For example, parse the control flow graph data, extract the logical relationship between nodes, create a JavaScript object for each logical relationship, including source node reference, target node reference and connection type identifier. Store all JavaScript objects in an array to obtain the JavaScript object array of logical relationships. Integrate the constructed array into the JavaScript code of the page.

[0111] Step A13, integrate and package the HTML container element and the JavaScript object array to generate the structured data.

[0112] Integrate and package the HTML container element and the JavaScript object array to generate the required structured data, ensuring that the data can be loaded and rendered, and realizing the visualization and interactive editing of the flowchart.

[0113] For example, according to the requirements of the target data format, create a JavaScript object as a container for structured data, containing node and relationship fields. Traverse the created HTML container element, extract the relevant information of each node, and convert it into a node object that meets the target data format, and add it to the node field of the structured data object. Assign the JavaScript object array encoded for the logical relationship directly to the relationship field of the structured data object.

[0114] Step A20, constructing the interactively editable flowchart visualization interface based on the structured data.

[0115] The generated structured data is used to construct the interactively editable flowchart visualization interface, enabling users to intuitively view, edit and operate the flowchart, improving the usability of the flowchart.

[0116] It should be noted that the interactively editable flowchart interface is an interface through which users can directly operate and edit the flowchart, supporting functions such as node dragging, adding, deleting, and attribute editing.

[0117] For example, a container element for displaying the flowchart is created in the HTML page using the graphical modeling tool, an instance of the graphical modeling tool is initialized, and it is bound to the created container element. According to the node information, the application programming interface (API) of the graphical modeling tool is used to create corresponding graphical elements, and attributes are set for each node element. According to the connection relationship, the application programming interface of the graphical modeling tool is used to create connection line elements, and attributes are set for each line element. All created node and connection line elements are rendered to the canvas of the graphical modeling tool.

[0118] The embodiment provides a flowchart image parsing and structured reconstruction method. The embodiment first converts the control flowchart data into a required structured data format, ensuring data compatibility and usability. A renderable interactive flowchart visualization interface is generated, improving the usability of the flowchart.

[0119] Based on embodiment one and embodiment three, embodiment four of the present application proposes a flowchart image parsing and structured reconstruction method, which is described with reference to Figure 4 , after step S50, including: Step S60, adding a version management module to the flowchart visualization interface, and in response to a version saving instruction, generating a version snapshot of the flowchart visualization interface and storing it as structured data.

[0120] The version management module is added to the flowchart visualization interface to record and save modifications of the flowchart at different stages. When the user completes the editing operation of the flowchart, a version snapshot is generated through the version saving instruction and stored in the form of structured data. This facilitates users to view the historical versions of the flowchart and understand the evolution process of the flowchart, and also enables the user to restore to a previous version when needed, avoiding data loss or incorrect modifications due to misoperation or other reasons.

[0121] It should be noted that the version management module is a software component or functional module for managing different versions of software or data, which can record the modification history of data, support version saving, comparison, rollback, etc. The version saving instruction is a command issued by the user to trigger the saving of the current flowchart visualization interface state. The version snapshot is a complete record of the flowchart visualization interface state at a certain time, including the information of all nodes in the flowchart, the connection relationship between the nodes, and other related settings of the interface, saved in the form of structured data.

[0122] For example, based on the user clicking the version saving button, the version saving instruction is triggered, all data of the current flowchart visualization interface is obtained, including node information, connection relationship, interface settings, etc., which is converted into structured data format, and a corresponding version number is generated based on the timestamp and saved.

[0123] Step S70, in response to the version comparison instruction, based on the two target versions selected by the user, difference analysis is performed and the difference points are displayed through visual markers.

[0124] In response to the user's version comparison instruction, after the user selects two target versions, difference analysis is performed and the difference points are displayed through visual markers. The user can intuitively understand the changes between the two versions, including the addition, deletion, modification of nodes, and the change of connection relationship. This helps users quickly locate and understand the modification content of the flowchart, facilitating auditing, evaluation or decision-making.

[0125] It should be noted that the version comparison instruction is a command issued by the user to compare two different versions of the flowchart visualization interface. The target version is the two flowchart versions selected by the user for comparison. Difference analysis is a comparison of the flowchart visualization interfaces of the two target versions. Visual markers are used to mark and highlight the difference points on the flowchart visualization interface by using different colors, shapes, icons, etc., in order to highlight the differences between the two versions.

[0126] For example, based on the version comparison instruction, the version numbers of the two target versions selected by the user are determined, and the version snapshot data of the two target versions is obtained from the database according to the version numbers. The two version snapshot data are compared item by item, and the difference points between the two versions are recorded. According to the results of the difference comparison, the difference points are visually marked on the flowchart visualization interface.

[0127] Step S80, in response to the version rollback instruction, the flowchart visualization interface is restored to the specified version state data.

[0128] By the version rollback instruction, the flowchart visualization interface is restored to the specified version state data, when the user finds that the current version of the flowchart has problems or needs to return to a previous version, the specified version state can be quickly and accurately restored, and the correctness of the flowchart is ensured.

[0129] It should be noted that the version rollback instruction is a command issued by the user to restore the flowchart visualization interface to a specified version. The specified version state data is the complete data of the flowchart visualization interface corresponding to the version selected by the user when performing version rollback.

[0130] For example, through the version rollback interface, a list of all saved versions is displayed. Based on the specified version selected by the user and the version rollback button clicked, the version rollback instruction is triggered. The version snapshot data of the specified version is obtained from the database. The data of the current flowchart visualization interface is replaced with the version snapshot data of the specified version, including updating node information, connection relationship, interface settings, etc.

[0131] The embodiment provides a flowchart image analysis and structured reconstruction method. The embodiment first stores version snapshots in the form of structured data through a version management function, thereby ensuring data stability.

[0132] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the flowchart image analysis and structured reconstruction method of the present application. Based on this technical concept, more forms of simple transformation are within the protection scope of the present application.

[0133] The present application provides a flowchart image analysis and structured reconstruction device. The flowchart image analysis and structured reconstruction device comprises at least one processor and a memory in communication connection with the at least one processor. The memory stores instructions executable by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the flowchart image analysis and structured reconstruction method in the above-mentioned embodiment one.

[0134] Reference will be made to the following Figure 5 , which shows a schematic structural diagram of a device for parsing and structurally reconstructing flowchart images suitable for implementing an embodiment of the present application. The device for parsing and structurally reconstructing flowchart images in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The device for analyzing and structurally reconstructing the flowchart image shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0135] like Figure 5 As shown, the flowchart image parsing and structural reconstruction device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for the operation of the flowchart image parsing and structural reconstruction device. Processing device 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, hard disk, etc.; and communication devices 1009. Communication devices 1009 can allow the flowchart image parsing and structural reconstruction apparatus to communicate wirelessly or wired with other devices to exchange data. While the diagram illustrates a flowchart image parsing and structural reconstruction apparatus having various systems, it should be understood that implementation or presence of all illustrated systems is not required. More or fewer systems may alternatively be implemented or present.

[0136] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for executing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network through a communication device, or installed from the storage device 1003, or installed from the read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiments disclosed in the present application are executed.

[0137] The flowchart image analysis and structured reconstruction device provided by the present application adopts the flowchart image analysis and structured reconstruction method in the above embodiments, which can solve the technical problem of how to accurately identify the control logic structure of the flowchart and reconstruct the structured flowchart. Compared with the prior art, the beneficial effects of the flowchart image analysis and structured reconstruction device provided by the present application are the same as those of the flowchart image analysis and structured reconstruction method provided by the above embodiments, and other technical features in the flowchart image analysis and structured reconstruction device are the same as those disclosed in the above embodiments, which will not be repeated here.

[0138] It should be understood that various parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0139] The above is merely specific implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0140] The present application provides a computer readable storage medium having stored thereon computer readable program instructions (i.e. computer program) for executing the flowchart image analysis and structured reconstruction method in the above embodiments.

[0141] The computer readable storage medium provided in the present application may, for example, be a U disk, but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), an optical fiber, a portable compact disk read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer readable storage medium can be transmitted in any suitable medium, including but not limited to electrical wires, optical cables, radio frequency (RF), and the like, or any suitable combination of the above.

[0142] The above computer readable storage medium can be included in the flowchart image analysis and structured reconstruction device, or can exist separately without being assembled into the flowchart image analysis and structured reconstruction device.

[0143] The above computer readable storage medium carries one or more programs, which, when executed by the flowchart image analysis and structured reconstruction device, enable the flowchart image analysis and structured reconstruction device to be written in one or more programming languages or combinations thereof for executing computer program codes for performing the operations of the present application, the programming languages including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" language or similar programming languages. The program code can be executed entirely on the user computer, partially on the user computer, as an independent software package, partially on the user computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, through an Internet service provider to connect through the Internet).

[0144] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functions, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the block can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0145] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0146] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer program) for executing the flowchart image analysis and structured reconstruction method described above, and can solve the technical problem of how to accurately identify the control logic structure of the flowchart and reconstruct the structured flowchart. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the flowchart image analysis and structured reconstruction method provided by the above-mentioned embodiments, and will not be described here.

[0147] The above only describes some embodiments of the present application, and does not limit the patent scope of the present application. Any equivalent structural transformation made by using the content of the present application specification and drawings, or direct / indirect application in other related technical fields is included in the patent protection scope of the present application.

Claims

1. A method for analyzing and structurally reconstructing a flowchart image, characterized in that: The method for analyzing and structurally reconstructing the flowchart image includes: Performing element-background separation on the input image using a pre-trained convolutional neural network to obtain a flowchart image, wherein the elements include text elements and graphic elements; Extracting rectangular frame information, arrow information, text information and position coordinates thereof from the pre-processed flowchart image; Determining the node type of each node in the flowchart image based on the rectangular frame information and the text information, and determining the connection relationship of each node based on the arrow information combined with the position coordinates; Analyzing the node type and the connection relationship of each node, modeling the logical relationship of each node, and generating a control flow graph of the flowchart image; The control flow graph is converted into structured data, and rendered to generate an interactively editable flowchart visualization interface.

2. The method for analyzing and structurally reconstructing a flowchart image according to claim 1, wherein: The step of performing element-background separation on the input image through a pre-trained convolutional neural network to obtain a flowchart image, wherein the elements include text elements and graphic elements, comprises: Performing pixel-level semantic segmentation on the input image using the pre-trained convolutional neural network to separate the text element area and the background area of ​​the input image; Extracting a feature map of the input image through the convolutional neural network and applying an edge detection operator to generate an edge feature map to extract a graphic element region of the input image; The flowchart image is obtained by combining the text element region and the graphic element region of the input image.

3. The method for analyzing and structurally reconstructing a flowchart image according to claim 1, wherein: The step of extracting rectangular frame information, arrow information, text information and position coordinates thereof from the pre-processed flowchart image comprises: Based on the graphic elements of the flowchart image, a contour that meets the shape characteristics of the rectangular frame is screened out as the rectangular frame information; According to a preset arrow shape template, matching is performed with the graphic element of the flowchart image to determine the starting point position information, the ending point position information and the direction information of the arrow as the arrow information; Character recognition is performed on the text elements of the flowchart image to obtain the content of each text element, and the coordinates corresponding to the center point of the text element are recorded as the text information and its position coordinates.

4. The method for analyzing and structurally reconstructing a flowchart image according to claim 1, wherein: The step of determining the node type of each node in the flowchart image based on the rectangular frame information and the text information, and determining the connection relationship of each node based on the arrow information combined with the position coordinates includes: Based on the rectangular frame information of each rectangular frame in the flowchart image and in combination with the rectangular frame shape specification corresponding to the node type, preliminarily determining the node type of the node corresponding to each rectangular frame, wherein the node type includes a start node, an end node, a conditional judgment node, and a processing node; Extracting keywords from the text information in each of the rectangular boxes, and matching the keywords with mapping keywords in a preset rule library to further determine the node type corresponding to each of the text information; The position coordinates matched by the starting point position information of the arrow information are used as the starting node of the connection relationship, and the position coordinates matched by the ending point position information of the arrow information are used as the ending node of the connection relationship to determine the connection relationship of each node.

5. The method for analyzing and structurally reconstructing a flowchart image according to claim 1, wherein: The step of analyzing the node type and the connection relationship of each node, modeling the logical relationship of each node, and generating a control flow graph of the flowchart image includes: Mapping each of the nodes and the connection relationships into node objects and edge objects in the control flow graph to construct a basic structure of the control flow graph; Inferring the logical relationship of each node based on the node type and the connection relationship of each node, and establishing a state transition model to determine the transfer direction of the control flow, wherein the logical relationship includes sequential execution, conditional judgment, parallel nodes, and loop nodes; The basic structure of the control flow graph and the state transition model are combined to form the control flow graph that describes the overall control logic of the flowchart image.

6. The method for analyzing and structurally reconstructing a flowchart image according to claim 1, wherein: The steps of converting the control flow graph into structured data and rendering to generate an interactively editable flow chart visualization interface include: Converting each node information and corresponding logical relationship in the control flow graph into structured data; The interactively editable flowchart visualization interface is constructed based on the structured data.

7. The method for analyzing and structurally reconstructing a flowchart image according to claim 6, wherein: The step of converting the node information and the corresponding logical relationship in the control flow graph into structured data includes: Encoding each node information in the control flow graph into an HTML container element, including node type, text content and position coordinates; Encoding the logical relationship in the control flow graph into a JavaScript object array, including a source node reference, a target node reference, and a connection type identifier; The HTML container element and the JavaScript object array are integrated and encapsulated to generate the structured data.

8. The method for analyzing and structurally reconstructing a flowchart image according to claim 1, wherein: After the step of converting the control flow graph into structured data and rendering the generated interactively editable flow chart visualization interface through a graphic modeling tool, the method includes: Adding a version management module to the flowchart visualization interface, generating a version snapshot of the flowchart visualization interface in response to a version save instruction, and storing the snapshot as structured data; In response to a version comparison instruction, performing a difference analysis based on two target versions selected by a user and displaying the difference points through visual marks; In response to the version rollback instruction, the flowchart visualization interface is restored to the specified version state data.

9. A device for analyzing and structurally reconstructing flowchart images, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the method for parsing and structurally reconstructing a flowchart image according to any one of claims 1 to 8.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the method for parsing and structurally reconstructing a flowchart image according to any one of claims 1 to 8 are implemented.

Images