Hand-drawn flowchart recognition method and device, storage medium and electronic equipment

Abstract

The application discloses a hand-drawn flowchart recognition method and device, a storage medium and equipment. Each element node and line segment key point in a hand-drawn flowchart is detected through a target detection network, and visual features of each element node and key point features of the line segment key point are extracted. The key point features are used to improve the accuracy of line segment representation. An initial graph network is constructed according to the visual features of each element node and the key point features of the line segment key point, and initial features of vertices and edges in the graph network are determined. The graph network is subjected to corresponding graph network feature extraction processing and classification processing to obtain the connection relationship between each element node in the hand-drawn flowchart. The target detection network and the graph network are cascaded to recognize the element nodes and the connection relationship in the hand-drawn flowchart, thereby improving the accuracy of the hand-drawn flowchart recognition.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a method, apparatus, computer-readable storage medium, and electronic device for recognizing hand-drawn flowcharts. Background Technology

[0002] With the emergence of various smart devices, especially the widespread use of tablets, electronic whiteboards, and smartphones, information recording has become more convenient. Among these, handwriting (drawing) is one of the most natural and effective ways for humans to record information.

[0003] Existing handwritten data can be broadly categorized into two types: text and graphics. While text remains the primary medium for information transmission in daily life, with societal progress, the use of graphics to convey information is becoming increasingly important, especially in office and educational settings, where flowcharts are extremely convenient and crucial for presenting important information. However, handwritten flowcharts cannot be standardized for display or editing. Furthermore, due to the greater freedom in handwriting, directly displaying handwritten flowcharts can be aesthetically unappealing. Therefore, structuring handwritten flowcharts to make them more visually appealing and editable has significant practical value and commercial potential.

[0004] Currently, mainstream flowchart recognition methods primarily focus on recognizing printed flowcharts. However, compared to printed flowcharts, handwritten flowcharts are characterized by their varied forms and irregular layouts. For example, different writers have different writing habits, resulting in significant differences even between the same flowchart. Furthermore, even the same writer will produce variations when writing the same flowchart at different times. These issues pose substantial challenges to handwritten flowchart recognition. Summary of the Invention

[0005] This application provides a method, apparatus, computer-readable storage medium, and electronic device for recognizing hand-drawn flowcharts. It utilizes a cascaded object detection network and graph network to recognize hand-drawn flowcharts, thereby improving the accuracy of hand-drawn flowchart recognition.

[0006] This application provides a method for recognizing hand-drawn flowcharts, including:

[0007] Obtain hand-drawn flowcharts;

[0008] An object detection network is used to detect each element node and line segment key point in the hand-drawn flowchart, and the visual features of each element node and the key point features of the line segment key point are extracted.

[0009] An initial graph network is constructed based on the visual features of each element node and the key point features of the line segment key points, and the initial features of the vertices and edges of the initial graph network are determined.

[0010] Based on the initial features of the vertices and the initial features of the edges, graph network features are extracted and classified on the initial graph network to identify the connection relationships between the vertices and edges, and the connection relationships are mapped to the connection relationships between each element node in the hand-drawn flowchart.

[0011] The recognition result of the hand-drawn flowchart is determined based on the connection relationship.

[0012] This application also provides a hand-drawn flowchart recognition device, including:

[0013] The acquisition module is used to acquire hand-drawn flowcharts;

[0014] The detection module is used to use an object detection network to detect each element node and line segment key point in the hand-drawn flowchart, and to extract the visual features of each element node and the key point features of the line segment key point.

[0015] The graph construction module is used to construct an initial graph network based on the visual features of each element node and the key point features of the line segment key points, and to determine the initial features of the vertices and the initial features of the edges of the initial graph network.

[0016] The graph feature classification module is used to extract and classify graph network features of the initial graph network based on the initial features of the vertices and the initial features of the edges, so as to identify the connection relationship between the vertices and the edges and map the connection relationship to the connection relationship between each element node in the hand-drawn flowchart.

[0017] The determination module is used to determine the recognition result of the hand-drawn flowchart based on the connection relationship.

[0018] This application also provides a computer-readable storage medium storing a computer program adapted for loading by a processor to perform the steps described in any of the above embodiments.

[0019] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the steps of the method described in any of the above embodiments by calling the computer program stored in the memory.

[0020] The hand-drawn flowchart recognition method, apparatus, computer-readable storage medium, and electronic device provided in this application's embodiments detect each element node and line segment key point in the hand-drawn flowchart using an object detection network, and extract the visual features of each element node and the key point features of the line segment key points. Specifically, the key point features of the line segment key points are used to represent the line segments instead of their visual features, improving the accuracy of line segment representation and enhancing the accuracy of determining connection relationships later. An initial graph network is constructed based on the visual features of each element node and the key point features of the line segment key points, and the initial vertices in the graph network are determined. This paper first extracts initial features and edge features from the graph network, and then performs corresponding graph network feature extraction and classification processing on the graph network to obtain the connection relationships between each element node in the hand-drawn flowchart. The graph network processing method is used to process the graph network composed of each element node and line segment key point to identify the connection relationships between each element node in the hand-drawn flowchart. Compared with using preset rules to identify connection relationships, the accuracy of connection relationship determination is improved. In this application, the object detection network and graph network are cascaded to identify element nodes and connection relationships in the hand-drawn flowchart, thereby improving the accuracy of hand-drawn flowchart recognition. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of the hand-drawn flowchart recognition method provided in the embodiments of this application.

[0023] Figure 2 A schematic diagram of the improved Faster RCNN detection module provided in the embodiments of this application.

[0024] Figure 3 This is a schematic diagram illustrating the process of target detection using the target detection network provided in this application embodiment.

[0025] Figure 4 This is a schematic diagram of the graph network processing provided in an embodiment of this application.

[0026] Figure 5 This is a schematic diagram illustrating the process of constructing an initial graph network as provided in an embodiment of this application.

[0027] Figure 6 This is a schematic diagram illustrating the process of determining the connection relationship between vertices and edges, as provided in an embodiment of this application.

[0028] Figure 7 This is a schematic diagram illustrating the process of obtaining a sparse graph network as provided in an embodiment of this application.

[0029] Figure 8a This is a schematic diagram of the neighboring vertices and neighboring edges of a point provided in an embodiment of this application.

[0030] Figure 8b This is a schematic diagram of the neighboring vertices of an edge provided in an embodiment of this application.

[0031] Figure 9 This is an example diagram of the entire processing procedure in a hand-drawn flowchart provided for an embodiment of this application.

[0032] Figure 10 This is a schematic diagram of the hand-drawn flowchart recognition device provided in an embodiment of this application.

[0033] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0034] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] This application provides a method, apparatus, computer-readable storage medium, and electronic device for recognizing hand-drawn flowcharts. Specifically, the hand-drawn flowchart recognition method of this application can be executed by an electronic device, and the hand-drawn flowchart recognition apparatus of this application is integrated into an electronic device, wherein the electronic device can be a terminal device or a server, etc. The terminal device includes, but is not limited to, smartphones, tablets, laptops, touch screens, personal computers (PCs), smart vehicle terminals, smart robots, smart whiteboards, smart conference equipment, and smart educational equipment such as large-screen teaching displays. The server can be an independent physical server, a service node in a blockchain system, a server cluster composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, and big data and artificial intelligence platforms. The electronic device may include a display screen for displaying the hand-drawn flowcharts and structured flowcharts of this application, as well as implementing the hand-drawn flowchart recognition method.

[0036] In this application, hand-drawn flowcharts and handwritten flowcharts have the same meaning, including unstructured flowcharts drawn directly on a display screen using hands or input tools such as various pens, and even unstructured flowcharts drawn directly on paper notebooks.

[0037] Before formally introducing the solutions in the embodiments of this application, a brief introduction to the hand-drawn flowchart will be given.

[0038] Hand-drawn flowcharts are characterized by variability, uncertainty, and a high degree of freedom. Specifically, they exhibit variability and uncertainty in writing style: different writers have different writing styles, resulting in vastly different forms for the same type of basic elements. Even the same writer may produce different forms of the same element at different times, leading to significant visual differences. They also exhibit variability and complexity in graphic composition: due to different needs, a flowchart may be composed of multiple different basic elements, and variations in writing style contribute to the overall diversity and complexity of the flowchart. Furthermore, space constraints can lead to complex layouts in handwritten flowcharts. Finally, data acquisition is challenging: in practical applications, obtaining accurate data for handwritten flowcharts is difficult, and annotation is also challenging. Therefore, acquiring a large amount of labeled data is a time-consuming and labor-intensive process.

[0039] Most current methods for recognizing hand-drawn flowcharts primarily address the recognition of printed flowcharts. Some methods tackle handwritten flowcharts, for example, by first obtaining multiple elements from the hand-drawn flowchart and then determining the relationships between them. In particular, the methods for determining relationships between elements mainly rely on heuristic thresholds or other rule-based relationship determination methods. However, handwritten flowcharts have greater flexibility in layout, which may result in two connected elements being spatially far apart, and the endpoints of line segments being far from the connected elements. Determining the relationship by identifying which two elements the endpoints of the line segments are closest to can lead to thresholds that cannot handle the complexity of hand-drawn flowcharts, making current relationship determination methods very unrobust.

[0040] To improve the accuracy and robustness of hand-drawn flowchart recognition, this application provides a hand-drawn flowchart recognition method, apparatus, computer-readable storage medium, and electronic device, which will be described below.

[0041] It should be noted that the hand-drawn flowchart recognition method in this application is an end-to-end recognition method, which takes a hand-drawn flowchart as input and outputs a structured recognition result, i.e., a structured flowchart.

[0042] Figure 1This is a schematic diagram of the hand-drawn flowchart recognition method provided in this application embodiment. The method is applied in electronic devices and specifically includes the following steps.

[0043] 101. Obtain the hand-drawn flowchart.

[0044] The hand-drawn flowchart can be an image uploaded to the electronic device from other electronic devices, an image drawn directly on the electronic device, or even an image containing the hand-drawn flowchart obtained by taking a picture of the hand-drawn flowchart using the electronic device.

[0045] 102. An object detection network is used to detect the key points of each element node and line segment in the hand-drawn flowchart, and the visual features of each element node and the key point features of the line segment are extracted.

[0046] The object detection network can be any deep neural network used for object detection; this application uses the Faster R-CNN detection network as an example. The embodiments in this application improve upon the Faster R-CNN detection network, which will be discussed later.

[0047] The Faster R-CNN detection network in this application includes a feature extraction network, a region candidate network, an interest region pooling network, and a classification and regression module. The connection relationships between these networks are as follows: Figure 2 As shown. From Figure 2 As can be seen, a prediction network has been added to the classification and regression module of the Faster R-CNN detection network, and an interest domain pooling network has been added after the classification and regression module. The implementation logic of the newly added interest domain pooling network is the same as that of the previous interest domain pooling network.

[0048] The feature extraction network is used to extract feature maps of the hand-drawn flowchart. The feature extraction network can be a cascaded convolutional neural network, for example, consisting of n convolutional layers, n activation layers and m pooling layers. Through convolution and downsampling operations, the receptive field is gradually expanded and background noise is filtered out, thereby obtaining an effective visual feature representation, that is, the feature map of the hand-drawn flowchart. The dimension of the hand-drawn flowchart can be M*N. Assuming there are 4 pooling layers, the dimension of the feature map is (M / 16)*(N / 16).

[0049] The Region Proposal Network (RPN) generates multiple candidate regions (also known as candidate boxes or detection boxes). It performs region detection processing on the feature map obtained from the feature extraction network to obtain multiple candidate regions and their location information. The RPN module mainly consists of two parts: one uses binary classification to determine whether a pre-defined set of candidate regions contains a target, and the other uses bounding box regression to obtain more accurate coordinates (location information, including the coordinates of the four points enclosing the candidate region) of the candidate regions (proposals) containing the target. The size of the candidate regions varies.

[0050] Among them, the Region of Interest Pooling (RoI Pooling) network is used to collect the location information of the target candidate regions generated by the region candidate network. Based on the location information of multiple candidate regions, it extracts the region visual feature maps corresponding to each candidate region from the feature map of the feature extraction network. The region visual feature maps corresponding to each candidate region have the same dimension.

[0051] The classification and regression module is used to classify the visual feature maps of the regions to determine the specific categories of multiple candidate regions in the hand-drawn flowchart, i.e., which basic element each candidate region corresponds to. It then performs a second candidate region regression process to obtain the precise location information of each candidate region within the hand-drawn flowchart. Basic elements include element nodes, line segments, and / or line segments with arrows. Element nodes include elements such as rectangles, circles, ellipses, and rhombuses.

[0052] When the basic elements are line segments and / or arrowed line segments, the classification and regression module is used to identify the key points of the line segments and / or arrowed line segments and their location information in the hand-drawn flowchart. Key points refer to the key points within a line segment or arrowed line segment. A line segment or arrowed line segment may contain multiple key points, such as two, three, or four. Three key points are sufficient to effectively represent a Bézier curve. For example, three key points may include the start point, the end point, and a point between the start and end points of the line segment or arrowed line segment. The point between the start and end points can be called the intermediate point.

[0053] Understandably, this application improves the classification regression module. This module can not only perform candidate region regression processing on element nodes to obtain accurate location information, but also predict multiple key points in line segments or line segments with arrows. Specifically, the prediction network in the classification regression module predicts multiple key points, and the classification regression module performs regression processing based on the predicted line segment key points to obtain the accurate location information of each key point in the line segment key points, such as... Figure 2 As shown.

[0054] In some embodiments, for ease of description, the feature extraction network, the region candidate network, and the (first) interest region pooling network are collectively referred to as the feature extraction module.

[0055] In one embodiment, such as Figure 3 As shown, the steps described above for using an object detection network to detect the various element nodes and key points of line segments in a hand-drawn flowchart include the following steps 201 to 203.

[0056] 201. Using the feature extraction module of the object detection network, multiple candidate regions and their corresponding visual feature maps are extracted from the hand-drawn flowchart.

[0057] Among them, the sizes of multiple candidate regions are different, and the dimensions of the visual feature maps corresponding to multiple candidate regions are the same.

[0058] Step 201 includes: extracting feature maps of the hand-drawn flowchart using a feature extraction network of an object detection network; performing region detection processing on the feature maps using a region candidate network of the object detection network to obtain multiple candidate regions in the feature maps and their position information in the hand-drawn flowchart; and extracting region visual feature maps corresponding to the multiple candidate regions from the feature maps based on their position information using a region pooling network of the object detection network. Please refer to the corresponding descriptions above for details.

[0059] 202. The classification and regression module of the object detection network is used to classify the visual feature map of the region to determine that multiple candidate regions correspond to the basic elements in the hand-drawn flowchart. The basic elements include element nodes and line segments or line segments with arrows.

[0060] The classification and regression module is a multi-class classification process that can classify the basic elements corresponding to each candidate region in order to obtain the basic elements corresponding to each candidate region.

[0061] 203. When the basic element is an element node, the classification and regression module is used to identify the position information of the element node in the hand-drawn flowchart; when the basic element is a line segment or a line segment with an arrow, the classification and regression module is used to identify the key points of the line segment or the line segment with an arrow and their position information in the hand-drawn flowchart.

[0062] When the basic elements are elements such as squares, circles, ellipses, and rhombuses, the classification and regression module is used to identify the position information of the element nodes in the hand-drawn flowchart. For example, the coordinate offset of the candidate region of the element node is identified, and the original coordinates of the candidate region are obtained. The position information of the element node in the hand-drawn flowchart is calculated based on the original coordinates and the coordinate offset.

[0063] When the basic element is a line segment or an arrowed line segment, the prediction network in the classification and regression module first predicts a preset number of key points within the line segment or arrowed line segment, along with the coordinates of these key points within the candidate regions corresponding to the line segment or arrowed line segment. The classification and regression module then identifies the coordinate offset of the candidate regions corresponding to the line segment or arrowed line segment. The preset number can be two, three, four, etc. Taking three as an example, three key points are predicted within the line segment or arrowed line segment. These three key points are then used as the line segment key points. Based on the coordinate offset of the candidate regions corresponding to the line segment or arrowed line segment, the position information of the line segment key points in the hand-drawn flowchart is determined. For example, the position information of the key points in the hand-drawn flowchart is determined based on the original coordinates of the candidate regions corresponding to the line segment or arrowed line segment, the coordinates of the key points within the candidate regions, and the coordinate offset.

[0064] It should be noted that in this embodiment, multiple key points in the line segment key points are used to simulate / represent a line segment.

[0065] In this embodiment, the prediction network of the classification regression module can predict the key points of line segments or arrowed line segments because, during the training phase, the key points of the line segments or arrowed line segments are labeled. The labeled key points are used for training to obtain the parameters of the corresponding classification regression module. Specifically, for example, a loss function is calculated using the labeled key points and the predicted key points of the line segments from the classification regression module, and the parameters of the classification regression module are updated based on the loss function. Therefore, when using the classification regression module, the corresponding parameters can be used to predict the key points of line segments.

[0066] Further, see next. Figure 3 The steps described above for extracting the visual features of each element node and the key point features of the line segment key points include steps 204 to 205.

[0067] 204. The regional visual feature map of the candidate region corresponding to each element node is used as the visual feature of each element node.

[0068] In terms of element nodes, it is understandable that the regional visual feature map of an element node can better reflect the characteristics of the element node itself.

[0069] 205. The interest region pooling network of the object detection network is used to determine the key point features of the key points of the line segment or the line segment with arrows.

[0070] Specifically, the interest region pooling network of the object detection network is used to extract key point features of line segments or line segments with arrows from the feature map based on the location information of the key points of the line segments. For example, features matching the corresponding location information are extracted from the feature map based on the location information of the key points of the line segments, and the extracted features are used as the key point features of the corresponding key points. The feature map is obtained by extracting hand-drawn flowcharts using the feature extraction network of the object detection network.

[0071] In this application, since multiple key points are used to represent line segments or line segments with arrows, multiple key points can better realize Bézier curves. Multiple key points are used to represent the points forming the Bézier curve, and the Bézier curve is used to simulate line segments or line segments with arrows. Therefore, the key point features of the key points can also be further called Bézier features.

[0072] For line segments or arrowed line segments, since the spatial span of line segments or arrowed line segments in a flowchart may be very large, directly using visual features to represent line segments or arrowed line segments would make it difficult to understand the features of the edges in the graph network during subsequent construction, thus affecting the judgment of the connection relationship of element nodes. Therefore, in this embodiment, the visual features of line segments or arrowed line segments are not directly used. Instead, the key points of the line segments or arrowed line segments are first determined, and then the key point features (Bezier features) of the key points are determined. The key point features (Bezier features) are used to represent the features of line segments or arrowed line segments, thereby improving the accuracy of connection relationship judgment.

[0073] The location information of the key points of the line segment is the location information in the hand-drawn flowchart (M*N). Therefore, each key point in the key points of the line segment is first mapped back to the scale of the feature map, that is, the scale of (M / 16)*(N / 16). Then, each key point is divided into a pooled_w*pooled_h grid, and max pooling is performed on each grid to obtain the feature of each key point. In this way, the feature of each key point is of the same dimension. Based on the feature of each key point of the line segment or the arrowed line segment, the key point feature (Bezier feature) of the line segment or the arrowed line segment is obtained. The feature of each key point is independent, that is, each key point corresponds to a key point feature (Bezier feature).

[0074] Here, pooled_w*pooled_h is a preset width and height value in the interest region pooling network. The dimension of the region visual feature map of each different candidate region is unified based on the preset width and height, as well as the dimension of the key point features (Bezier features) of the key points of each different line segment / arrowed line segment.

[0075] Understandably, after obtaining the key points of the line segments and their position information in the hand-drawn flowchart, this application inputs the key points of the line segments and their position information into a newly added interest region pooling network in the Faster RCNN detection network to obtain key point features (Bezier features).

[0076] After obtaining the visual features of the element nodes and the keypoint features (Bezier features) of each keypoint in the line segment keypoints, a graph network is used to determine the connection relationships between the element nodes, specifically including steps 103 and 104. This can be based on... Figure 4 Let's understand the content about graph networks described below. Specifically, in... Figure 4 The structure includes an input layer (input features, including visual features of each element and key point features of line segment key points), an initial graph network, a hidden layer (the hidden layer includes multiple graph convolutional networks based on self-attention modules, resulting in a sparse graph network), and an output layer.

[0077] 103. Construct an initial graph network based on the visual features of each element node and the key point features of the line segment key points, and determine the initial features of the vertices and edges of the initial graph network.

[0078] Step 103 mainly involves constructing and initializing the graph network to obtain the initial graph network.

[0079] The initial graph network consists of vertices including each keypoint in the line segment keypoints and each element node. The line connecting any two vertices is used as the edge of the initial graph network. The keypoint features of each keypoint in the line segment keypoints and the visual features of each element node are used as the initial features of the vertices of the initial graph network. The initial features of the edge between any two vertices are determined based on the initial features of any two vertices.

[0080] Among them, such as Figure 5 As shown, step 103 includes steps 301 to 303 as follows.

[0081] 301. The initial graph network is constructed by taking each key point and element node in the line segment key points as the vertices of the initial graph network and the line connecting any two vertices as the edges of the initial graph network.

[0082] It should be noted that, in general, each element node is used as a vertex of the initial graph network. However, in this application, each key point in the line segment key points is also used as a vertex of the initial graph network to determine the corresponding connection relationship, which can improve the accuracy of the connection relationship determination.

[0083] The method for constructing the initial graph network can be any method for constructing graph networks. The constructed initial graph network is a dense graph network and is an undirected graph.

[0084] 302. The key point features of each key point in the line segment key points and the visual features of each element node are used as the initial features of the corresponding vertices of the initial graph network.

[0085] If the vertices of the graph network correspond to each element, then the initial features of the vertices are the visual features of each element. If the vertices of the graph network correspond to each key point in the line segment keypoints, then the initial features of the vertices are the keypoint features of the corresponding keypoints.

[0086] For example, suppose two element nodes A and B are identified, and the identified line segment key points include three key points, such as point 1, point 2 and point 3. Then the initial graph network includes 5 vertices, namely A, point 1, point 2, point 3 and B. An edge is formed between any two vertices of these 5 vertices. The initial feature of vertex A is the visual feature of element node A, and the initial feature of the vertex corresponding to point 1 is the key point feature of point 1.

[0087] 303. Determine the initial features of the edge between any two vertices based on the initial features of any two vertices.

[0088] Specifically, the initial features of each vertex in the initial graph network can be processed by nonlinear mapping to obtain the mapping features of each vertex; the mapping features of any two vertices can be fused to obtain the initial features of the edge between any two vertices.

[0089] The nonlinear mapping process can be a mapping process through a fully connected layer, or it can be other nonlinear mapping processes, such as embedding feature extraction. For example, the initial features of each vertex can be input into a fully connected layer for nonlinear mapping, and the output of the fully connected layer can be used as the mapping feature of each vertex. Alternatively, the initial features of each vertex can be input into an embedding layer for nonlinear mapping, and the output of the embedding layer (embedded features) can be used as the mapping feature of each vertex.

[0090] The mapping features of any two vertices are fused, such as by concatenation, to obtain the initial features of the edge between any two vertices.

[0091] The initial features of the vertices of the graph network are represented as V∈R. c×h×w The initial features of the edges are represented as E∈R c The initial value of E can be represented by the embedding features of the initial features of the two vertices, such as the embedding vector, as shown in formula (1).

[0092] E ij =Embed(V i V j (1)

[0093] Among them, V i V represents the i-th vertex. j Let j represent the j-th vertex, and Embed indicates that non-linear mapping processing is performed to obtain the corresponding mapping features, such as embedding features.

[0094] After constructing the graph network, the goal of the graph network construction can be represented as P(V,E,R), where R represents the label of the connection relationship between vertices, that is, processing the graph network to obtain the label of the connection relationship between vertices. Please refer to step 104 below for this part.

[0095] 104. Based on the initial features of vertices and edges, perform graph network feature extraction and classification on the initial graph network to identify the connection relationships between vertices and edges, and map the connection relationships to the connection relationships between each element node in the hand-drawn flowchart.

[0096] To obtain more accurate connection relationships, this application introduces a graph convolutional network based on self-attention modules. The graph convolutional network based on self-attention modules can be described as follows: Figure 4 The GCN Attention part is shown in the diagram.

[0097] The network input for this step of graph network processing includes the initial features of vertices and the initial features of edges.

[0098] Among them, such as Figure 6 As shown, the steps of extracting and classifying graph network features from the initial features of vertices and edges to identify the connection relationships between vertices and edges include the following steps 401 to 403.

[0099] 401. Based on the initial features of vertices and edges, a graph convolutional network based on a self-attention module is used to perform graph network feature extraction and de-densification processing on the initial graph network to obtain a sparse graph network.

[0100] This application adds a graph convolutional network based on a self-attention module to perform feature extraction processing on the graph network. The self-attention module utilizes the self-attention mechanism to bring connected nodes (vertices) closer together and push away unconnected nodes (vertices). The graph convolutional network can extract more surface-level features from the graph network, such as features related to corresponding connections, improving the accuracy of determining the connections between nodes. Feature processing of the graph network using the graph convolutional network based on the self-attention module performs preliminary de-densification, removing weak connections to obtain a sparse graph network. It is important to note that the convolutional layer mentioned below refers to the convolution in the graph network, which is different from image convolution.

[0101] The content of step 401 corresponds to Figure 4 The content of the hidder layer includes at least one graph convolutional network based on a self-attention module.

[0102] Specifically, such as Figure 7 As shown, step 401 includes the following steps.

[0103] 4011. Based on the initial features of the vertices, the initial graph network is processed by a self-attention mechanism using a graph convolutional network based on a self-attention module to obtain the first attention weight between any two vertices.

[0104] For details, please refer to the graph convolutional network based on the self-attention module. Figure 4 The GCN Attention section is shown in the diagram. The initial features of each vertex are input into a graph convolutional network based on a self-attention module for similarity processing to obtain the similarity between any two vertices. The similarity between any two vertices is then normalized to obtain the first attention weight between any two vertices.

[0105] The step described above, which inputs the initial features of each vertex into a graph convolutional network based on a self-attention module for similarity processing to obtain the similarity between any two vertices, includes: inputting the initial features of any two vertices in each vertex into the graph convolutional network based on a self-attention module, and performing activation processing using an activation function. After activation processing, the similarity value between the two vertices is obtained. Therefore, activation processing can also be summarized as similarity processing. One way to perform similarity processing is to use activation processing. Thus, the similarity between any two vertices can be obtained.

[0106] Formula (2) can be used to represent the process of obtaining the similarity between any two vertices.

[0107]

[0108] in, This represents the features of the i-th and j-th vertices processed by the l-th convolutional layer. If the current layer is the first convolutional layer, l = 1, corresponding to the initial features of the i-th and j-th vertices. σ represents the activation function, such as the ReLU activation function. and W a For learnable parameters, u ij This represents the similarity between the i-th and j-th vertices processed by the l-th convolutional layer. It's important to note that similar operations are performed in each convolutional layer; therefore, the formula represents the overall implementation process.

[0109] The step of normalizing the similarity between any two vertices to obtain the first attention weight between any two vertices includes: inputting the similarity between any two vertices into the softmax function for normalization to obtain the first attention weight between any two vertices.

[0110] Formula (3) can be used to express the first attention weight a between any two vertices, such as the i-th vertex and the j-th vertex. ij The process.

[0111] a ij =softmax(u ij (3)

[0112] 4012. For each vertex, a graph convolutional network based on a self-attention module is used to determine the intermediate features of each vertex in the first convolutional layer based on the first attention weights between the vertex and its neighboring vertices and the initial features from the neighboring vertices.

[0113] For each vertex, a graph convolutional network based on a self-attention module is used to obtain the first attention weights between the vertex and all its neighboring vertices, as well as the initial features of all its neighboring vertices and the vertex's own initial features. Then, based on the first attention weights between the vertex and all its neighboring vertices, the initial features of all its neighboring vertices, and the vertex's own initial features, the intermediate features of each vertex in the first convolutional layer are determined. This can be understood as the intermediate features of each vertex in the first convolutional layer including the vertex's own initial features and features from all its neighboring vertices.

[0114] The process of obtaining the intermediate features of each vertex is equivalent to the process of extracting high-level feature representations using graph convolution operations. The process of obtaining the intermediate features of each vertex using graph convolution operations can be represented by the following formula (4), where formula (4) takes the intermediate features of the i-th vertex as an example.

[0115]

[0116] Among them, a ij This represents the first attention weight between the i-th vertex and the j-th vertex processed by the l-th convolutional layer. It represents all neighboring vertices of the i-th vertex, including the i-th vertex itself. Let represent the feature of the j-th neighboring vertex connected to the i-th vertex. This feature is the initial feature in the first convolutional layer. W represents the learnable parameters, obtained using the activation function in formula (4). This refers to the intermediate feature of the i-th vertex.

[0117] 4013. Update the features of each vertex and each edge based on the intermediate features of each vertex and the initial features of each edge to obtain the output features of each vertex and each edge in the first convolutional layer. Use the output features of each vertex and each edge in the first convolutional layer as the input features of each vertex and each edge in the next convolutional layer, respectively.

[0118] Specifically, the features of each vertex in the first convolutional layer are updated based on the intermediate features of each vertex and the initial features of each edge. Similarly, the features of each edge in the first convolutional layer are updated based on the intermediate features of each vertex and the initial features of each edge. These updated features are then used as the output features of each vertex and each edge in the first convolutional layer, respectively, as the input features of each vertex and each edge in the next convolutional layer. This process is performed on the first convolutional layer.

[0119] The step of updating the features of each vertex in the first convolutional layer based on the intermediate features of each vertex and the initial features of each edge includes: for each vertex in the first convolutional layer, obtaining the intermediate features from neighboring vertices and the initial features from neighboring edges; updating the features of the vertex based on the intermediate features from neighboring vertices and the initial features from neighboring edges to obtain the output features of each vertex in the first convolutional layer.

[0120] Formula (5) can be used to represent the process of updating the features of each vertex in the current convolutional layer to obtain the output features of each vertex in the current convolutional layer. Formula (5) takes obtaining the output features of the i-th vertex as an example.

[0121]

[0122] Here, "message from Other Objects" refers to intermediate features from neighboring vertices, and "message from Neighboring Relationships" refers to features from neighboring edges. If it's the first convolutional layer, the features from neighboring edges are the initial features from those edges. For example... Figure 8a As shown, the middle vertex is the i-th vertex, i. There are 4 neighboring vertices currently connected to i, namely 11, 12, 13 and 14. There are also 4 neighboring edges currently connected to i, namely edge 11, edge 12, edge 13 and 14.

[0123] in, Z represents the output feature of the i-th vertex, i.e., the updated feature of the i-th vertex. o The intermediate features of neighboring vertices correspond to Figure 8a This includes the intermediate features of points 11, 12, 13, and 14. `skip` means removing its neighboring vertices. W skip a skip This refers to the two distinct weights of the vertices after removing the weights of their neighboring vertices, which can be learned / trained. Z r This represents the features of neighboring edges. If it's the first convolutional layer, then this feature refers to the initial features of the neighboring edges, corresponding to... Figure 8a The features of the neighboring edges include those of edge 11, edge 12, edge 13, and edge 14.

[0124] Among them, W sr α sr The two distinct weights representing edges from neighboring vertices to the current vertex (e.g., the i-th vertex) can be learned, corresponding to... Figure 8a Then the edges from neighboring vertices to the current vertex include the edges from vertex 11 to vertex i, from vertex 12 to vertex i, from vertex 13 to vertex i, and from vertex 14 to vertex i.or α or This represents two distinct weights of the edges pointing from the current vertex (e.g., the i-th vertex) to its neighboring vertices, which can be learned, corresponding to... Figure 8a Then, the edges pointing from the current vertex to its neighboring vertices include the edges from vertex i to vertex 11, from vertex i to vertex 12, from vertex i to vertex 13, and from vertex i to vertex 14. It should be noted that since the initial graph network is an undirected graph, each edge is considered to have two directions during computation: the direction from the neighboring vertex to the current vertex and the direction from the current vertex to its neighboring vertex.

[0125] The step of updating the features of each edge in the first convolutional layer based on the intermediate features of each vertex and the initial features of each edge includes: for each edge in the first convolutional layer, obtaining the initial features of the edge and the intermediate features from neighboring vertices; updating the features of the edge based on the initial features of the edge and the intermediate features of neighboring vertices to obtain the output features of each edge in the first convolutional layer.

[0126] Formula (6) can be used to represent the process of updating the features of each edge in the current convolutional layer to obtain the output features of each edge in the current convolutional layer. Formula (6) takes obtaining the output features of the i-th edge as an example.

[0127]

[0128] Here, "message from Neighboring Objects" refers to intermediate features derived from neighboring vertices. For example... Figure 8b As shown, the neighboring vertices of edge i include points 21 and 22. In formula (6), Z o This includes the intermediate features of the neighboring vertices of edge i, specifically the intermediate features between vertices 21 and 22. Understandably, an edge is formed between two vertices. (The text inside the parentheses...) This represents the feature of edge i. If it is the first convolutional layer, then this feature of edge i refers to the initial feature of edge i. The left side of the equals sign... This represents the updated features of edge i, i.e., the output features of edge i. Points 21 and 22 are referred to as the principal vertex and child vertex, or the master node and child node, respectively. W rs α rs W represents two different weights that indicate a child vertex pointing to a parent vertex, which can be learned. ro α ro These represent two different weights that point from the main vertex to the child vertex, and can be learned.

[0129] The above explanation uses the example of updating the features of each vertex and each edge in the first convolutional layer to obtain the output features of each vertex and each edge of the first convolutional layer. After obtaining the output features of each vertex and each edge of the first convolutional layer, these output features are used as the input features of each vertex and each edge of the next convolutional layer, respectively.

[0130] 4014, process in the same way until the output features of each vertex and each edge of the last convolutional layer of the graph convolutional network based on the self-attention module are obtained.

[0131] The convolutional layers in a graph convolutional network based on self-attention modules can include multiple layers. For each convolutional layer in the multiple layers, the intermediate features of each vertex in each convolutional layer are obtained in the same way as described above, and the features of each vertex and each edge are updated to obtain the output features of each vertex and each edge in each convolutional layer. Since the processing method is the same, it will not be repeated here, until the output features of each vertex and each edge of the last convolutional layer of the graph convolutional network based on self-attention modules are obtained.

[0132] 4015. Based on the output features of each vertex and each edge of the last convolutional layer, the graph network is de-densified to obtain a sparse graph network.

[0133] It is important to note that since the first attention weights between vertices, as well as the features of edges and vertices, need to be updated each time, after multiple processing steps by the graph convolutional network based on the self-attention module, the features of some vertices become smaller, while the features of some vertices are enhanced, and the features of some edges become very small, meaning that the connection between the corresponding vertices has become very weak. In some cases, the features of edges are also enhanced. Therefore, based on the output features of each vertex and each edge in the last convolutional layer, the graph network is de-densified. For example, edges with smaller features are removed, such as edges with features smaller than a preset edge feature, to obtain a sparse graph network.

[0134] In this regard, de-densification can be performed in each convolutional layer. When the features of the edges in a convolutional layer become small or weak, de-densification can be performed to obtain a sparse graph network, thereby reducing the computational cost of the next convolutional layer and improving efficiency.

[0135] 402. A graph convolutional network based on a self-attention module is used to perform graph network feature extraction processing on the features of vertices and edges in a sparse graph network to obtain enhanced features.

[0136] Both the graph network feature extraction process mentioned above and the graph network feature extraction process here aim to extract features, thereby weakening some features and enhancing others. In some cases, this can also be understood as feature enhancement. This step utilizes a graph convolutional network based on a self-attention module to perform graph network feature extraction on the features of vertices and edges in the sparse graph network, consistent with step 401. Please refer to the description of the corresponding step above for details, which will not be repeated here. The enhanced features include the output features of each vertex and each edge after the graph network feature extraction process.

[0137] 403. Based on the enhanced features, the connection relationships between vertices and edges in the sparse graph network are classified to obtain the connection relationships between vertices and edges.

[0138] After obtaining the enhanced features, namely the output features of each vertex and each edge, a classification process is performed. For example, the output features of each vertex and each edge are input into the ReLU activation function, and the output of the ReLU activation function is input into the softmax function to classify the edges between each vertex. If there is no connection between two vertices, the edge is classified as the first classification result; if there is a connection between two vertices, the edge is classified as the second classification result. The first classification result can be represented by 1, and the second classification result can be represented by 2. In this way, the connection relationship between vertices and edges is obtained.

[0139] After obtaining the connection relationships between vertices and edges in the graph network, these connections are mapped to the connection relationships between element nodes in the hand-drawn flowchart.

[0140] 105. The recognition result of the hand-drawn flowchart is determined based on the connection relationship.

[0141] In some cases, after obtaining the individual nodes and their connections in a hand-drawn flowchart, the connections between these nodes are directly used as the recognition result of the flowchart; that is, only the framework of the flowchart is recognized. Figure 9 As shown. In some cases, it is also possible to obtain the framework of a hand-drawn flowchart by recognizing it, and then identify the text within the flowchart, mapping the text to the corresponding areas within the flowchart's framework to achieve complete flowchart recognition. The recognition result includes all elements and nodes in the flowchart, the connections between them, and the text. In some cases, only the corresponding connections need to be used to determine the recognition result.

[0142] In one embodiment, the hand-drawn flowchart recognition method further includes:

[0143] 106. Perform structured processing on each element node and / or the connection relationships between each element node to obtain the structured recognition result corresponding to the hand-drawn flowchart.

[0144] After obtaining the individual nodes and their connections in the hand-drawn flowchart, to achieve structured recognition results, the nodes and connections need to be rendered in printed format. Therefore, a standard format for each node must first be established, and then the nodes are connected according to their relationships to obtain a more aesthetically pleasing display. In some cases, the text recognized from the hand-drawn flowchart can also be rendered in a standard font format for a more visually appealing presentation.

[0145] like Figure 9 The diagram shown is an example of the entire processing procedure in a hand-drawn flowchart under one scenario. The hand-drawn flowchart first undergoes object detection to obtain basic elements, including element nodes and line segment key points. Then, it is detected based on connection relationships to obtain the connection relationships between the element nodes. Finally, it is structured to obtain the structured recognition result.

[0146] The above embodiments utilize key point features to represent line segments instead of their visual features, improving the accuracy of line segment representation and enhancing the accuracy of determining connection relationships later. Graph network processing is used to process the graph network composed of element nodes and line segment key points to identify the connection relationships between element nodes in the hand-drawn flowchart. Compared to using preset rules to identify connection relationships, this improves the accuracy of connection relationship determination. Furthermore, the element nodes and connection relationships are structured to obtain structured recognition results corresponding to the hand-drawn flowchart, improving its readability and understandability.

[0147] All of the above technical solutions can be combined in any way to form optional embodiments of this application, and will not be described in detail here.

[0148] To facilitate better implementation of the hand-drawn flowchart recognition method of this application, this application also provides a hand-drawn flowchart recognition device. Please refer to... Figure 10 , Figure 10 This is a schematic diagram of the hand-drawn flowchart recognition device provided in an embodiment of this application. The hand-drawn flowchart recognition device 500 may include an acquisition module 501, a detection module 502, a graph construction module 503, a graph feature classification module 504, and a determination module 505.

[0149] Module 501 is used to obtain hand-drawn flowcharts.

[0150] The detection module 502 is used to detect each element node and line segment key point in the hand-drawn flowchart using an object detection network, and to extract the visual features of each element node and the key point features of the line segment key point.

[0151] Graph construction module 503 is used to construct an initial graph network based on the visual features of each element node and the key point features of the line segment key points, and to determine the initial features of the vertices and the initial features of the edges of the initial graph network.

[0152] The graph feature classification module 504 is used to perform graph network feature extraction and classification processing on the initial graph network based on the initial features of the vertices and the initial features of the edges, so as to identify the connection relationship between the vertices and the edges and map the connection relationship to the connection relationship between each element node in the hand-drawn flowchart.

[0153] The determination module 505 is used to determine the recognition result of the hand-drawn flowchart based on the connection relationship.

[0154] In one embodiment, the detection module 502 is specifically used to extract multiple candidate regions and corresponding visual feature maps from the hand-drawn flowchart using the feature extraction module of the object detection network; and to classify the visual feature maps using the classification and regression module of the object detection network to determine that the multiple candidate regions correspond to various basic elements in the hand-drawn flowchart, wherein the basic elements include element nodes and line segments or line segments with arrows; when the basic element is an element node, the classification and regression module identifies the position information of the element node in the hand-drawn flowchart; when the basic element is a line segment or a line segment with arrows, the classification and regression module identifies the line segment key points and their position information in the hand-drawn flowchart. The multiple candidate regions have different sizes, but the visual feature maps corresponding to the multiple candidate regions have the same dimension.

[0155] In one embodiment, the graph construction module 503 is specifically used to construct an initial graph network by using each key point and each element node in the line segment key points as vertices of an initial graph network and the line connecting any two vertices as edges of the initial graph network; using the key point features of each key point in the line segment key points and the visual features of each element node as the initial features of the corresponding vertices of the initial graph network; and determining the initial features of the edges between any two vertices based on the initial features of any two vertices.

[0156] The step of determining the initial features of the edge between any two vertices based on their initial features includes: performing nonlinear mapping on the initial features of each vertex of the initial graph network to obtain the mapping features of each vertex; and fusing the mapping features of any two vertices to obtain the initial features of the edge between any two vertices.

[0157] In one embodiment, the graph feature classification module 504 is specifically used to perform graph network feature extraction and de-densification processing on the initial graph network using a graph convolutional network based on a self-attention module, based on the initial features of the vertices and the initial features of the edges, to obtain a sparse graph network; to perform graph network feature extraction processing on the features of vertices and edges in the sparse graph network using the graph convolutional network based on a self-attention module, to obtain enhanced features; to classify the connection relationships between vertices and edges in the sparse graph network based on the enhanced features, to obtain the connection relationships between the vertices and edges, and to map the connection relationships to the connection relationships between various element nodes in the hand-drawn flowchart.

[0158] In one embodiment, the step of performing graph network feature extraction and de-densification processing on the initial graph network using a graph convolutional network based on a self-attention module, based on the initial features of the vertices and the initial features of the edges, to obtain a sparse graph network, includes:

[0159] Based on the initial features of the vertices, a graph convolutional network based on a self-attention module is used to process the initial graph network using a self-attention mechanism to obtain the first attention weight between any two vertices. For each vertex, the graph convolutional network based on the self-attention module determines the intermediate features of each vertex in the first convolutional layer based on the first attention weight between the vertex and its neighboring vertices and the initial features from the neighboring vertices. The features of each vertex and each edge are updated based on the intermediate features of each vertex and the initial features of each edge to obtain the output features of each vertex and each edge in the first convolutional layer. The output features of each vertex and each edge in the first convolutional layer are used as the input features of each vertex and each edge in the next convolutional layer, respectively. This process is repeated until the output features of each vertex and each edge in the last convolutional layer of the graph convolutional network based on the self-attention module are obtained. Based on the output features of each vertex and each edge in the last convolutional layer, the graph network is de-densified to obtain a sparse graph network.

[0160] In one embodiment, the step of updating the features of each vertex and each edge based on the intermediate features of each vertex and the initial features of each edge to obtain the output features of each vertex and each edge in the first convolutional layer includes: for each vertex in the first convolutional layer, obtaining the intermediate features from neighboring vertices and the initial features from neighboring edges; updating the features of the vertex based on the intermediate features from neighboring vertices and the initial features from neighboring edges to obtain the output features of each vertex in the first convolutional layer; for each edge in the first convolutional layer, obtaining the initial features of the edge and the intermediate features from neighboring vertices; updating the features of the edge based on the initial features of the edge and the intermediate features from neighboring vertices to obtain the output features of each edge in the first convolutional layer.

[0161] In one embodiment, the device further includes a structuring module 506, used to perform structuring processing on the element nodes and / or the connection relationships to obtain the structuring recognition result corresponding to the hand-drawn flowchart.

[0162] All the above technical solutions can be combined in any way to form optional embodiments of this application. For the specific content and more detailed content of all the above technical solutions, please refer to the content described in the method embodiments above. The beneficial effects that can be achieved can also be referred to the parts described in the method embodiments above, and will not be repeated here.

[0163] Accordingly, embodiments of this application also provide an electronic device, which can be a terminal or a server, capable of implementing the hand-drawn flowchart recognition method described in this application. Figure 11 As shown, Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 600 includes a processor 601 with one or more processing cores, a memory 602 with one or more computer-readable storage media, and a computer program stored in the memory 602 and executable on the processor. The processor 601 and the memory 602 are electrically connected.

[0164] The processor 601 is the control center of the electronic device 600. It connects various parts of the electronic device 600 through various interfaces and lines. By running or loading software programs (computer programs) and / or modules stored in the memory 602, and calling data stored in the memory 602, it performs various functions of the electronic device 600 and processes data, thereby monitoring the electronic device 600 as a whole.

[0165] In this embodiment of the application, the processor 601 in the electronic device 600 loads the instructions corresponding to the processes of one or more applications into the memory 602 according to the following steps, and the processor 601 runs the applications stored in the memory 602 to realize the functions of any of the above method embodiments, such as the steps in any embodiment of the above hand-drawn flowchart recognition method, for example, the following steps may be included:

[0166] A hand-drawn flowchart is obtained; an object detection network is used to detect each element node and line segment key point in the hand-drawn flowchart, and the visual features of each element node and the key point features of the line segment key points are extracted; an initial graph network is constructed based on the visual features of each element node and the key point features of the line segment key points, and the initial features of the vertices and edges of the initial graph network are determined; based on the initial features of the vertices and the initial features of the edges, graph network feature extraction and classification processing are performed on the initial graph network to identify the connection relationship between the vertices and edges, and the connection relationship is mapped to the connection relationship between each element node in the hand-drawn flowchart; the recognition result of the hand-drawn flowchart is determined based on the connection relationship.

[0167] The steps in other embodiments of the above-described hand-drawn flowchart recognition method will not be repeated here. The specific implementation and beneficial effects of each operation / step executable by the processor can be found in the preceding method embodiments, and will not be repeated here.

[0168] Optional, such as Figure 11 As shown, the electronic device 600 also includes: a touch display screen 603, a radio frequency circuit 604, an audio circuit 605, an input unit 606, and a power supply 607. The processor 601 is electrically connected to the touch display screen 603, the radio frequency circuit 604, the audio circuit 605, the input unit 606, and the power supply 607. Those skilled in the art will understand that... Figure 11 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0169] The touch display screen 603 can be used to display a graphical user interface (GUI) and receive operation commands generated by the user interacting with the GUI. The touch display screen 603 may include a display panel and a touch panel. The display panel can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the electronic device. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. Optionally, the display panel can be configured using a liquid crystal display (LCD), an organic light-emitting diode (OLED), or other similar devices. The touch panel can be used to collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel), generate corresponding operation commands, and execute the corresponding program. The touch panel may cover the display panel. When the touch panel detects a touch operation on or near it, it transmits the data to the processor 601 to determine the type of touch event. Subsequently, the processor 601 provides corresponding visual output on the display panel based on the type of touch event. In this embodiment, the touch panel and display panel can be integrated into the touch display screen 603 to achieve input and output functions. However, in some embodiments, the touch panel and the touch display screen 603 can be implemented as two independent components to achieve input and output functions. That is, the touch display screen 603 can also be used as part of the input unit 606 to achieve input functions.

[0170] In this embodiment, the touch display screen 603 is used to present a graphical user interface and receive operation commands generated by the user interacting with the graphical user interface.

[0171] The radio frequency circuit 604 can be used to transmit and receive radio frequency signals to establish wireless communication with network devices or other electronic devices, and to transmit and receive signals with network devices or other electronic devices.

[0172] Audio circuit 605 can be used to provide an audio interface between a user and an electronic device via a speaker and a microphone. Audio circuit 605 can convert received audio data into electrical signals and transmit them to the speaker, where the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuit 605, converted back into audio data, and then processed by processor 601 before being transmitted via radio frequency circuit 604 to, for example, another electronic device, or output to memory 602 for further processing. Audio circuit 605 may also include an earphone jack to provide communication between peripheral headphones and electronic devices.

[0173] The input unit 606 can be used to receive input numbers, characters, or user characteristic information (such as fingerprints, iris, facial information, etc.), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.

[0174] Power supply 607 is used to supply power to various components of electronic device 600. Optionally, power supply 607 can be logically connected to processor 601 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. Power supply 607 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0175] although Figure 11 As not shown in the diagram, the electronic device 600 may also include a camera, sensor, wireless fidelity module, Bluetooth module, etc., which will not be described in detail here.

[0176] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0177] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0178] Therefore, embodiments of this application provide a computer-readable storage medium storing multiple computer programs that can be loaded by a processor to execute steps in any of the hand-drawn flowchart recognition methods provided in this application. For example, the computer program can execute steps in any embodiment of the above-described hand-drawn flowchart recognition method and achieve the beneficial effects of the corresponding embodiment. For example, the following steps can be executed:

[0179] A hand-drawn flowchart is obtained; an object detection network is used to detect each element node and line segment key point in the hand-drawn flowchart, and the visual features of each element node and the key point features of the line segment key points are extracted; an initial graph network is constructed based on the visual features of each element node and the key point features of the line segment key points, and the initial features of the vertices and edges of the initial graph network are determined; based on the initial features of the vertices and the initial features of the edges, graph network feature extraction and classification processing are performed on the initial graph network to identify the connection relationship between the vertices and edges, and the connection relationship is mapped to the connection relationship between each element node in the hand-drawn flowchart; the recognition result of the hand-drawn flowchart is determined based on the connection relationship.

[0180] The steps in other embodiments of the above-described hand-drawn flowchart recognition method will not be repeated here. For specific implementation details of each of the above operations, please refer to the preceding embodiments; they will not be repeated here.

[0181] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0182] Since the computer program stored in the storage medium can execute the steps in any of the hand-drawn flowchart recognition methods provided in the embodiments of this application, the beneficial effects that any of the hand-drawn flowchart recognition methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0183] The foregoing has provided a detailed description of a hand-drawn flowchart recognition method, apparatus, storage medium, and electronic device provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for recognizing hand-drawn flowcharts, characterized in that, include: Obtain hand-drawn flowcharts; An object detection network is used to detect each element node and line segment key point in the hand-drawn flowchart, and the visual features of each element node and the key point features of the line segment key point are extracted. Based on the visual features of each element node and the key point features of the line segment key points, each key point in the line segment key points and each element node are used as vertices of the initial graph network, and the line connecting any two vertices is used as the edge of the initial graph network to construct the initial graph network, and the initial features of the vertices and the initial features of the edges of the initial graph network are determined. Based on the initial features of the vertices and the initial features of the edges, graph network features are extracted and classified on the initial graph network to identify the connection relationships between the vertices and edges, and the connection relationships are mapped to the connection relationships between each element node in the hand-drawn flowchart. The recognition result of the hand-drawn flowchart is determined based on the connection relationship; The step of extracting and classifying graph network features from the initial graph network based on the initial features of the vertices and the initial features of the edges, in order to identify the connection relationships between the vertices and the edges, includes: Based on the initial features of the vertices and the initial features of the edges, a graph convolutional network based on a self-attention module is used to perform graph network feature extraction and de-densification processing on the initial graph network to obtain a sparse graph network. The graph convolutional network based on the self-attention module is used to perform graph network feature extraction processing on the features of vertices and edges in the sparse graph network to obtain enhanced features; The connection relationships between vertices and edges in the sparse graph network are classified based on the enhanced features to obtain the connection relationships between vertices and edges.

2. The method according to claim 1, characterized in that, The key points of the line segments are the key points in the line segments or arrowed line segments in the hand-drawn flowchart. The step of determining the initial features of the vertices and edges of the initial graph network includes: The key point features of each key point in the line segment key points and the visual features of each element node are used as the initial features of the corresponding vertices of the initial graph network. The initial features of the edge between any two vertices are determined based on the initial features of any two vertices.

3. The method according to claim 2, characterized in that, The step of determining the initial features of the edge between any two vertices based on the initial features of any two vertices includes: The initial features of each vertex in the initial graph network are subjected to nonlinear mapping to obtain the mapped features of each vertex. The mapping features of any two vertices are fused to obtain the initial features of the edge between any two vertices.

4. The method according to claim 1, characterized in that, The step of using an object detection network to detect each element node and line segment key point in the hand-drawn flowchart includes: Using the feature extraction module of the object detection network, multiple candidate regions and their corresponding visual feature maps are extracted from the hand-drawn flowchart. The classification and regression module of the target detection network is used to classify the visual feature map of the region to determine that multiple candidate regions correspond to various basic elements in the hand-drawn flowchart. The basic elements include element nodes and line segments or line segments with arrows. When the basic element is an element node, the classification and regression module is used to identify the position information of the element node in the hand-drawn flowchart. When the basic element is a line segment or a line segment with an arrow, the classification and regression module is used to identify the key points of the line segment or the line segment with an arrow and the position information of the key points of the line segment in the hand-drawn flowchart.

5. The method according to claim 4, characterized in that, The feature extraction module includes a feature extraction network and an interest region pooling network. The steps of extracting the visual features of each element node and the key point features of line segment key points include: The regional visual feature map of the candidate region corresponding to each element node is used as the visual feature of each element node. Using an interest region pooling network of an object detection network, key point features of the key points of the line segment or the line segment with arrows are extracted from the feature map based on the location information of the key points of the line segment.

6. The method according to claim 4, characterized in that, The step of using the classification and regression module to identify the key points of the line segment or the arrowed line segment and the position information of the key points of the line segment in the hand-drawn flowchart includes: The prediction network in the classification and regression module is used to predict a preset number of key points in the line segment or the arrowed line segment and the position coordinates of the key points in the candidate region corresponding to the line segment or the arrowed line segment. The classification and regression module is used to identify the coordinate offset of the candidate region corresponding to the line segment or the line segment with the arrow. Based on the coordinate offset, the original coordinates of the candidate area corresponding to the line segment or the arrowed line segment, and the position coordinates, the position information of the key point in the hand-drawn flowchart is determined, the preset number of key points are used as line segment key points, and the position information of the preset number of key points in the hand-drawn flowchart is used as the position information of the line segment key points in the hand-drawn flowchart.

7. The method according to claim 1, characterized in that, Also includes: The structured processing of each element node and the connection relationships between each element node is performed to obtain the structured recognition result corresponding to the hand-drawn flowchart.

8. A hand-drawn flowchart recognition device, characterized in that, include: The acquisition module is used to acquire hand-drawn flowcharts; The detection module is used to use an object detection network to detect each element node and line segment key point in the hand-drawn flowchart, and to extract the visual features of each element node and the key point features of the line segment key point. The graph construction module is used to construct an initial graph network based on the visual features of each element node and the key point features of the line segment key points, taking each key point in the line segment key points and each element node as the vertex of the initial graph network, and taking the line connecting any two vertices as the edge of the initial graph network, and determining the initial features of the vertices and the initial features of the edges of the initial graph network. The graph feature classification module is used to extract and classify graph network features of the initial graph network based on the initial features of the vertices and the initial features of the edges, so as to identify the connection relationship between the vertices and the edges and map the connection relationship to the connection relationship between each element node in the hand-drawn flowchart. The graph feature classification module is further configured to: based on the initial features of the vertices and the initial features of the edges, use a graph convolutional network based on a self-attention module to perform graph network feature extraction and de-densification processing on the initial graph network to obtain a sparse graph network; and use the graph convolutional network based on a self-attention module to perform graph network feature extraction processing on the features of vertices and edges in the sparse graph network to obtain enhanced features. The connection relationships between vertices and edges in the sparse graph network are classified according to the enhanced features to obtain the connection relationships between vertices and edges. The determination module is used to determine the recognition result of the hand-drawn flowchart based on the connection relationship.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted for loading by a processor to perform the steps of the method as described in any one of claims 1-7.

10. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the steps of the method as described in any one of claims 1-7 by invoking the computer program stored in the memory.

Images