Method for recognizing hand-drawn flowchart, electronic device, medium and computer product

Abstract

A method, electronic device, medium, and computer product for recognizing hand-drawn flowcharts are disclosed. The method includes: acquiring stroke features corresponding to strokes in the hand-drawn flowchart; inputting the stroke features into a graph neural network model to obtain the predicted categories of the strokes in the hand-drawn flowchart output by the graph neural network model, and symbol features obtained by clustering the stroke features, wherein the graph neural network model is trained from sample stroke features; and determining the category of the symbol features based on the predicted categories of the strokes in the hand-drawn flowchart. Implementing the technical solution provided by this invention can improve the recognition accuracy of symbols in hand-drawn flowcharts and effectively solve the problem of difficulty in recognizing hand-drawn flowcharts in the prior art.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a method, electronic device, medium, and computer product for recognizing hand-drawn flowcharts. Background Technology

[0002] With the rapid development of information technology, people are increasingly using flowcharts to describe business processes, algorithmic processes, and so on. Currently, flowcharts are mainly drawn using computer-aided design tools. However, in some application scenarios, such as meeting discussions and brainstorming, people are more accustomed to hand-drawing flowcharts. Compared with computer-aided design tools, hand-drawn flowcharts are more flexible, intuitive, and easier to express complex ideas and concepts.

[0003] However, the recognition and understanding of hand-drawn flowcharts still faces many challenges. Symbols in hand-drawn flowcharts, such as boxes and arrows, are irregular in shape, vary in size, and are placed arbitrarily. Furthermore, the drawing styles of different people vary significantly, making automatic flowchart recognition very difficult. Currently used symbol recognition methods mainly target printed symbols, resulting in lower accuracy in recognizing hand-drawn symbols. Summary of the Invention

[0004] This invention provides a method, electronic device, medium, and computer product for recognizing hand-drawn flowcharts, which can improve the recognition accuracy of symbols in hand-drawn flowcharts and effectively solve the problem of difficulty in recognizing hand-drawn flowcharts in the prior art.

[0005] In a first aspect of the present invention, a method for recognizing hand-drawn flowcharts is provided, comprising:

[0006] Obtain the stroke features corresponding to the strokes in the hand-drawn flowchart;

[0007] The stroke features are input into a graph neural network model to obtain the predicted category of the strokes in the hand-drawn flowchart output by the graph neural network model, as well as the symbol features obtained by clustering the stroke features, wherein the graph neural network model is trained from the sample stroke features.

[0008] The category of the symbol feature is determined based on the predicted category of the strokes in the hand-drawn flowchart.

[0009] Optionally, obtaining the stroke features corresponding to the strokes in the hand-drawn flowchart includes:

[0010] Extract strokes from a hand-drawn flowchart;

[0011] Obtain the geometric features of the stroke, which are used to characterize the positional information of the stroke;

[0012] Obtain the edge features between the strokes, and the edge features are used to characterize the spatiotemporal relationship between the strokes;

[0013] The geometric features and edge features corresponding to the strokes are used as stroke features.

[0014] Optionally, determining the category of the symbol feature based on the predicted category of the strokes in the hand-drawn flowchart includes:

[0015] Based on the Euclidean distance between the strokes, the strokes are merged into stroke clusters;

[0016] The category of the symbol feature is obtained based on the predicted category of the strokes in the stroke cluster.

[0017] Optionally, the graph neural network model includes a node classification branch and a node clustering branch, and the training process of the graph neural network model includes:

[0018] Input the stroke features of the sample into the node classification branch, and obtain the sample prediction category corresponding to the stroke features of the sample output by the node classification branch;

[0019] The sample stroke features are input into the node clustering branch to obtain the sample symbol features clustered from the sample stroke features output by the node clustering branch.

[0020] Obtain the first difference value of the predicted category of the sample, and obtain the second difference value of the symbolic feature of the sample;

[0021] The parameters of the graph neural network model are adjusted based on the first difference value and the second difference value.

[0022] Optionally, the node clustering branch includes a message-passing neural network and a mean-shift clustering layer. The step of inputting the sample stroke features into the node clustering branch and obtaining the sample symbol features clustered from the sample stroke features output by the node clustering branch includes:

[0023] The sample stroke features are input into the message passing neural network to obtain the sample high-dimensional features output by the message passing neural network. The sample high-dimensional features are used to characterize the shape and position relationship between sample stroke features.

[0024] The high-dimensional features of the sample are input into the mean-shift clustering layer to obtain the symbolic features of the sample output by the mean-shift clustering layer, wherein the mean-shift clustering layer is used to cluster the high-dimensional features of the sample.

[0025] Optionally, obtaining the second difference value of the sample symbol features includes:

[0026] Based on the high-dimensional features of the sample, the third difference value of the message passing neural network is calculated using the cross loss function;

[0027] Based on the sample symbol features, a node embedding function is used to calculate the fourth difference value of the mean-shift clustering layer. The node embedding function is used to optimize the clustering results of the mean-shift clustering layer to enhance the degree of aggregation between stroke features in the same sample symbol features and the degree of separation between stroke features in different sample symbol features.

[0028] The sum of the third difference value and the fourth difference value is taken as the second difference value.

[0029] Optionally, obtaining the first difference value of the predicted category of the sample includes:

[0030] If the sample set of the stroke features of the sample is less than or equal to the sample set threshold, then the weighted difference entropy loss function is used to calculate the first difference value of the predicted category of the sample.

[0031] If the sample set of the stroke features is larger than the sample set threshold, then the first difference value of the predicted category of the sample is calculated using the balanced Softmax loss function.

[0032] In a second aspect of the invention, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the hand-drawn flowchart recognition method as described above.

[0033] In a third aspect of the invention, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the hand-drawn flowchart recognition method as described above.

[0034] In a fourth aspect of the invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the hand-drawn flowchart recognition method as described above.

[0035] In summary, one or more technical solutions provided in this invention have at least the following technical effects or advantages:

[0036] By acquiring the stroke features of hand-drawn flowcharts and inputting them into a graph neural network model trained on sample stroke features, various symbols in the flowcharts can be effectively identified. The graph neural network model can not only output predicted stroke categories but also obtain symbol features through stroke feature clustering, thus improving the accuracy of hand-drawn flowchart recognition through dual output. Determining the category of symbol features based on the predicted stroke categories further enhances the reliability of recognition.

[0037] The above method fully leverages the advantages of graph neural networks in processing graphical data, enabling it to better capture the spatial relationships and topological structures between hand-drawn strokes, thereby improving the recognition ability of complex hand-drawn flowcharts. Furthermore, because it uses sample stroke features for training, this method exhibits good generalization ability, adapting to different hand-drawing styles and various complex flowchart structures.

[0038] Compared to traditional methods for recognizing printed symbols, this method can improve the recognition accuracy of hand-drawn flowcharts and effectively solves the problem of difficulty in recognizing hand-drawn flowcharts in existing technologies. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0040] Figure 1 This is a schematic diagram of a method for recognizing hand-drawn flowcharts provided in an embodiment of the present invention.

[0041] Figure 2 This is a feature extraction network framework diagram provided in an embodiment of the present invention.

[0042] Figure 3 This is a schematic diagram of the architecture of a graph neural network model provided in an embodiment of the present invention.

[0043] Figure 4 This is a hand-drawn flowchart recognition example diagram provided in an embodiment of the present invention.

[0044] Figure 5 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0046] Please refer to Figure 1 , Figure 1This is a schematic diagram of a method for recognizing hand-drawn flowcharts provided in an embodiment of this application. This method can be implemented using a computer program, a microcontroller, or run on a hand-drawn flowchart recognition system based on the von Neumann architecture. The computer program can be integrated into an application or run as a standalone utility application. Specifically, the method for recognizing hand-drawn flowcharts may include the following steps:

[0047] Step 101: Obtain the stroke features corresponding to the strokes in the hand-drawn flowchart.

[0048] In this context, a hand-drawn flowchart refers to a graphical representation created by a user through manual drawing, used to visually describe workflows, system structures, decision-making processes, or other complex logical relationships. In this embodiment, a hand-drawn flowchart can be understood as an online handwritten flowchart drawn in real time on a digital platform using an electronic handwriting device. These flowcharts consist of a series of geometric symbols (such as rectangles, rhombuses, circles, etc.), connecting lines (such as arrows), and handwritten text annotations, forming a two-dimensional spatial structure.

[0049] Correspondingly, stroke features refer to the digital representation of a single stroke in a hand-drawn flowchart. Stroke features comprehensively capture the geometric, temporal, and dynamic information of a stroke. In this embodiment, the stroke features corresponding to a stroke can be understood as a set of multi-dimensional vectors. These vectors comprehensively encode information such as the shape, direction, pressure changes, and spatial relationship with other strokes of the stroke.

[0050] Specifically, stroke features can include the spatial shape and positional information of the stroke, such as the coordinates of the starting and ending points, and the relative coordinate sequence of sampling points on the trajectory; temporal information reflecting the drawing process, such as the drawing duration and the timestamp sequence of each point; information describing dynamic changes, such as changes in pen tip pressure and drawing speed; and the relationship between the stroke and other surrounding strokes, such as relative position, direction, and distance. In addition, it includes statistical descriptions of the overall characteristics of the stroke, such as average speed, average pressure, and direction histograms.

[0051] Based on the above embodiments, as an optional embodiment, step 101, the step of obtaining the stroke features corresponding to the strokes in the hand-drawn flowchart, may further include the following steps:

[0052] Step 201: Obtain the strokes from the hand-drawn flowchart.

[0053] In this embodiment of the application, in order to effectively represent and process the structure of hand-drawn flowcharts, each flowchart is modeled as a graph structure with attributes, denoted as G=(V,E). This graph structure representation method can intuitively and efficiently capture the spatial relationships and connection information between strokes in the flowchart.

[0054] Specifically, in graph G, the node set V represents each stroke in the flowchart. Each node v i ∈V corresponds to an independent stroke in the flowchart. The above representation transforms the basic building blocks of the flowchart into nodes in the graph structure, laying the foundation for subsequent analysis and processing.

[0055] The edge set E represents the connection relationships between strokes. During the initial graph construction, a spatial distance-based method was used to determine the existence of edges. For each stroke v in the flowchart... i First, calculate its Euclidean distance from all other strokes. Then, choose the one with v. i The closest There are 100 strokes. The nearest strokes are considered the spatial neighbors of s, denoted as N. s (s). The above method can effectively capture the local spatial relationships between strokes, while also limiting the complexity of the graph and avoiding the computational burden that fully connected graphs may bring.

[0056] After determining the spatial neighbors of each stroke, we begin adding elements to the edge set E of graph G. For each stroke v in the flowchart... i , will v i Its spatial neighbors The connections between each stroke in the graph are added to the edge set E. Specifically, for v... i ∈V and v j ∈N(v i Add edge e to E. ij =(v i ,v j These edges reflect the spatial structure information of the flowchart, providing input for subsequent graph neural network processing.

[0057] To more comprehensively describe the characteristics of nodes and edges, this invention uses real-valued eigenvectors. For each node v i Using a feature vector f(v) i This is used to describe the attributes of the stroke, such as its geometric features and temporal information. Similarly, for each edge e... ij A feature vector can be used to describe the relationship between two strokes, such as relative position, relative spatial position, and relative temporal relationship.

[0058] Using the above method, a hand-drawn flowchart can be transformed into a graph structure with rich attribute information. This representation not only preserves the spatial structure information of the flowchart but also encodes detailed features of strokes and stroke relationships through feature vectors. This provides an ideal input format for subsequent graph neural network processing, enabling the model to better understand and recognize the structure and content of the hand-drawn flowchart.

[0059] Step 202: Obtain the geometric features of the strokes. The geometric features are used to characterize the positional information of the strokes.

[0060] In this context, geometric features refer to the digital representation of the spatial attributes and shape characteristics of individual strokes in a hand-drawn flowchart. In this embodiment, geometric features can be understood as a set of multi-dimensional vectors that comprehensively encode information such as the shape, direction, curvature, and spatial relationship of the stroke with other strokes. Specifically, these features include the relative coordinate sequence of the stroke, changes in direction, changes in curvature, stroke length, and the relative positions of the start and end points. Geometric features are used to accurately characterize the positional information and shape characteristics of strokes, providing crucial spatial information input for subsequent symbol segmentation and recognition tasks. This enables the model to effectively distinguish and recognize different types of flowchart symbols, even if these symbols may have similar appearances or complex structures.

[0061] After acquiring the strokes in the hand-drawn flowchart, this embodiment of the invention further extracts the geometric features of each stroke. These features are used to accurately characterize the positional information of the strokes. Acquiring the geometric features of the strokes can capture their spatial attributes, laying the foundation for subsequent symbol segmentation and recognition tasks.

[0062] Specifically, to ensure the model effectively captures the core geometric features of the strokes while avoiding the influence of changes in absolute position, each stroke is preprocessed. Specifically, each stroke is transformed into a sequence containing a 3D vector. This processing method is achieved by calculating the first-order difference of the trajectory point coordinates. The transformed sequence can be represented as:

[0063] ;

[0064] In the formula, , , represent the changes in the x and y coordinates of adjacent trajectory points, respectively, and m is the length of stroke s. Indicates the state of the pen. This indicates that the stroke was placed in that store; otherwise... This representation method not only preserves the shape information of the strokes, but also achieves translation invariance through relative coordinates, allowing the model to better focus on the geometric features of the strokes themselves without being affected by their absolute position in the flowchart.

[0065] Please refer to Figure 2 , Figure 2This illustration shows a feature extraction network framework provided by an embodiment of the present invention. To further extract high-level geometric features of strokes, this embodiment employs a feature extraction network based on Recurrent Neural Networks (RNNs). This network consists of multiple stacked bidirectional RNN layers, each containing both forward and backward RNN units. This design allows the network to fully consider contextual information when processing sequential data, thereby providing a more comprehensive understanding of the temporal and spatial characteristics of stroke trajectories.

[0066] Furthermore, the forward units process the forward information of the time series, while the backward units ingest the backward information, enabling the model to more comprehensively understand the contextual information of the stroke trajectory sequence data. After processing through all RNN layers, the output vector from the final RNN layer is fed into a global max pooling layer, which reduces the dimensionality of the output features, retaining the most salient features while discarding redundant information. Finally, the result after the pooling layer is passed to a fully connected layer for further feature compression and integration.

[0067] Through the above-described methods and steps, this invention can effectively extract highly generalized and information-rich geometric features from raw stroke data. These geometric features not only include the shape and direction information of the strokes but also preserve the dynamic features generated during the stroke drawing process. This multi-dimensional feature representation greatly enhances the model's ability to recognize various strokes in hand-drawn flowcharts.

[0068] Meanwhile, by employing relative coordinates and an RNN structure, the extracted features exhibit good translation invariance and the ability to capture long-term dependencies. This allows the model to maintain stable performance when processing flowcharts of varying sizes, positions, and complexities. Ultimately, these extracted geometric features serve as feature vectors for each node in the graph structure, providing rich and effective input information for subsequent graph neural network processing, thereby significantly improving the accuracy and robustness of the entire hand-drawn flowchart recognition system.

[0069] Step 203: Obtain the edge features between strokes. The edge features are used to characterize the spatiotemporal relationship between strokes.

[0070] In this context, edge features refer to the digital representation describing the relationship between two strokes in a hand-drawn flowchart. In this embodiment of the invention, an edge feature can be understood as a multi-dimensional vector that comprehensively encodes the spatial relationship and temporal information between two strokes. Specifically, these features include the relative positional information between the two strokes. Edge features are used to accurately characterize the spatiotemporal relationship between strokes, providing crucial contextual information for graph neural networks. This enables the model to effectively understand the logical connections and structural relationships between symbols in the flowchart, thereby improving the accuracy of symbol segmentation and recognition, especially when dealing with complex flowchart structures.

[0071] Specifically, firstly, the trajectory points of the strokes are resampled using an equal-interval interpolation method to ensure they have the same length. Then, the difference in coordinates between the sampled points of two strokes is calculated and concatenated to obtain the position embedding (PE). The position embedding can be represented as:

[0072] ;

[0073] Besides spatial relationships, the relative order of strokes is also important temporal information, which is described using time embedding (TE):

[0074] ;

[0075] In the formula, It is a time factor that controls the importance of time neighbors. It is the number of factors. It's a concatenation operator. The final edge feature is the position embedding. and time embedding The concatenation is represented as:

[0076] ;

[0077] By employing relative coordinates and temporal embedding, the extracted edge features exhibit good translation invariance and the ability to capture long-term dependencies, enabling the model to maintain stable performance when processing flowcharts of different sizes, layouts, and complexities.

[0078] Step 204: Use the geometric features and edge features corresponding to the strokes as stroke features.

[0079] Step 102: Input the stroke features into the graph neural network model to obtain the predicted categories of strokes in the hand-drawn flowchart output by the graph neural network model, as well as the symbol features obtained by clustering the stroke features. The graph neural network model is trained from the sample stroke features.

[0080] Graph neural network models refer to a class of deep learning models specifically designed for processing and analyzing graph-structured data. In this embodiment of the invention, a graph neural network model can be understood as a complex network structure based on message-passing neural networks (MPNNs). This model includes node classification branches and node clustering branches, as well as a learnable mean-shift clustering layer.

[0081] Specifically, the model takes a graph structure as input, with strokes as nodes and relationships between strokes as edges. Through multi-layer message passing and feature aggregation operations, it learns a high-level representation of strokes and their relationships. The graph neural network model transforms the raw stroke and edge features in hand-drawn flowcharts into higher-level representations, while outputting predicted stroke categories and symbol features. It effectively captures the complex spatial and logical relationships in flowcharts, providing a crucial computational foundation for accurately recognizing symbols and their structures in hand-drawn flowcharts, thereby significantly improving the performance and adaptability of the entire hand-drawn flowchart recognition system.

[0082] In this context, the predicted category of a stroke refers to the model's prediction of the symbol type to which each individual stroke in a hand-drawn flowchart belongs. In this embodiment, the predicted category of a stroke can be understood as a probability distribution vector, where each element corresponds to a category in a predefined set of symbol categories, and its value represents the probability that the stroke belongs to the corresponding category. For example, for a flowchart symbol set containing categories such as "rectangle," "circle," "diamond," and "arrow," the predicted category of a stroke might be represented as [0.7, 0.1, 0.05, 0.15], meaning that the stroke has a 70% probability of belonging to a rectangle, a 10% probability of belonging to a circle, and so on. The predicted category of a stroke is used to initially determine the possible symbol type to which each stroke belongs.

[0083] Correspondingly, symbolic features refer to the abstract representation of a complete symbol in a hand-drawn flowchart. In this embodiment of the invention, symbolic features can be understood as a set of multi-dimensional vectors obtained by mean-shift clustering, output by the node clustering branch of the graph neural network model. Each vector represents a complete symbol, which integrates the feature information of all the strokes that constitute the symbol.

[0084] Specifically, the aforementioned features include the geometric attributes of the symbols (such as shape, size, and orientation) and their relative positional relationships with other symbols. Symbol features are used to accurately characterize each individual symbol in the flowchart, providing crucial input for subsequent symbol recognition and flowchart structure analysis. This enables the system to integrate the original stroke-level information into higher-level symbol-level information.

[0085] For further details, please refer to Figure 3 , Figure 3 A schematic diagram of the architecture of a graph neural network model is shown, such as... Figure 3As shown, when stroke features are input into a graph neural network model, the node classification branch processes the input graph structure through multiple MPNN layers, outputting a predicted class probability distribution for each stroke node, representing the probability that each stroke belongs to each possible symbol class. For example, for a specific stroke, the model might output that it has a 70% probability of being part of a rectangle, a 20% probability of being part of a circle, and a 10% probability of belonging to other classes.

[0086] Simultaneously, the node clustering branch also processes the input graph structure, but its goal is to generate an embedding representation for each stroke node. At the end of the node clustering branch, a pre-trained mean-shift clustering layer is used to cluster these embeddings, grouping strokes belonging to the same symbol together to obtain symbol features. Symbol features are high-level abstract representations of the complete symbol, integrating feature information from all the strokes that constitute the symbol, including geometric attributes, topological structure, and relative positional relationships.

[0087] Step 103: Determine the category of symbol features based on the predicted categories of strokes in the hand-drawn flowchart.

[0088] In this invention, the category of symbol features refers to the final recognition result of each complete symbol in the hand-drawn flowchart. In this embodiment, the category of symbol features can be understood as a discrete label, indicating that the symbol belongs to a specific type within a predefined symbol set, such as "rectangle," "circle," "diamond," or "arrow." The category of symbol features is used to ultimately determine the specific type of each symbol in the flowchart and is the final output of the entire hand-drawn flowchart recognition process, providing crucial information for subsequent flowchart understanding and analysis.

[0089] Based on the above embodiments, as an optional embodiment, step 103, which determines the category of symbol features based on the predicted category of strokes in the hand-drawn flowchart, may further include the following steps:

[0090] Step 301: Based on the Euclidean distance between strokes, merge strokes into stroke clusters.

[0091] In the process of determining the category of symbol features, embodiments of the present invention can more accurately capture the overall structure of the symbol by grouping spatially similar strokes together, thereby improving the accuracy of the final symbol category recognition.

[0092] Specifically, we first calculate the Euclidean distance between each stroke in the hand-drawn flowchart and all other strokes. Then, we set a distance threshold θ. If the Euclidean distance between two strokes is less than this threshold, we consider the two strokes to belong to the same stroke cluster.

[0093] The merging process employs an iterative approach. Initially, each stroke is treated as an independent stroke cluster. Then, for each stroke 1, all strokes less than θ in distance are searched. If a stroke 2 that satisfies the condition is found, the stroke cluster containing stroke 1 is merged with the stroke cluster containing stroke 2. This process is repeated until no further merging occurs.

[0094] Step 302: Based on the predicted category of strokes in the stroke cluster, obtain the category of symbol features.

[0095] Specifically, for each stroke cluster obtained by merging using Euclidean distance, consider the predicted categories of all strokes within the cluster. A voting mechanism can be used to determine the final category of the symbol feature. First, count the predicted categories of all strokes belonging to the stroke cluster. The predicted category of each stroke is a probability distribution, representing the probability that the stroke belongs to each possible category. This yields a comprehensive category probability distribution. Finally, select the category with the highest probability in this comprehensive probability distribution as the final category of the symbol feature.

[0096] Please refer to Figure 4 , Figure 4 The illustration shows a sample diagram of hand-drawn flowchart recognition provided by an embodiment of the present invention.

[0097] The above embodiments illustrate the application process of graph neural network models. Based on the above embodiments, the training process of graph neural network models will be described below. Specifically, this process may include the following steps:

[0098] Step 401: Input the sample stroke features into the node classification branch and obtain the sample prediction category corresponding to the sample stroke features output by the node classification branch.

[0099] The node classification branch refers to a sub-network structure in a graph neural network model specifically designed for stroke classification tasks. In this embodiment, the node classification branch can be understood as a complex network structure composed of multiple layers of MPNN. This branch receives stroke features as input, and through a series of feature extraction and transformation operations, ultimately outputs the probability distribution of each stroke belonging to various predefined symbol categories. The node classification branch is used to identify and classify various strokes in a hand-drawn flowchart, predicting the possible symbol type for each stroke. By learning the geometric features and contextual information of the strokes, it generates high-level feature representations that reflect the stroke type, providing basic information for subsequent symbol recognition, thereby improving the classification accuracy and robustness of the entire hand-drawn flowchart recognition system.

[0100] Specifically, the preprocessed sample stroke features are first input into the node classification branch of the graph neural network model. This node classification branch consists of multiple MPNN layers, which can effectively capture the spatial and logical relationships between strokes. The MPNN layers gradually extract and integrate the high-level feature representations of the strokes through iterative message passing and aggregation operations. In this process, each stroke node not only considers its own features but also receives information from neighboring strokes, thereby obtaining a more comprehensive contextual understanding. After processing by multiple layers of MPNN, the node classification branch outputs a predicted class probability distribution for each sample stroke, representing the probability that the stroke belongs to each possible symbol class.

[0101] Through the steps described above, the graph neural network model learns the mapping relationship from original stroke features to symbol categories, improving its ability to recognize various stroke types. This enables it to handle the complex and diverse stroke forms in hand-drawn flowcharts and adapt to different users' drawing styles. Furthermore, by considering the relationships between strokes, the graph neural network model can better understand the role of strokes within the overall symbol, improving classification accuracy.

[0102] Step 402: Input the sample stroke features into the node clustering branch, and obtain the sample symbol features obtained by clustering the sample stroke features from the node clustering branch.

[0103] The node clustering branch refers to a sub-network structure in a graph neural network model specifically designed for symbol segmentation tasks. In this embodiment, the node clustering branch can be understood as a complex network structure consisting of a multi-layer MPNN and a learnable mean-shift clustering layer. This branch receives stroke features as input, and through a series of feature extraction and transformation operations, ultimately outputs the embedding representation of each stroke and the clustering result. The node clustering branch is used to group strokes belonging to the same symbol together, achieving symbol segmentation in hand-drawn flowcharts. By learning the spatial and logical relationships between strokes, it generates high-level feature representations that reflect the symbol structure, providing crucial intermediate results for subsequent symbol recognition.

[0104] Correspondingly, the mean-shift clustering layer refers to a learnable neural network layer designed based on the mean-shift algorithm principle. In this embodiment of the invention, the mean-shift clustering layer can be understood as a special network layer at the end of the node clustering branch. It receives the embedded representation of strokes as input and outputs the symbol features of the clustered samples through iterative density estimation and centroid update processes.

[0105] Furthermore, the core operations of the mean-shift clustering layer include density estimation using a Gaussian kernel and moving data points along the density gradient direction. The mean-shift clustering layer effectively groups strokes belonging to the same symbol together, achieving accurate symbol segmentation. It can adaptively handle hand-drawn symbols of varying complexity and structure, without requiring a pre-specified number of clusters. By using preset clustering parameters, it improves the model's flexibility and accuracy in handling various flowchart styles. The introduction of the mean-shift clustering layer significantly enhances the performance of node clustering branches, providing high-quality symbol-level features for subsequent symbol recognition.

[0106] Based on the above embodiments, step 402 may further include the following steps:

[0107] Step 501: Input the sample stroke features into the message passing neural network to obtain the sample high-dimensional features output by the message passing neural network. The sample high-dimensional features are used to characterize the shape and position relationship between sample stroke features.

[0108] Specifically, the preprocessed sample stroke features are first used as input to the MPNN. The MPNN then extracts and integrates high-level feature representations of the strokes through multiple rounds of message passing and aggregation operations. In each round, each stroke node considers not only its own features but also information from neighboring strokes. This message passing mechanism enables the MPNN to effectively capture the spatial and logical relationships between strokes.

[0109] Simultaneously, MPNN integrates the absolute positional information of strokes, including the starting and ending coordinates of the strokes and the bounding box coordinates, which provides important spatial context for the model. After processing by multiple layers of MPNN, the network outputs high-dimensional features of the samples, which are representations of the original sample stroke features in high-dimensional space. The high-dimensional features of the samples not only contain the geometric information of individual strokes but also encode the relative positions, orientations, and structural relationships between strokes.

[0110] By employing the above methods, the complex spatial structure and interdependencies between strokes in hand-drawn flowcharts can be captured, improving the model's ability to understand complex symbols. Secondly, through the message passing mechanism, the model can consider a wider range of contextual information, thus more accurately identifying the role of strokes within the entire symbol. Finally, the generated high-dimensional features of the samples provide more discriminative input for subsequent clustering operations, helping to improve the accuracy of symbol segmentation. This enables the model to extract more abstract and information-rich representations from the original stroke-level features, laying the foundation for symbol segmentation and recognition in hand-drawn flowcharts, thereby improving the system's performance and adaptability when handling various complex flowcharts.

[0111] Step 502: Input the high-dimensional features of the sample into the mean-shift clustering layer to obtain the symbolic features of the sample output by the mean-shift clustering layer. The mean-shift clustering layer is used to cluster the high-dimensional features of the sample.

[0112] Specifically, the high-dimensional features of the samples generated by the message-passing neural network are first input into the mean-shift clustering layer. The mean-shift clustering layer is a trainable neural network layer designed based on the mean-shift algorithm. In this layer, for each sample's high-dimensional features, an isotropic multivariate Gaussian kernel is used to estimate the point density, employing the Parsons window method. The Gaussian kernel is defined as...

[0113] ;

[0114] In the formula, σ controls the bandwidth of the Gaussian kernel. Based on density estimation, the mean-shift clustering algorithm moves data points in the direction of the density pattern at each iteration.

[0115] Furthermore, based on density estimation, the mean-shift clustering algorithm moves in the direction of the density pattern at each iteration, corresponding to the direction of maximum density increase. Specifically, for each stroke feature... Moving along the gradient direction results in updated stroke features. This operation has a total of Layers, each layer is represented as:

[0116] ;

[0117] Mean-shift clustering layers adjust the cluster center points step by step, so that the final clusters can accurately reflect the symbol distribution in the flowchart, thus providing strong support for subsequent reasoning.

[0118] Step 403: Obtain the first difference value of the predicted category of the sample, and obtain the second difference value of the symbol feature of the sample.

[0119] The first difference value refers to the measure of the difference between the model's prediction results and the true labels in the stroke classification task. In this embodiment of the invention, the first difference value can be understood as the cross-entropy loss between the sample predicted class probability distribution output by the node classification branch and the true class label. It is used to quantify the accuracy of the model in recognizing and classifying various strokes in hand-drawn flowcharts, providing direct numerical guidance for optimizing the model's classification performance.

[0120] Correspondingly, the second difference value refers to a measure of the difference between the feature representation generated by the model in the symbol segmentation task and the ideal feature representation. In this embodiment of the invention, the second difference value can be understood as the distance or similarity between the sample symbol features output by the node clustering branch and the target symbol features. It can take the form of contrastive loss to evaluate the model's effect in aggregating strokes belonging to the same symbol.

[0121] These two difference values ​​are used together to comprehensively evaluate the model's performance on the two key tasks of stroke classification and symbol segmentation, providing important feedback information for parameter adjustment and performance optimization during model training, thereby improving the accuracy and robustness of the entire hand-drawn flowchart recognition system.

[0122] Based on the above embodiments, as an optional embodiment, step 403, which involves obtaining the first difference value of the predicted sample category, may further include the following steps:

[0123] Step 601: If the sample set of stroke features is less than or equal to the sample set threshold, then the weighted difference entropy loss function is used to calculate the first difference value of the predicted category of the sample.

[0124] Specifically, because certain symbol categories in small datasets may appear with low frequency, the model tends to ignore these rare categories during training. To address this issue, this embodiment of the invention employs a weighted cross-entropy loss function, assigning different weights to different categories, enabling the model to pay more attention to categories with fewer samples, thereby improving the overall classification accuracy.

[0125] Specifically, first, it is determined whether the sample set size of the stroke features is less than or equal to a preset sample set threshold. If the condition is met, the first difference value is calculated using the following weighted cross-entropy loss function:

[0126] ;

[0127] in, For the number of flowcharts, The number of strokes in a single flowchart. Indicate category The weighting coefficients represent the frequency of each category in the training set. Categories with lower frequencies are given higher weights to ensure the model pays more attention to them. for The predicted class probability is obtained by softmax calculation of the output of the stroke classification branch.

[0128] In summary, using the weighted cross-entropy loss function to calculate the first difference value can effectively alleviate the class imbalance problem, ensuring that the model pays sufficient attention to each class during training, avoiding overfitting to common classes and neglecting rare classes. Secondly, by adjusting the weights of different classes, the model can better learn the features of rare classes, improving its ability to recognize these classes. Finally, the above methods improve the model's generalization ability on small datasets, enabling it to maintain high recognition accuracy when faced with hand-drawn flowcharts of varying complexity and class distribution.

[0129] Step 602: If the sample set of stroke features is greater than the sample set threshold, then the first difference value of the predicted class of the sample is calculated using the balanced Softmax loss function.

[0130] Specifically, since large-scale datasets typically contain more diverse and complex samples, while the problem of class imbalance may still exist, the greater challenge lies in how to effectively utilize rich data to improve the model's generalization ability and its ability to identify tail classes. This invention employs a balanced Softmax loss function, introducing an adjustment factor for class frequency, which not only alleviates the class imbalance problem but also improves the model's sensitivity to tail classes while maintaining a high recognition rate for common classes.

[0131] Specifically, first, it is determined whether the sample size of the stroke features is greater than a preset sample threshold. If the condition is met, the first difference value is calculated using the following balanced softmax loss function:

[0132] ;

[0133] In the formula, For the number of flowcharts, This refers to the number of strokes in a single flowchart. It refers to the number of symbols in the flowchart. Indicate category The weighting coefficients represent the frequency of that category in the training set. This represents the output of the node classification branch.

[0134] In summary, employing the balanced softmax loss function significantly improves the model's ability to identify tail-end categories while maintaining high accuracy in common categories. By considering category frequency in the loss calculation, the model is encouraged to focus more on categories with fewer samples, thus improving overall classification balance. Secondly, this method better utilizes the rich information in large-scale datasets, enhancing the model's generalization ability. Since the loss function directly affects the model's raw output, it allows for finer adjustments to the model's sensitivity to different categories, resulting in more stable performance when dealing with complex hand-drawn flowcharts. Finally, the balanced softmax loss function is computationally efficient, suitable for training on large-scale datasets, and effectively utilizes modern hardware resources to accelerate the model's convergence process.

[0135] Based on the above embodiments, as an optional embodiment, step 403, which involves obtaining the second difference value of the sample symbol features, may further include the following steps:

[0136] Step 701: Based on the high-dimensional features of the samples, the third difference value of the message passing neural network is calculated using the cross loss function.

[0137] In the process of obtaining the second difference value of the sample symbol features, the third difference value of the message passing neural network is calculated based on the high-dimensional features of the samples using the cross loss function. This can evaluate the performance of the message passing neural network in extracting and integrating stroke features, and ensure that the generated high-dimensional features can effectively capture the spatial and logical relationships between strokes.

[0138] Specifically, the high-dimensional features of the samples output by the message-passing neural network are first obtained. These high-dimensional features are representations of the original sample stroke features after multiple rounds of message passing and aggregation operations, containing rich contextual information. Then, a cross-loss function can be used to calculate the difference between these high-dimensional features and the ideal feature representation. The cross-loss function is designed to make stroke features belonging to the same symbol closer together in the high-dimensional space, while features of different symbols are further apart.

[0139] Its calculation formula can be expressed as:

[0140] ;

[0141] In the formula, For the number of flowcharts, for The true labels are set to {−1, 1}, where 1 indicates that the symbols being connected are the same and -1 indicates that the symbols being connected are different. These labels can be inferred during training. for The predicted probability category. This is the edge feature set corresponding to the hand-drawn flowchart.

[0142] Step 702: Based on the sample symbol features, the fourth difference value of the mean-drift clustering layer is calculated using the node embedding function. The node embedding function is used to optimize the clustering results of the mean-drift clustering layer to enhance the degree of aggregation between stroke features in the same sample symbol features and the degree of separation between stroke features in different sample symbol features.

[0143] Specifically, firstly, the sample sign features output by the mean-drift clustering layer are obtained. These sign features are high-level representations after clustering, containing structural information at the sign level. Then, a node embedding function can be used to calculate the fourth difference value. The node embedding function is designed to optimize the clustering results, and its calculation process is continuously optimized for each iteration of the mean-drift clustering layer. Specific calculation formulas may include:

[0144] (1) Average embedding function of symbol classes:

[0145] ;

[0146] In the formula, Let represent the center embedding vector of symbol c, which is the average value of the embeddings of all strokes within symbol c. This represents the embedding vector of stroke i. Let represent the set of strokes in symbol c, containing all strokes i that belong to symbol c. This indicates the number of strokes in the symbol c.

[0147] (2) Intra-class compact loss function:

[0148] ;

[0149] In the formula, The intra-class compactness loss measures the distance between the stroke embedding and the symbol center embedding within the same symbol. The square of the L2 norm represents the distance between the center embedding and the stroke embedding of the symbol, and C represents the number of hand-drawn flowcharts. Indicates the number of strokes in a symbol.

[0150] (3) Inter-class separation loss:

[0151] ;

[0152] In the formula, This represents the inter-class separation loss, which measures the center embeddings among different symbols. and The distance between them This represents a predefined inter-class distance threshold to ensure sufficient separation between different symbol center embeddings, [x]. + This represents the Hinge function, if the distance is less than 2. If a penalty is imposed, then a punishment will be imposed; otherwise, no punishment will be imposed.

[0153] (4) Regularization function:

[0154] ;

[0155] In the formula, This represents the weight hyperparameter of the regularization term, used to control the strength of the regularization term. The central embedding of the symbol c The squared L2 norm is used to limit the size of the embedding vector.

[0156] For the sake of simplicity, the total loss is expressed as:

[0157]

[0158] In the formula, This represents the total number of iterations in the mean-shift clustering layer, with the goal of simultaneously optimizing the compactness of strokes within the symbol (through...). ) and the separability between symbols (through ), and apply regularization to stabilize the size of the embedding.

[0159] Step 703: Take the sum of the third and fourth difference values ​​as the second difference value.

[0160] Step 404: Adjust the parameters of the graph neural network model based on the first difference value and the second difference value.

[0161] Specifically, this process involves parameters for all components, including node classification branches, node clustering branches, message-passing neural network layers, and mean-shift clustering layers. Optimization algorithms can be used to update the model parameters based on the calculated differences.

[0162] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, communications interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a method for recognizing hand-drawn flowcharts. This method includes: acquiring stroke features corresponding to strokes in the hand-drawn flowchart; inputting the stroke features into a graph neural network model to obtain the predicted categories of strokes in the hand-drawn flowchart output by the graph neural network model, and symbol features obtained by clustering the stroke features, wherein the graph neural network model is trained from sample stroke features; and determining the category of the symbol features based on the predicted categories of strokes in the hand-drawn flowchart.

[0163] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0164] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the hand-drawn flowchart recognition method provided by the above methods. The method includes: obtaining stroke features corresponding to strokes in the hand-drawn flowchart; inputting the stroke features into a graph neural network model to obtain the predicted category of the strokes in the hand-drawn flowchart output by the graph neural network model, and symbol features obtained by clustering the stroke features, wherein the graph neural network model is trained from sample stroke features; and determining the category of the symbol features based on the predicted category of the strokes in the hand-drawn flowchart.

[0165] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a method for recognizing hand-drawn flowcharts provided by the methods described above. This method includes: acquiring stroke features corresponding to strokes in the hand-drawn flowchart; inputting the stroke features into a graph neural network model to acquire the predicted categories of strokes in the hand-drawn flowchart output by the graph neural network model, and symbol features obtained by clustering the stroke features, wherein the graph neural network model is trained from sample stroke features; and determining the category of the symbol features based on the predicted categories of strokes in the hand-drawn flowchart.

[0166] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0167] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0168] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for recognizing hand-drawn flowcharts, characterized in that, include: Obtain the stroke features corresponding to the strokes in the hand-drawn flowchart; The stroke features are input into a graph neural network model to obtain the predicted category of the strokes in the hand-drawn flowchart output by the graph neural network model, as well as the symbol features obtained by clustering the stroke features, wherein the graph neural network model is trained from the sample stroke features. Based on the predicted categories of strokes in the hand-drawn flowchart, the category of the symbol feature is determined; The process of obtaining the stroke features corresponding to the strokes in the hand-drawn flowchart includes: Extract strokes from a hand-drawn flowchart; The geometric features of the strokes are obtained, and these geometric features are used to characterize the positional information of the strokes. Geometric features refer to the digital representation of the spatial attributes and shape characteristics of a single stroke in a hand-drawn flowchart. Each stroke is preprocessed, transforming it into a sequence containing a 3D vector. This processing method is achieved by calculating the first-order difference of the trajectory point coordinates. The transformed sequence can be represented as follows: ； In the formula, , , represent the changes in x and y coordinates of adjacent trajectory points, respectively, and m is the length of stroke s. Indicates the state of the pen. This indicates that the stroke falls at that point; otherwise... Geometric features are extracted using a recurrent neural network (RNN). This network consists of multiple stacked bidirectional RNN layers, each containing both forward and backward RNN units. The forward units process the forward information of the time series, while the backward units capture the backward information, enabling the model to more comprehensively understand the contextual information of the stroke trajectory sequence data. After processing by all RNN layers, the output vector from the final RNN layer is fed into a global max pooling layer. The result of the pooling layer is then passed to a fully connected layer for further feature compression and integration to obtain the geometric features. Obtain the edge features between the strokes, which are used to characterize the spatiotemporal relationship between the strokes; the edge features are used to characterize the spatiotemporal relationship between the strokes, providing contextual information for the graph neural network; The trajectory points of the strokes are resampled using an equal-interval interpolation method to ensure they have the same length. Then, the difference in coordinates between the sampled points of two strokes is calculated and concatenated to obtain the position embedding (PE). The position embedding is represented as: ; The relative order of strokes is described using temporal embedding (TE): ; In the formula, It is a time factor that controls the importance of time neighbors. It is the number of factors. It is a concatenation operator; Edge features are position embeddings and time embedding The concatenation is represented as: The geometric features and edge features corresponding to the strokes are used as stroke features. The graph neural network model includes a node classification branch and a node clustering branch. The node clustering branch includes a message-passing neural network and a mean-shift clustering layer. The training process of the graph neural network model includes: Input the stroke features of the sample into the node classification branch, and obtain the sample prediction category corresponding to the stroke features of the sample output by the node classification branch; The sample stroke features are input into the message passing neural network to obtain the sample high-dimensional features output by the message passing neural network. The sample high-dimensional features are used to characterize the shape and position relationship between sample stroke features. The high-dimensional features of the sample are input into the mean-shift clustering layer to obtain the sample symbol features output by the mean-shift clustering layer, wherein the mean-shift clustering layer is used to cluster the high-dimensional features of the sample; Obtain the first difference value of the predicted category of the sample, and obtain the second difference value of the symbolic feature of the sample; Based on the first difference value and the second difference value, the parameters of the graph neural network model are adjusted; wherein, the second difference value is the sum of the third difference value and the fourth difference value; the second difference value is determined in the following way: ； In the formula, Let represent the center embedding vector of symbol c, which is the average value of the embeddings of all strokes within symbol c. Let i represent the embedding vector of stroke i. Let represent the set of strokes in symbol c, containing all strokes i that belong to symbol c. This indicates the number of strokes in the symbol 'c'; ； In the formula, The intra-class compactness loss measures the distance between the stroke embedding and the symbol center embedding within the same symbol. Let represent the square of the L2 norm, represent the distance between the center embedding and the stroke embedding of the symbol, and C represent the number of hand-drawn flowcharts. Indicates the number of strokes in a symbol; ； In the formula, This represents the inter-class separation loss, which measures the center embeddings among different symbols. and The distance between them This represents a predefined inter-class distance threshold to ensure sufficient separation between different symbol center embeddings, [x]. + Represents the Hinge function; Regularization function: ； In the formula, This represents the fourth difference value. This represents the weight hyperparameter of the regularization term, used to control the strength of the regularization term. The central embedding of the symbol c The squared L2 norm is used to limit the size of the embedding vector; In the formula, It is the total loss. It is the first difference value. It is the third difference value. It represents the total number of iterations in the mean-shift clustering layer.

2. The method for recognizing hand-drawn flowcharts according to claim 1, characterized in that, The process of determining the category of the symbol feature based on the predicted category of strokes in the hand-drawn flowchart includes: Based on the Euclidean distance between the strokes, the strokes are merged into stroke clusters; The category of the symbol feature is obtained based on the predicted category of the strokes in the stroke cluster.

3. The method for recognizing hand-drawn flowcharts according to claim 1, characterized in that, The step of obtaining the second difference value of the sample symbol features includes: Based on the high-dimensional features of the sample, the third difference value of the message passing neural network is calculated using the cross loss function; Based on the sample symbol features, a node embedding function is used to calculate the fourth difference value of the mean-shift clustering layer. The node embedding function is used to optimize the clustering results of the mean-shift clustering layer to enhance the degree of aggregation between stroke features in the same sample symbol features and the degree of separation between stroke features in different sample symbol features. The sum of the third difference value and the fourth difference value is taken as the second difference value.

4. The method for recognizing hand-drawn flowcharts according to claim 1, characterized in that, Obtaining the first difference value of the predicted category of the sample includes: If the sample set of the stroke features of the sample is less than or equal to the sample set threshold, then the weighted difference entropy loss function is used to calculate the first difference value of the predicted category of the sample. If the sample set of the stroke features is larger than the sample set threshold, then the first difference value of the predicted category of the sample is calculated using the balanced Softmax loss function.

5. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the stroke recognition method for hand-drawn flowcharts as described in any one of claims 1 to 4.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the stroke recognition method for hand-drawn flowcharts as described in any one of claims 1 to 4.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the stroke recognition method for hand-drawn flowcharts as described in any one of claims 1 to 4.

Images