Freehand-drawn flow chart recognition method, electronic equipment, medium and computer product

Abstract

The invention discloses a recognition method of a hand-drawn flow chart, electronic equipment, a medium and a computer product. The method comprises the following steps: acquiring stroke characteristics corresponding to strokes in the hand-drawn flow chart; the stroke features are input into a graph neural network model, prediction categories of strokes in the freehand flow chart output by the graph neural network model and symbol features obtained through stroke feature clustering are obtained, and the graph neural network model is obtained through sample stroke feature training; and determining the category of the symbol feature based on the prediction category of the strokes in the hand-drawn flow chart. By implementing the technical scheme provided by the invention, the recognition precision of the symbols in the hand-painted flow chart can be improved, and the problem that the hand-painted flow chart is difficult to recognize in the prior art is effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a hand-drawn flowchart recognition method, electronic equipment, medium and computer product. Background Art

[0002] With the rapid development of information technology, people are increasingly using flowcharts to describe business processes, algorithmic processes, etc. At present, flowcharts are mainly drawn using computer drawing tools. However, in some application scenarios, such as meeting discussions and brainstorming, people are more accustomed to hand-drawn flowcharts. Compared with computer drawing tools, hand-drawn flowcharts are more flexible and intuitive, and are easy to express complex ideas and concepts.

[0003] However, the recognition and understanding of hand-drawn flowcharts still face many challenges. The symbols in hand-drawn flowcharts, such as boxes and arrows, are irregular in shape, different in size, and randomly positioned. In addition, different people's drawing styles vary greatly, which brings great difficulties to the automatic recognition of flowcharts. Currently, the commonly used symbol recognition methods are mainly aimed at printed symbols, and the recognition accuracy of hand-drawn symbols is relatively low. Summary of the invention

[0004] The present invention provides a hand-drawn flowchart recognition method, electronic equipment, medium and computer product, which can improve the recognition accuracy of symbols in the hand-drawn flowchart and effectively solve the problem of difficulty in hand-drawn flowchart recognition in the prior art.

[0005] In a first aspect of the present invention, a method for recognizing a hand-drawn flowchart is provided, comprising: Get the stroke features corresponding to the strokes in the hand-drawn flowchart; Inputting the stroke features into a graph neural network model, obtaining predicted categories of strokes in the hand-drawn flowchart output by the graph neural network model, and symbol features obtained by clustering the stroke features, wherein the graph neural network model is trained by sample stroke features; Based on the predicted category of the strokes in the hand-drawn flowchart, the category of the symbol feature is determined.

[0006] Optionally, obtaining stroke features corresponding to the strokes in the hand-drawn flowchart includes: Get the strokes in the hand-drawn flowchart; Acquire geometric features of the strokes, where the geometric features are used to represent position information of the strokes; Acquire edge features between the strokes, where the edge features are used to characterize the spatiotemporal relationship between the strokes; The geometric features and edge features corresponding to the strokes are used as stroke features.

[0007] Optionally, the determining the category of the symbol feature based on the predicted category of the strokes in the hand-drawn flowchart includes: Based on the Euclidean distance between the strokes, merging the strokes into a stroke cluster; Based on the predicted categories of the strokes in the stroke cluster, the category of the symbol feature is obtained.

[0008] Optionally, the graph neural network model includes a node classification branch and a node clustering branch, and the training process of the graph neural network model includes: Inputting the sample stroke feature into the node classification branch, and obtaining the sample prediction category corresponding to the sample stroke feature output by the node classification branch; Inputting the sample stroke features into the node clustering branch, and obtaining the sample symbol features output by the node clustering branch and obtained by clustering the sample stroke features; Obtaining a first difference value of the sample prediction category, and obtaining a second difference value of the sample symbol feature; Based on the first difference value and the second difference value, adjust the parameters of the graph neural network model.

[0009] Optionally, the node clustering branch includes a message passing neural network and a mean shift clustering layer, and the step of inputting the sample stroke features into the node clustering branch and obtaining the sample symbol features output by the node clustering branch and obtained by clustering the sample stroke features includes: Inputting the sample stroke features into the message passing neural network to obtain sample high-dimensional features output by the message passing neural network, wherein the sample high-dimensional features are used to characterize the shape and position relationship between the sample stroke features; The sample high-dimensional features are input into the mean shift clustering layer to obtain the sample symbol features output by the mean shift clustering layer, wherein the mean shift clustering layer is used to cluster the sample high-dimensional features.

[0010] Optionally, obtaining the second difference value of the sample symbol feature includes: Based on the high-dimensional features of the sample, a cross loss function is used to calculate a third difference value of the message passing neural network; Based on the sample symbol feature, a node embedding function is used to calculate a fourth difference value of the mean shift clustering layer, wherein the node embedding function is used to optimize the clustering result of the mean shift clustering layer to enhance the degree of aggregation between stroke features in the same sample symbol feature and the degree of separation between stroke features in different sample symbol features; The sum of the third difference value and the fourth difference value is taken as the second difference value.

[0011] Optionally, obtaining the first difference value of the sample prediction category includes: If the sample set of the sample stroke feature is less than or equal to the sample set threshold, a weighted difference entropy loss function is used to calculate a first difference value of the sample prediction category; If the sample set of the sample stroke feature is greater than the sample set threshold, a balanced Softmax loss function is used to calculate a first difference value of the sample prediction category.

[0012] In a second aspect of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a method for recognizing a hand-drawn flowchart as described above is implemented.

[0013] In a third aspect of the present invention, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the method for recognizing a hand-drawn flowchart as described in any one of the above is implemented.

[0014] In a fourth aspect of the present invention, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned methods for recognizing a hand-drawn flowchart.

[0015] In summary, one or more technical solutions provided in the present invention have at least the following technical effects or advantages: By obtaining the stroke features of the hand-drawn flowchart and inputting them into the graph neural network model trained with sample stroke features, various symbols in the hand-drawn flowchart can be effectively recognized. The graph neural network model can not only output the predicted category of the strokes, but also obtain the symbol features by clustering the stroke features. The dual output improves the accuracy of hand-drawn flowchart recognition. The category of the symbol feature is determined based on the predicted stroke category, which further enhances the reliability of recognition.

[0016] The above method makes full use of the advantages of graph neural networks in processing graphical data, and can better capture the spatial relationship and topological structure between hand-drawn strokes, thereby improving the recognition ability of complex hand-drawn flowcharts. At the same time, due to the use of sample stroke features for training, this method has good generalization ability and can adapt to different people's hand-drawn styles and various complex flowchart structures.

[0017] Compared with the traditional recognition method for printed symbols, this method can improve the recognition accuracy of hand-drawn flowcharts and effectively solve the problem of difficulty in recognizing hand-drawn flowcharts in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 The present invention is a flowchart of a method for identifying a hand-drawn flowchart provided by an embodiment of the present invention.

[0020] Figure 2 It is a feature extraction network framework diagram provided by an embodiment of the present invention.

[0021] Figure 3 It is a schematic diagram of the architecture of a graph neural network model provided by an embodiment of the present invention.

[0022] Figure 4 This is a sample diagram of hand-drawn flowchart recognition provided by an embodiment of the present invention.

[0023] Figure 5 It is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] Please refer to Figure 1 , Figure 1 This is a flowchart of a method for identifying a hand-drawn flowchart provided in an embodiment of the present application. The method can be implemented by a computer program, can be implemented by a single-chip microcomputer, or can be run on a hand-drawn flowchart recognition system based on the von Neumann system. The computer program can be integrated into an application or run as an independent tool application. Specifically, the hand-drawn flowchart recognition method can include the following steps: Step 101: Obtain stroke features corresponding to the strokes in the hand-drawn flowchart.

[0026] Among them, a hand-drawn flowchart refers to a graphical representation created by a user through manual drawing, which is used to visually describe a workflow, system structure, decision-making process or other complex logical relationships. In the embodiment of the present application, a hand-drawn flowchart can be understood as an online handwritten flowchart drawn in real time by a user on a digital platform through an electronic handwriting device. These flowcharts are composed of a series of geometric symbols (such as rectangles, diamonds, circles, etc.), connecting lines (such as arrows) and handwritten text annotations to form a two-dimensional spatial structure.

[0027] Correspondingly, stroke features refer to the digital representation of a single stroke in a hand-drawn flowchart, and the stroke features comprehensively capture the geometric, temporal and dynamic information of the strokes. In the embodiment of the present application, the stroke features corresponding to the strokes can be understood as a set of multidimensional vectors, which comprehensively encode information such as the shape, direction, pressure change and spatial relationship of the strokes with other strokes.

[0028] Specifically, stroke features can include the spatial shape and position information of the stroke, such as the coordinates of the starting and ending points, and the relative coordinate sequence of sampling points on the trajectory; time information reflecting the drawing process, such as the drawing duration and the timestamp sequence of each point; information describing dynamic changes, such as changes in pen tip pressure and drawing speed; and the relationship between the stroke and other surrounding strokes, such as relative position, direction, and distance. In addition, it also includes statistical descriptions of the overall characteristics of the stroke, such as average speed, average pressure, and direction histograms.

[0029] Based on the above embodiment, as an optional embodiment, in step 101, the step of obtaining stroke features corresponding to the strokes in the hand-drawn flowchart may further include the following steps: Step 201: Obtain the strokes in the hand-drawn flowchart.

[0030] In the embodiment of the present application, in order to effectively represent and process the structure of the hand-drawn flowchart, each flowchart is modeled as a graph structure with attributes, denoted as G=(V,E). This graph structure representation method can intuitively and efficiently capture the spatial relationship and connection information between the strokes in the flowchart.

[0031] Specifically, in the graph G, the node set V represents each stroke in the flow chart. Each node v i ∈V corresponds to an independent stroke in the flowchart. The above representation method converts the basic components of the flowchart into nodes in the graph structure, laying the foundation for subsequent analysis and processing.

[0032] The edge set E represents the connection relationship between the strokes. In the process of constructing the initial graph, a method based on spatial distance is used to determine the existence of edges. i, first calculate the Euclidean distance between it and all other strokes. Then, select i The nearest These closest strokes are regarded as the spatial neighbors of s, denoted as N s The above method can effectively capture the local spatial relationship between strokes, while also limiting the complexity of the graph and avoiding the computational burden that may be brought by the fully connected graph.

[0033] After determining the spatial neighbors of each stroke, start adding elements to the edge set E of the graph G. For each stroke v in the flowchart i , v i Its spatial neighbors The connection between each stroke in is added to the edge set E. Specifically, for v i ∈V and v j ∈N(v i ), add edge e to E ij =(v i ,v j ). These edges reflect the spatial structure information of the flowchart and provide input for subsequent graph neural network processing.

[0034] In order to more comprehensively describe the characteristics of nodes and edges, the present invention uses real-valued feature vectors to represent them. i , using a feature vector f(v i ) to describe the attributes of the stroke, such as the geometric features of the stroke, timing information, etc. Similarly, for each edge e ij , a feature vector can be used to describe the relationship between two strokes, such as relative position, relative spatial position and relative time relationship.

[0035] In this way, the hand-drawn flowchart can be transformed into a graph structure with rich attribute information. This representation method not only retains the spatial structure information of the flowchart, but also encodes the detailed features of the strokes and stroke relationships through feature vectors. This provides an ideal input form for subsequent graph neural network processing, which helps the model better understand and recognize the structure and content of the hand-drawn flowchart.

[0036] Step 202: Acquire geometric features of the strokes, where the geometric features are used to represent position information of the strokes.

[0037] Among them, geometric features refer to the digital representation of the spatial attributes and shape characteristics of a single stroke in a hand-drawn flowchart. In an embodiment of the present application, geometric features can be understood as a set of multidimensional vectors that comprehensively encode information such as the shape, direction, curvature of the strokes, and the spatial relationship with other strokes. Specifically, the above features include the relative coordinate sequence of the strokes, direction changes, curvature changes, stroke length, relative position of the starting point and the end point, etc. The geometric features are used to accurately characterize the position information and shape characteristics of the strokes, and provide key spatial information input for subsequent symbol segmentation and recognition tasks, so that the model can effectively distinguish and recognize different types of flowchart symbols, even if these symbols may have similar appearances or complex structures.

[0038] After obtaining the strokes in the hand-drawn flowchart, the embodiment of the present invention further extracts the geometric features of each stroke, which are used to accurately characterize the position information of the stroke. Obtaining the geometric features of the stroke can capture the spatial attributes of the stroke, laying the foundation for subsequent symbol segmentation and recognition tasks.

[0039] Specifically, in order to ensure that the model can effectively capture the core geometric features of the strokes while avoiding the influence of absolute position changes, each stroke is first preprocessed. Specifically, we convert each stroke into a sequence containing 3D vectors by calculating the first-order difference of the coordinates of the trajectory points. The converted sequence can be expressed as: ; In the formula, , , respectively represent the changes of adjacent trajectory points in the x and y coordinate directions, and m is the length of the stroke s. Indicates the status of the pen. Indicates that the stroke falls on this store, otherwise This representation method not only retains the shape information of the strokes, but also achieves translation invariance through relative coordinates, so that the model can better focus on the geometric features of the strokes themselves without being affected by their absolute position in the flowchart.

[0040] Please refer to Figure 2 , Figure 2 A feature extraction network framework diagram provided by an embodiment of the present invention is shown. In order to further extract the high-level geometric features of the strokes, the embodiment of the present application adopts a feature extraction network based on recurrent neural networks (RNN). The network is stacked with multiple bidirectional RNN layers, and each layer contains both forward and reverse RNN units. This design allows the network to fully consider the context information when processing sequence data, thereby more comprehensively understanding the temporal and spatial characteristics of the stroke trajectory.

[0041] Furthermore, the forward unit processes the forward information of the time series, while the reverse unit absorbs the reverse information, so that the model can more comprehensively understand the contextual information of the stroke trajectory sequence data. After being processed by all RNN layers, the output vector from the final RNN layer will be sent to a global maximum pooling layer, which is used to reduce the dimension of the output features, thereby retaining the most significant features and discarding redundant information. Finally, the results of the pooling layer will be passed to a fully connected layer for further feature compression and integration.

[0042] Through the above method steps, the embodiment of the present invention can effectively extract highly generalized and information-rich geometric features from the original stroke data. The geometric features not only contain the shape and direction information of the strokes, but also retain the dynamic features of the stroke drawing process. This multi-dimensional feature representation greatly enhances the model's ability to recognize various strokes in the hand-drawn flowchart.

[0043] At the same time, due to the use of relative coordinates and RNN structure, the extracted features have good translation invariance and the ability to capture long-term dependencies, which enables the model to maintain stable performance when processing flowcharts of different sizes, positions and complexities. Ultimately, these extracted geometric features will serve as feature vectors for each node in the graph structure, providing rich and effective input information for subsequent graph neural network processing, thereby significantly improving the accuracy and robustness of the entire hand-drawn flowchart recognition system.

[0044] Step 203: Acquire edge features between strokes. The edge features are used to characterize the spatiotemporal relationship between strokes.

[0045] Among them, the edge feature refers to a digital representation that describes the relationship between two strokes in a hand-drawn flowchart. In an embodiment of the present invention, the edge feature can be understood as a multidimensional vector that comprehensively encodes the spatial relationship and timing information between the two strokes. Specifically, these features include relative position information between the two strokes. The edge feature is used to accurately characterize the spatiotemporal relationship between the strokes, providing key contextual information for the graph neural network, so that the model can effectively understand the logical connection and structural relationship between the symbols in the flowchart, thereby improving the accuracy of symbol segmentation and recognition, especially when dealing with complex flowchart structures.

[0046] Specifically, we first resample the stroke trajectory points through the equally spaced interpolation method so that they have the same length. Then, we calculate the difference between the sampling point coordinates of the two strokes and concatenate them to obtain the position embedding (PE). Position embedding can be expressed as: ; In addition to spatial relationships, the relative order of strokes is also important temporal information. The relative order of strokes is described by time embedding (TE): ; In the formula, is a temporal factor that controls the importance of temporal neighbors, is the number of factors. is the concatenation operator. The final edge feature is the position embedding and time embedding The concatenation is expressed as: ; Due to the use of relative coordinates and time embedding, the extracted edge features have good translation invariance and the ability to capture long-term dependencies, which enables the model to maintain stable performance when processing flowcharts of different sizes, layouts and complexities.

[0047] Step 204: Use the geometric features and edge features corresponding to the strokes as stroke features.

[0048] Step 102: Input the stroke features into the graph neural network model to obtain the predicted categories of the strokes in the hand-drawn flowchart output by the graph neural network model and the symbol features obtained by clustering the stroke features, wherein the graph neural network model is trained by sample stroke features.

[0049] Among them, the graph neural network model refers to a type of deep learning model specifically used to process and analyze graph structure data. In an embodiment of the present invention, the graph neural network model can be understood as a complex network structure based on message passing neural networks (MPNN), which includes node classification branches and node clustering branches, as well as a learnable mean shift clustering layer.

[0050] Specifically, the model accepts a graph structure with strokes as nodes and relationships between strokes as edges as input, and learns high-level representations of strokes and relationships between strokes through multi-layer message passing and feature aggregation operations. The graph neural network model is used to transform the original stroke features and edge features in the hand-drawn flowchart into a higher-level representation, while outputting the predicted category and symbol features of the strokes. It can effectively capture the complex spatial and logical relationships in the flowchart, and provide a key computational basis for accurately identifying the symbols and their structures in the hand-drawn flowchart, thereby greatly improving the performance and adaptability of the entire hand-drawn flowchart recognition system.

[0051] Among them, the predicted category of a stroke refers to the prediction result of the model on the symbol type to which each individual stroke in the hand-drawn flowchart belongs. In an embodiment of the present invention, the predicted category of a stroke can be understood as a probability distribution vector, each element of which corresponds to a category in a predefined symbol category set, and its value represents the probability that the stroke belongs to the corresponding category. For example, for a flowchart symbol set containing categories such as "rectangle", "circle", "diamond", and "arrow", the predicted category of a stroke may be expressed as [0.7, 0.1, 0.05, 0.15], which means that the stroke has a 70% probability of belonging to a rectangle, a 10% probability of belonging to a circle, and so on. The predicted category of a stroke is used to preliminarily determine the symbol type to which each stroke may belong.

[0052] Correspondingly, symbol features refer to abstract representations that describe complete symbols in hand-drawn flowcharts. In an embodiment of the present invention, symbol features can be understood as a set of multidimensional vectors output by the node clustering branch of the graph neural network model and obtained after mean shift clustering. Each vector represents a complete symbol, which integrates the feature information of all strokes that constitute the symbol.

[0053] Specifically, the above features include information such as the geometric properties of the symbol (such as shape, size, direction) and the relative position relationship with other symbols. Symbol features are used to accurately characterize each independent symbol in the flowchart, providing key input for subsequent symbol recognition and flowchart structure analysis. This enables the system to integrate the original stroke-level information into higher-level symbol-level information.

[0054] For further information, please refer to Figure 3 , Figure 3 A schematic diagram of the architecture of a graph neural network model is shown, Figure 3 As shown in the figure, after the stroke features are input into the graph neural network model, the node classification branch processes the input graph structure through multiple MPNN layers and outputs a predicted category probability distribution for each stroke node, indicating the probability that each stroke belongs to each possible symbol category. For example, for a specific stroke, the model may output that it has a 70% probability of being part of a rectangle, a 20% probability of being part of a circle, and a 10% probability of belonging to other categories.

[0055] At the same time, the node clustering branch also processes the input graph structure, but its goal is to generate an embedding representation of each stroke node. At the end of the node clustering branch, a pre-trained mean shift clustering layer is used to cluster these embeddings, grouping the strokes belonging to the same symbol together to obtain the symbol feature. The symbol feature is a high-level abstract representation of the complete symbol, which integrates the characteristic information of all the strokes that make up the symbol, including geometric properties, topological structure, and relative positional relationships.

[0056] Step 103: Determine the category of the symbol feature based on the predicted category of the strokes in the hand-drawn flowchart.

[0057] The category of symbol features refers to the final recognition result of each complete symbol in the hand-drawn flowchart. In the embodiment of the present invention, the category of symbol features can be understood as a discrete label, indicating that the symbol belongs to a specific type in the predefined symbol set, such as "rectangle", "circle", "diamond", "arrow", etc. The category of symbol features is used to finally determine the specific type of each symbol in the flowchart, which is the final output of the entire hand-drawn flowchart recognition process and provides key information for subsequent flowchart understanding and analysis.

[0058] Based on the above embodiment, as an optional embodiment, the step of determining the category of the symbol feature based on the predicted category of the strokes in the hand-drawn flowchart in step 103 may further include the following steps: Step 301: Based on the Euclidean distance between the strokes, the strokes are merged into a stroke cluster.

[0059] In the process of determining the category of the symbol feature, the embodiment of the present invention can more accurately capture the overall structure of the symbol by combining spatially similar strokes, thereby improving the accuracy of the final symbol category recognition.

[0060] Specifically, we first calculate the Euclidean distance between each stroke and all other strokes in the hand-drawn flowchart. Then, we set a distance threshold θ. If the Euclidean distance between two strokes is less than this threshold, we consider that the two strokes belong to the same stroke cluster.

[0061] The merging process adopts an iterative method. Initially, each stroke is regarded as an independent stroke cluster. Then, for each stroke 1, all strokes whose distance to it is less than θ are searched. If a stroke 2 that meets the conditions is found, the stroke cluster where stroke 1 is located is merged with the stroke cluster where stroke 2 is located. The above process is repeated until no new merge occurs.

[0062] Step 302: Obtain the category of the symbol feature based on the predicted category of the strokes in the stroke cluster.

[0063] Specifically, for each stroke cluster obtained by merging the Euclidean distance, the predicted categories of all the strokes in the cluster are considered. A voting mechanism can be used to determine the final category of the symbol feature. First, the predicted categories of all the strokes belonging to the stroke cluster are counted. The predicted category of each stroke is a probability distribution, which indicates the probability that the stroke belongs to each possible category. Thus, a comprehensive category probability distribution is obtained. Finally, the category with the highest probability in this comprehensive probability distribution is selected as the final category of the symbol feature.

[0064] Please refer to Figure 4 , Figure 4 A sample diagram of hand-drawn flowchart recognition provided by an embodiment of the present invention is shown.

[0065] The above embodiment illustrates the application process of the graph neural network model. Based on the above embodiment, the training process of the graph neural network model will be described below. Specifically, the process may include the following steps: Step 401: Input sample stroke features into a node classification branch, and obtain a sample prediction category corresponding to the sample stroke features output by the node classification branch.

[0066] Among them, the node classification branch refers to a subnetwork structure in the graph neural network model that is specifically used to process stroke classification tasks. In an embodiment of the present invention, the node classification branch can be understood as a complex network structure composed of multiple layers of MPNN. This branch receives stroke features as input, and through a series of feature extraction and transformation operations, it finally outputs the probability distribution of each stroke belonging to each predefined symbol category. The node classification branch is used to identify and classify various strokes in the hand-drawn flowchart, and predict the symbol type to which each stroke may belong. It generates a high-level feature representation that can reflect the stroke type by learning the geometric features and contextual information of the strokes, and provides basic information for subsequent symbol recognition, thereby improving the classification accuracy and robustness of the entire hand-drawn flowchart recognition system.

[0067] Specifically, the preprocessed sample stroke features are first input into the node classification branch of the graph neural network model. This node classification branch consists of multiple MPNN layers, which can effectively capture the spatial and logical relationships between strokes. The MPNN layer gradually extracts and integrates the high-level feature representation of the strokes through iterative message passing and aggregation operations. In this process, each stroke node not only considers its own features, but also receives information from adjacent strokes to obtain a more comprehensive contextual understanding. After being processed by multiple layers of MPNN, the node classification branch outputs a predicted category probability distribution for each sample stroke, indicating the probability that the stroke belongs to each possible symbol category.

[0068] Through the above steps, the graph neural network model learns the mapping relationship from the original stroke features to the symbol category, and improves the recognition ability of various stroke types. It can handle the complex and diverse stroke forms in the hand-drawn flowchart and adapt to the drawing styles of different users; secondly, by considering the relationship between the strokes, the graph neural network model can better understand the role of the strokes in the entire symbol and improve the accuracy of classification.

[0069] Step 402: Input the sample stroke features into the node clustering branch, and obtain the sample symbol features output by the node clustering branch and obtained by clustering the sample stroke features.

[0070] Among them, the node clustering branch refers to a subnetwork structure in the graph neural network model that is specifically used to process symbol segmentation tasks. In an embodiment of the present invention, the node clustering branch can be understood as a complex network structure consisting of a multi-layer MPNN and a learnable mean shift clustering layer. This branch receives stroke features as input, and through a series of feature extraction and transformation operations, it finally outputs the embedded representation and clustering results of each stroke. The node clustering branch is used to aggregate strokes belonging to the same symbol to achieve symbol segmentation in hand-drawn flowcharts. It generates high-level feature representations that can reflect the symbol structure by learning the spatial and logical relationships between strokes, providing key intermediate results for subsequent symbol recognition.

[0071] Correspondingly, the mean shift clustering layer refers to a learnable neural network layer designed based on the principle of the mean shift algorithm. In the embodiment of the present invention, the mean shift clustering layer can be understood as a special network layer at the end of the node clustering branch, which receives the embedded representation of the strokes as input, and outputs the clustered sample symbol features through iterative density estimation and center point update process.

[0072] Furthermore, the core operations of the mean shift clustering layer include density estimation using Gaussian kernels and moving data points along the density gradient direction. The mean shift clustering layer is used to effectively aggregate strokes belonging to the same symbol to achieve accurate symbol segmentation. It can adaptively process hand-drawn symbols of different complexity and structure without pre-specifying the number of clusters. Through the preset clustering parameters, the flexibility and accuracy of the model in processing various flowchart styles are improved. The introduction of the mean shift clustering layer significantly improves the performance of the node clustering branch, providing high-quality symbol-level features for subsequent symbol recognition.

[0073] Based on the above embodiment, step 402 may further include the following steps: Step 501: Input sample stroke features into a message passing neural network to obtain sample high-dimensional features output by the message passing neural network. The sample high-dimensional features are used to characterize the shape and position relationship between the sample stroke features.

[0074] Specifically, the preprocessed sample stroke features are first used as the input of MPNN. MPNN gradually extracts and integrates the high-level feature representation of strokes through multiple rounds of message passing and aggregation operations. In each round of operation, each stroke node not only considers its own features, but also receives information from adjacent strokes. The above message passing mechanism enables MPNN to effectively capture the spatial and logical relationships between strokes.

[0075] At the same time, MPNN also integrates the absolute position information of the strokes, including the starting and ending coordinates of the strokes and the bounding box coordinates, which provides important spatial context for the model. After being processed by multiple layers of MPNN, the network outputs sample high-dimensional features, which are the representation of the original sample stroke features in high-dimensional space. Sample high-dimensional features not only contain the geometric information of a single stroke, but also encode the relative position, direction and structural relationship between strokes.

[0076] Through the above methods, the complex spatial structure and interdependence between strokes in hand-drawn flowcharts can be captured, improving the model's ability to understand complex symbols; secondly, through the message passing mechanism, the model can consider a wider range of contextual information, thereby more accurately identifying the role of strokes in the entire symbol; finally, the generated sample high-dimensional features provide more discriminative input for subsequent clustering operations, which helps to improve the accuracy of symbol segmentation. This enables the model to extract more abstract and information-rich representations from the original stroke-level features, laying the foundation for symbol segmentation and recognition of hand-drawn flowcharts, thereby improving the system's performance and adaptability when processing various complex flowcharts.

[0077] Step 502: Input the sample high-dimensional features into the mean shift clustering layer to obtain the sample symbol features output by the mean shift clustering layer, wherein the mean shift clustering layer is used to cluster the sample high-dimensional features.

[0078] Specifically, the sample high-dimensional features generated by the message passing neural network are first input into the mean shift clustering layer. The mean shift clustering layer is a trainable neural network layer designed based on the principle of the mean shift algorithm. In this layer, for each sample high-dimensional feature, an isotropic multivariate Gaussian kernel is used to estimate the point density using the Parsons window method. The Gaussian kernel is defined as

[0079] ; Where σ is used to control the bandwidth of the Gaussian kernel. Based on density estimation, the mean shift clustering algorithm moves data points in the direction of the density pattern in each iteration.

[0080] Furthermore, based on density estimation, the mean shift clustering algorithm moves in the direction of the density pattern at each iteration, corresponding to the direction of maximum density increase. Moving along the gradient direction results in updated stroke features This operation has a total of Layers, each layer is represented as: ; The mean shift clustering layer gradually adjusts the center points of the clusters so that the final generated clusters can very accurately reflect the symbol distribution in the flowchart, thus providing strong support for subsequent reasoning.

[0081] Step 403: Obtain a first difference value of the sample prediction category, and obtain a second difference value of the sample symbol feature.

[0082] Among them, the first difference value refers to the difference measure between the prediction result of the model in the stroke classification task and the true label. In the embodiment of the present invention, the first difference value can be understood as the cross entropy loss between the sample prediction category probability distribution output by the node classification branch and the true category label. It is used to quantify the accuracy of the model in identifying and classifying various strokes in the hand-drawn flowchart, and provide direct numerical guidance for optimizing the classification performance of the model.

[0083] Correspondingly, the second difference value refers to the difference measure between the feature representation generated by the model in the symbol segmentation task and the ideal feature representation. In an embodiment of the present invention, the second difference value can be understood as the distance or similarity between the sample symbol feature output by the node clustering branch and the target symbol feature. It can be in the form of contrast loss, etc., to evaluate the model's effect in aggregating strokes belonging to the same symbol.

[0084] These two difference values ​​are used together to comprehensively evaluate the model's performance in the two key tasks of stroke classification and symbol segmentation, providing important feedback information for parameter adjustment and performance optimization during model training, thereby improving the accuracy and robustness of the entire hand-drawn flowchart recognition system.

[0085] Based on the above embodiment, as an optional embodiment, the step of obtaining the first difference value of the sample prediction category in step 403 may further include the following steps: Step 601: If the sample set of the sample stroke feature is less than or equal to the sample set threshold, a weighted difference entropy loss function is used to calculate a first difference value of the sample prediction category.

[0086] Specifically, since some symbol categories in a small-scale data set may appear less frequently, the model tends to ignore these rare categories during training. To solve this problem, the embodiment of the present invention uses a weighted cross entropy loss function to assign different weights to different categories, so that the model can pay more attention to those categories with fewer samples, thereby improving the overall classification accuracy.

[0087] Specifically, first determine whether the sample set size of the sample stroke feature is less than or equal to a preset sample set threshold. If the condition is met, the following weighted cross entropy loss function is used to calculate the first difference value: ; in, is the number of flow charts, is the number of strokes in a single flowchart, Indicates category The weighting coefficient of , which indicates the frequency of the category in the training set. For categories with lower frequency, the weight will be higher, ensuring that the model can pay more attention to these categories. for The predicted category probability is obtained by calculating the output of the stroke classification branch through softmax.

[0088] In summary, using the weighted cross entropy loss function to calculate the first difference value can effectively alleviate the problem of category imbalance, so that the model pays enough attention to each category during the training process, avoiding overfitting of common categories and neglect of rare categories. Secondly, by adjusting the weights of different categories, the model can better learn the characteristics of rare categories and improve the recognition ability of these categories. Finally, the above method improves the generalization ability of the model on small-scale data sets, so that it can maintain a high recognition accuracy when facing hand-drawn flowcharts of various complexities and category distributions.

[0089] Step 602: If the sample set of the sample stroke feature is greater than the sample set threshold, a balanced Softmax loss function is used to calculate a first difference value of the sample prediction category.

[0090] Specifically, since large-scale data sets usually contain more diverse and complex samples, although the problem of class imbalance may still exist, the greater challenge lies in how to effectively use rich data to improve the generalization ability of the model and the recognition ability of tail categories. The embodiment of the present invention adopts a balanced Softmax loss function by introducing an adjustment factor of category frequency, which can not only alleviate the problem of class imbalance, but also improve its sensitivity to tail categories while maintaining a high recognition rate of common categories.

[0091] Specifically, first determine whether the sample set size of the sample stroke feature is greater than the preset sample set threshold. If the condition is met, the following balanced Softmax loss function is used to calculate the first difference value: ; In the formula, is the number of flow charts, is the number of strokes in a single flow chart, is the number of symbols in the flowchart, Indicates category The weighting coefficient of represents the frequency of the category in the training set. Represents the output of the node classification branch.

[0092] In summary, the use of balanced Softmax loss function can significantly improve the model's ability to recognize tail categories while maintaining the model's high recognition accuracy for common categories. By considering category frequency in the loss calculation, the model is encouraged to pay more attention to categories with fewer samples, thereby improving the overall balance of classification. Secondly, this method can better utilize the rich information in large-scale datasets and improve the generalization ability of the model. Since the loss function directly acts on the original output of the model, it can more finely adjust the model's sensitivity to different categories, making the model more stable when facing complex hand-drawn flowcharts. Finally, the balanced Softmax loss function has high computational efficiency and is suitable for training on large-scale datasets. It can effectively use modern hardware resources to accelerate the convergence process of the model.

[0093] Based on the above embodiment, as an optional embodiment, the step of obtaining the second difference value of the sample symbol feature in step 403 may further include the following steps: Step 701: Based on the high-dimensional features of the samples, a cross loss function is used to calculate a third difference value of the message passing neural network.

[0094] In the process of obtaining the second difference value of the sample symbol feature, based on the sample high-dimensional features, the cross loss function is used to calculate the third difference value of the message passing neural network. This can evaluate the performance of the message passing neural network in extracting and integrating stroke features, ensuring that the generated high-dimensional features can effectively capture the spatial and logical relationships between strokes.

[0095] Specifically, we first obtain the sample high-dimensional features output by the message passing neural network. These high-dimensional features are the representations of the original sample stroke features after multiple rounds of message passing and aggregation operations, and contain rich contextual information. Then, we can use the cross loss function to calculate the difference between these high-dimensional features and the ideal feature representation. The cross loss function is designed to make the stroke features belonging to the same symbol closer in the high-dimensional space, while the features of different symbols are farther apart.

[0096] The calculation formula can be expressed as: ; In the formula, is the number of flow charts, for The true label of is {−1,1}, where 1 indicates that the connection is the same symbol and -1 indicates that the connection is different symbols. It can be inferred during training. for The predicted probability category. is the edge feature set corresponding to the hand-drawn flowchart.

[0097] Step 702: Based on the sample symbol features, a node embedding function is used to calculate the fourth difference value of the mean shift clustering layer. The node embedding function is used to optimize the clustering results of the mean shift clustering layer to enhance the degree of aggregation between the stroke features in the same sample symbol features and the degree of separation between the stroke features in different sample symbol features.

[0098] Specifically, first obtain the sample symbol features output by the mean shift clustering layer. The sample symbol features are high-level representations after clustering processing, which contain symbol-level structural information. Then, the node embedding function can be used to calculate the fourth difference value. The node embedding function is designed to optimize the clustering results, and its calculation process is continuously optimized for each iteration of the mean shift clustering layer. The specific calculation formula may include: (1) Average embedding function of symbolic class: ; In the formula, represents the central embedding vector of symbol c, that is, the average value of all stroke embeddings in symbol c, represents the embedding vector of stroke i, represents the stroke set in symbol c, including all strokes i belonging to symbol c, Indicates the number of strokes in symbol c.

[0099] (2) Intra-class compact loss function: ; In the formula, represents the intra-class compactness loss, which measures the distance between the stroke embedding and the symbol center embedding within the same symbol. represents the square of the L2 norm, represents the distance between the center embedding of the symbol and the stroke embedding, C represents the number of hand-drawn flowcharts, Indicates the number of strokes in the symbol.

[0100] (3) Inter-class separation loss: ; In the formula, Represents the inter-class separation loss, measuring the central embedding between different symbols and The distance between represents the predefined inter-class distance threshold to ensure sufficient separation between different symbol center embeddings, [x] + Represents the Hinge function. If the distance is less than 2 If so, punishment will occur; otherwise, no punishment will occur.

[0101] (4) Regularization function: ; In the formula, represents the weight hyperparameter of the regularization term, which is used to control the strength of the regularization term. represents the central embedding of symbol c The squared L2 norm of is used to limit the size of the embedding vector.

[0102] For simplicity, the total loss is expressed as:

[0103] In the formula, is the total number of iterations of the mean shift clustering layer, the goal is to simultaneously optimize the compactness of the internal strokes of the symbol (by ) and the separability between symbols (through ), and apply regularization to stabilize the scale of the embedding.

[0104] Step 703: Take the sum of the third difference value and the fourth difference value as the second difference value.

[0105] Step 404: Adjust the parameters of the graph neural network model based on the first difference value and the second difference value.

[0106] Specifically, the process involves parameters of all components such as node classification branch, node clustering branch, message passing neural network layer and mean shift clustering layer. The optimization algorithm can be used to update the model parameters according to the calculated difference value.

[0107] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the recognition method of the hand-drawn flowchart, the method comprising: obtaining the stroke features corresponding to the strokes in the hand-drawn flowchart; inputting the stroke features into the graph neural network model, obtaining the predicted category of the strokes in the hand-drawn flowchart output by the graph neural network model, and the symbol features obtained by clustering the stroke features, wherein the graph neural network model is trained by the sample stroke features; and determining the category of the symbol features based on the predicted category of the strokes in the hand-drawn flowchart.

[0108] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0109] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the hand-drawn flowchart recognition method provided by the above methods, which includes: obtaining stroke features corresponding to the strokes in the hand-drawn flowchart; inputting the stroke features into a graph neural network model, obtaining the predicted category of the strokes in the hand-drawn flowchart output by the graph neural network model, and the symbol features obtained by clustering the stroke features, wherein the graph neural network model is trained by sample stroke features; based on the predicted category of the strokes in the hand-drawn flowchart, determining the category of the symbol feature.

[0110] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the hand-drawn flowchart recognition method provided by the above-mentioned methods, the method comprising: obtaining stroke features corresponding to the strokes in the hand-drawn flowchart; inputting the stroke features into a graph neural network model, obtaining the predicted category of the strokes in the hand-drawn flowchart output by the graph neural network model, and the symbol features obtained by clustering the stroke features, wherein the graph neural network model is trained by sample stroke features; and determining the category of the symbol feature based on the predicted category of the strokes in the hand-drawn flowchart.

[0111] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0112] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for recognizing a hand-drawn flowchart, characterized in that: include: Get the stroke features corresponding to the strokes in the hand-drawn flowchart; Inputting the stroke features into a graph neural network model, obtaining predicted categories of strokes in the hand-drawn flowchart output by the graph neural network model, and symbol features obtained by clustering the stroke features, wherein the graph neural network model is trained by sample stroke features; Based on the predicted category of the strokes in the hand-drawn flowchart, the category of the symbol feature is determined.

2. The method for recognizing a hand-drawn flowchart according to claim 1, characterized in that: The step of obtaining stroke features corresponding to the strokes in the hand-drawn flowchart includes: Get the strokes in the hand-drawn flowchart; Acquire geometric features of the strokes, where the geometric features are used to represent position information of the strokes; Acquire edge features between the strokes, where the edge features are used to characterize the spatiotemporal relationship between the strokes; The geometric features and edge features corresponding to the strokes are used as stroke features.

3. The method for recognizing a hand-drawn flowchart according to claim 1, characterized in that: The step of determining the category of the symbol feature based on the predicted category of the strokes in the hand-drawn flowchart includes: Based on the Euclidean distance between the strokes, merging the strokes into a stroke cluster; Based on the predicted categories of the strokes in the stroke cluster, the category of the symbol feature is obtained.

4. The method for recognizing a hand-drawn flowchart according to any one of claims 1 to 3, characterized in that: The graph neural network model includes a node classification branch and a node clustering branch, and the training process of the graph neural network model includes: Inputting the sample stroke feature into the node classification branch, and obtaining the sample prediction category corresponding to the sample stroke feature output by the node classification branch; Inputting the sample stroke features into the node clustering branch, and obtaining the sample symbol features output by the node clustering branch and obtained by clustering the sample stroke features; Obtaining a first difference value of the sample prediction category, and obtaining a second difference value of the sample symbol feature; Based on the first difference value and the second difference value, adjust the parameters of the graph neural network model.

5. The method for recognizing a hand-drawn flowchart according to claim 4, characterized in that: The node clustering branch includes a message passing neural network and a mean shift clustering layer, and the step of inputting the sample stroke features into the node clustering branch and obtaining the sample symbol features output by the node clustering branch and obtained by clustering the sample stroke features includes: Inputting the sample stroke features into the message passing neural network to obtain sample high-dimensional features output by the message passing neural network, wherein the sample high-dimensional features are used to characterize the shape and position relationship between the sample stroke features; The sample high-dimensional features are input into the mean shift clustering layer to obtain the sample symbol features output by the mean shift clustering layer, wherein the mean shift clustering layer is used to cluster the sample high-dimensional features.

6. The method for recognizing a hand-drawn flowchart according to claim 5, characterized in that: The obtaining of the second difference value of the sample symbol feature comprises: Based on the high-dimensional features of the sample, a cross loss function is used to calculate a third difference value of the message passing neural network; Based on the sample symbol feature, a node embedding function is used to calculate a fourth difference value of the mean shift clustering layer, wherein the node embedding function is used to optimize the clustering result of the mean shift clustering layer to enhance the degree of aggregation between stroke features in the same sample symbol feature and the degree of separation between stroke features in different sample symbol features; The sum of the third difference value and the fourth difference value is taken as the second difference value.

7. The method for recognizing a hand-drawn flowchart according to claim 4, characterized in that: The obtaining of the first difference value of the sample prediction category includes: If the sample set of the sample stroke feature is less than or equal to the sample set threshold, a weighted difference entropy loss function is used to calculate a first difference value of the sample prediction category; If the sample set of the sample stroke feature is greater than the sample set threshold, a balanced Softmax loss function is used to calculate a first difference value of the sample prediction category.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the stroke recognition method of the hand-drawn flowchart according to any one of claims 1 to 7 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the stroke recognition method of the hand-drawn flowchart according to any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the stroke recognition method of the hand-drawn flowchart according to any one of claims 1 to 7 is implemented.

Images