Hand-drawn chart recognition method based on key point prediction

Abstract

This invention relates to the field of hand-drawn chart recognition technology, and specifically to a method for recognizing hand-drawn charts based on keypoint prediction. The method includes the following steps: 1. Detecting each element in the chart as a pair of keypoints, i.e., jointly determining the upper left and lower right corners of the element's bounding box; 2. Performing parallel pooling operations using Maximum Cross-Point Pooling (MICP) and Cumulative Cross-Point Pooling (CICP); 3. Fusing the feature maps from each pooling operation; 4. For each arrow connecting two elements, using an arrow direction prediction network to predict the head and tail keypoints of the arrow, and using a Snowflake Corner Pooling (SCP) module to enhance the arrow keypoint information; thus, the recognition of the entire chart structure is completed. This invention can perform hand-drawn chart recognition more effectively.

Description

Technical Field

[0001] This invention relates to the field of hand-drawn chart recognition technology, and more specifically, to a hand-drawn chart recognition method based on key point prediction. Background Technology

[0002] Hand-drawn charts are a simple, efficient, and convenient way to graphically express human thought and intent, extending verbal and written communication from a graphical perspective. Typically, one of the ultimate goals of hand-drawn charts is to preserve these handwritten drafts, and even to identify the graphic elements within the chart and understand the information it reflects, followed by reasoning, classification, reconstruction, or archiving. Hand-drawn chart recognition refers to the process of recognizing hand-drawn charts using computers, including element recognition and chart understanding. Hand-drawn chart recognition is fundamental to many hand-drawn chart digitization tasks, such as hand-drawn chart classification, reconstruction, and digitization. Therefore, hand-drawn chart recognition technology is a key technology that provides strong support for the practical application of this graphical form of expression.

[0003] However, the identification and analysis of hand-drawn diagrams, such as flowcharts, finite state machines, circuit diagrams, chemical molecular structure diagrams, and musical notation, remain challenging because these diagrams have complex two-dimensional structures and diverse primitive variations. Summary of the Invention

[0004] The present invention provides a hand-drawn chart recognition method based on key point prediction, which can overcome some of the defects of the prior art.

[0005] The hand-drawn chart recognition method based on key point prediction according to the present invention includes the following steps:

[0006] 1. Detect each graphic element in the chart as a pair of key points, that is, jointly determine the upper left and lower right corner points of the bounding box of the graphic element;

[0007] 2. Perform pooling operations in parallel using Maximum Cross-Corner Pooling (MICP) and Cumulative Cross-Corner Pooling (CICP).

[0008] Third, perform feature fusion on the feature maps of each pooling process;

[0009] Fourth, for each arrow connecting two primitives, the arrow direction prediction network is used to predict the head and tail key points of the arrow, and the snowflake corner pooling SCP module is used to enhance the arrow key point information; in this way, the recognition of the entire graph structure is completed.

[0010] As a preferred method, in Cumulative Cross-Pooling (CICP), to determine whether the activation value of a primitive is the top-left corner point, CICP searches for the top boundary of the target horizontally to the right and the leftmost boundary of the target vertically downwards; let F t and F l The input feature map is used for corner pooling. and F respectively t and F l The response at position (i,j) is analyzed; using the H×W feature map, CICP determines whether the activation value of the neuron at position (i,j) is at the top left corner, and this is determined by parallel accumulation on F. t All responses in the horizontal distribution of (i,j) and (i,H) in F, and the vertical distribution in F l The sum of (i,j) and (W,j) in T is used as T ij and L ij Finally, they are added together to generate the feature map F. CICP The calculation process can be expressed by the following formula:

[0011]

[0012]

[0013] The CICP pooling calculation method for the bottom right corner is similar to that for the top left corner. That is, all responses vertically distributed in (0,j) and (i,j) and all responses horizontally distributed in (i,0) and (i,j) are accumulated in parallel, and then the pooled results are summed.

[0014] As a preferred approach, the arrow direction prediction network uses a skip connection method, with the SCP module inserted in the middle of the skip connection.

[0015] As a preferred option, in the snowflake corner pooling SCP module:

[0016] Let F be the characteristic map of SCP, F ij This is the response at position (i,j) in F. Using the H×W feature map, the pooled feature map F... SCP The activation response of the neuron at position (i,j) in the SCP is expressed by the following formula:

[0017]

[0018] Preferably, the arrow direction prediction network uses heatmaps, offsets, and semantic embeddings to predict the head and tail keypoints of the arrow, as follows:

[0019] Let P cij It predicts the probability of class c at position (i,j) in the heatmap, ycij It's a ground-truth heatmap with C channels; then, the class loss of the arrow keypoints is estimated using the Focal Loss function.

[0020]

[0021] Where N is the number of objects in the image, and α is a hyperparameter that controls the contribution of each point;

[0022] The position of the arrow keypoint is slightly corrected by predicting the offset. Let (x, y) be the position in the image. This is its downsampling position in the heatmap, where s is the downsampling factor; estimate the deviation of the arrow key point k between these two positions:

[0023]

[0024] The prediction bias and ground-truth bias can be calculated using the Smooth-L1 Loss function:

[0025]

[0026] To determine which pair of leading and trailing keypoints belong to the same arrow, semantic embedding is used to match the two keypoints with the highest similarity; let e hk For the semantic embedding of the head keypoint of arrow k, e tk The semantic embeddings for the tail keypoints are all four-dimensional vectors; the network is trained using pull loss to match the keypoints, and push loss is used to remove keypoints that do not belong to the same object.

[0027]

[0028]

[0029]

[0030] Finally, the loss functions of all network branches are linearly combined to obtain the final loss function for the entire model. Here, loss functions with the same task properties share the same combination coefficients:

[0031]

[0032] here and It is the loss function for the arrow keypoint prediction task; while L det L push L pull and L offThis is the loss function for the corner keypoint prediction task; here α, β, γ and λ are the coefficient weights of the sub-loss function.

[0033] Preferably, if the chart is a nested chart, each element is associated with multiple locations across multiple layers in the feature pyramid, thereby enabling each element to be predicted by one or more locations, specifically:

[0034] For any feature level P k any position on Use the following formula:

[0035]

[0036] Map it back to obtain its representation in the input image The corresponding position on the graph is called the mapping position, and it is close to l. k The center of the receptive field; for any two feature levels P k and P j any two positions l k and l j , l' k and l' j They will not overlap; therefore, each mapping location is globally unique in the input image; secondly, for If there exists a position l from the feature pyramid such that l' and b (i) The following conditions must be met:

[0037] 0<ξ(l′,b (i) (30) of which

[0038]

[0039] Then l is b (i) Candidate predicted locations for the target; obviously, b (i) Multiple positions can be used, and all of these positions constitute b. (i) The set of candidate positions θ (i) Furthermore, if Make l i and b (i) The following conditions must be met:

[0040] γ k-1 <δ(l′,b (i) )≤γ k (32)

[0041] in

[0042]

[0043] Then l is b (i) At feature level Pk The true predicted location, here γ k (k = 1, 4, ..., 5) is a hyperparameter; in P k Predicting b (i) All positions contain b (i) Location set and Reflects b (i) Capacity across the entire feature pyramid;

[0044] In addition, for and Will be responsible for predicting b (i) The feature hierarchy is determined by the discriminant function δ(l′,b) (i) The supremacy and infimum of b are determined, and they reflect the upper and lower bounds of b. (i) The regression range is calculated using the following formula:

[0045]

[0046]

[0047] Therefore, if Then there must be In other words, any primitive in a flowchart can be predicted at two adjacent feature levels; for primitive detection in a flowchart, this feature can effectively prevent nested primitives and their nested primitives from being assigned to the same feature level at the same time, thus avoiding target stacking.

[0048] For any two objectives b (i) and b (j) ,pass Determine if b is 1 (i) Is b nested? (j) ,in

[0049]

[0050] Now, let b (t) For a nested primitive, all its nested primitives form its nested set. The purpose of multi-level nested primitive detection is to detect b (t) and N (t) All nested primitives are appropriately associated with their positions on the feature pyramid, while maintaining high recall for each primitive.

[0051] For the recognition of ordinary charts, the proposed method in this invention identifies each graphic element in the chart based on keypoint prediction. It determines the connection relationship between arrows and graphic elements by predicting the start and end keypoints of arrows, thereby recognizing the overall chart structure. Keypoint prediction is a distinctive object detection paradigm that classifies and locates objects by accurately predicting their keypoints. To further enhance the semantic information of object keypoints in the feature map, this invention proposes two keypoint pooling models. Both modules embed the keypoint information represented by the geometric contour features of graphic elements as prior knowledge into the feature map through pooling, effectively enhancing the keypoint information in the feature map. Finally, this invention also proposes an arrow direction prediction branch network for chart recognition tasks, used to predict the start and end keypoint information of arrows. This allows inference of the connection relationship between arrows and other graphic elements based on the arrow keypoints, making chart-level understanding possible. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating a hand-drawn chart recognition method based on key point prediction in one embodiment.

[0053] Figure 2 This is a schematic diagram of the overall architecture of DrawnNet in the embodiment;

[0054] Figure 3 This is a schematic diagram illustrating the pooling results of CICP and MICP on the same feature map in the embodiment;

[0055] Figure 4 This is a schematic diagram of the network structure for predicting the network branch at the top left corner point in the embodiment;

[0056] Figure 5 This is a schematic diagram illustrating the direction of the arrows in the example to predict the network branches;

[0057] Figure 6 This is a schematic diagram of the snowflake corner pooling module in the embodiment;

[0058] Figure 7 This is a schematic diagram of nested charts in the embodiment;

[0059] Figure 8 This is a schematic diagram of nested primitive recognition using a feature pyramid in the embodiment;

[0060] Figure 9 This is a schematic diagram illustrating the recognition performance of DrawnNet on three datasets in the example.

[0061] Figure 10 This is a schematic diagram of some defective samples in the FC-A dataset in the example;

[0062] Figure 11This is a schematic diagram showing the change of mAP with the number of iterations when different λ values ​​are used in the example. Detailed Implementation

[0063] To further understand the content of this invention, a detailed description of the invention will be provided in conjunction with the accompanying drawings and embodiments. It should be understood that the embodiments are merely illustrative and not limiting of the invention.

[0064] Example

[0065] like Figure 1 As shown, this embodiment provides a hand-drawn chart recognition method based on key point prediction, which includes the following steps:

[0066] 1. Detect each graphic element in the chart as a pair of key points, that is, jointly determine the upper left and lower right corner points of the bounding box of the graphic element;

[0067] 2. Perform pooling operations in parallel using Maximum Cross-Corner Pooling (MICP) and Cumulative Cross-Corner Pooling (CICP).

[0068] Third, perform feature fusion on the feature maps of each pooling process;

[0069] Fourth, for each arrow connecting two primitives, the arrow direction prediction network is used to predict the head and tail key points of the arrow, and the snowflake corner pooling SCP module is used to enhance the arrow key point information; in this way, the recognition of the entire graph structure is completed.

[0070] DrawnNet model

[0071] This embodiment proposes DrawnNet, a keypoint-based detection model based on state-of-the-art convolutional neural network technology. DrawnNet is designed to address CornerNet's inability to understand graph structures. By introducing new module mechanisms into DrawnNet to extend CornerNet's detection capabilities, these modules can effectively utilize existing prior knowledge within the graph, making the improved model architecture adaptable to hand-drawn graph recognition tasks. Specifically, this embodiment proposes two novel keypoint pooling modules to explicitly embed prior knowledge, such as geometric features present in the graph, into the feature map, and then fuse these features into keypoint prediction. Furthermore, to understand the graph structure, an arrow direction prediction branch is proposed, aiming to predict the arrow's direction by predicting the keypoints at the head and rear of the arrow.

[0072] In DrawnNet, this embodiment detects each primitive in the graph as a pair of keypoints, which together determine the top-left and bottom-right corners of the primitive's bounding box. Furthermore, for each arrow connecting two primitives, DrawnNet has a branch network called arrow direction prediction, which predicts the head and tail keypoints of the arrow; this pair of keypoints determines the direction the arrow is pointing. Therefore, through this branch, the structure of the graph can be fully understood. Figure 2 This demonstrates the overall architecture of DrawnNet. DrawnNet uses the Hourglass model as its backbone. Hourglass is a typical encoder-decoder structure, widely used in visual tasks such as pose estimation and image segmentation, keypoint detection, or pixel-level prediction.

[0073] Corner Key Point Prediction

[0074] In DrawnNet, primitives are represented as a pair of keypoints: the top-left corner and the bottom-right corner. However, feature maps often lack obvious local visual cues to indicate the potential locations of keypoints. To locate potential keypoints in CornerNet, this embodiment proposes a pooling method called Maximum Cross-Corner Pooling (MICP). The MICP module starts from a pixel and searches for the maximum activation of neurons in both the horizontal and vertical directions, then sums them as the pooling result at that point. In the graph, most primitives are rectangular outlines, and their corners clearly appear where several boundary lines intersect. Therefore, in DrawnNet, this embodiment extends the original corner pooling module by introducing another pooling reduction method, embedding these geometric features presented in the graph as explicit prior knowledge into the prediction of keypoints through pooling.

[0075] In DrawnNet, the corner pooling module proposed in this embodiment traverses each pixel in both the horizontal and vertical directions. If a neuron has the strongest response in its neighborhood within the pooled feature map, that location is likely a potential key corner, situated at the intersection of the horizontal and vertical pooling vectors. This embodiment refers to this corner pooling method as Intersection Corner Pooling (ICP). CornerNet uses max as the reduction function of the ICP module to calculate the final response, known as Maximum Intersection Corner Pooling (MICP); while in DrawnNet, this embodiment uses sum to accumulate all responses along both the vertical and horizontal axes, known as Cumulative Intersection Corner Pooling (CICP).

[0076] like Figure 3As shown, MICP and CICP are used to pool the same feature map. Figure 3 In this context, MICP and CICP are expected to capture the top-left corner of a rectangular pattern composed of responses where neurons are activated to 1. This rectangular visual pattern is undoubtedly a fundamental visual pattern that frequently appears in primitive objects within a graph. Figure 3 (b) clearly demonstrates that MICP failed to effectively capture the top-left corner (circled in solid line) because in the feature map of the MICP pooling result, the response intensity of other neurons in the neighborhood of the corner point is almost the same as that of the neuron itself, thus failing to effectively highlight the visual features of this key point in the pooled feature map. In contrast, Figure 3 The CICP module in (a) handles this situation in the feature map by maximizing the response of the corresponding neuron in its neighborhood (e.g., within 3×3 and 5×5 regions).

[0077] As mentioned above, to determine whether the activation value of a primitive is the top-left corner, CICP searches horizontally to the right for the top boundary of the target and vertically downwards for the leftmost boundary of the target; let F t and F l The input feature map is used for corner pooling. and F respectively t and F l The response at position (i,j) is determined using the H×W feature map; CICP determines whether the activation value of the neuron at position (i,j) is at the top left corner, and this is achieved by parallel accumulation on F. t All responses in the horizontal distribution of (i,j) and (i,H) in F, and the vertical distribution in F l The sum of (i,j) and (W,j) in T is used as T ij and L ij Finally, they are added together to generate the feature map F. CICP The calculation process can be expressed by the following formula:

[0078]

[0079]

[0080] The CICP pooling calculation method for the bottom right corner is similar to that for the top left corner. That is, all responses vertically distributed in (0,j) and (i,j) and all responses horizontally distributed in (i,0) and (i,j) are accumulated in parallel, and then the pooled results are summed.

[0081] Geometric feature fusion

[0082] The network structure of the network branch predicted by the top-left corner point is as follows: Figure 4As shown. Compared to CornerNet, the improvements in this embodiment include extending the corner pooling module in the corner prediction branch of DrawnNet, and proposing a method for feature fusion of feature maps from the pooling results of multiple pooling modules. Feature fusion is typically used to extract feature information from feature maps of different channels or spaces. Many visualization task models are already equipped with feature fusion modules. For example, Feature Pyramid Networks fuse features of multi-scale objects by connecting a pyramid of downsampled convolutional features.

[0083] In this embodiment, the feature fusion method used in DrawnNet is based on pooling. The proposed keypoint pooling method is first applied to the feature map of each branch by channel to fully utilize the geometric information in the image. Then, adaptive feature refinement is performed by multiplying, adding, or concatenating pooled feature maps using different pooling methods. As mentioned earlier, this embodiment proposes a CICP module as a supplement to the MICP module, especially enriching the semantic features of corner information in the feature maps. Therefore, in order to enable the subsequent specific detection tasks to effectively utilize the corner semantic information already encoded in the feature maps, this embodiment refers to the design of the ResNet residual network. First, two parallel 128-channel 3×3 Conv-BN-ReLU layers are used to replace the original 3×3 convolutional module in the residual module to construct the entire corner pooling module to process features from the backbone network. Then, MICP and CICP are used in parallel to process the two feature maps (e.g., the top left corner F). t and F l Pooling operations are performed, with one pooling operation for vertical pooling and the other for horizontal pooling. The feature maps obtained after each pooling are then summed to obtain F. CICP and F MICP .

[0084] Arrow direction prediction

[0085] In flowchart-like arrow connection diagrams, structure recognition involves identifying which primitives each arrow connects to and which direction each arrow points. While object detection models can classify and locate primitives in a diagram using bounding boxes, this information is insufficient for diagram structure recognition. This embodiment finds that predicting arrow keypoint information can effectively solve this problem. The planar vector formed by the head and tail keypoints of an arrow indicates its direction. Furthermore, the positional relationship between each arrow keypoint and surrounding primitives can predict which primitives the arrow connects to. To predict arrow keypoints, this embodiment adds a parallel arrow direction prediction network branch to the DrwanNet backbone model. Figure 5 The network is shown. The arrow network reuses the feature map of the autonomous network and uses the SCP module to enhance the arrow key point information.

[0086] The output of the arrow direction prediction network branch is similar in function to that of the corner prediction network branch, both including heatmap generation, semantic embedding, and position offset prediction. The arrow direction prediction network adopts a skip connection method similar to the ResNet residual network module, with the SCP module inserted in the middle of the skip connections, which can effectively enhance the semantic features of key points in the feature map.

[0087] Snowflake corner pooling

[0088] The direction of the arrow is determined by locating its head and tail, and DrawnNet uses its inherent intelligence to perform keypoint detection. To address this issue, this embodiment proposes a Snowflake Corner Pooling (SCP) module to capture richer and more easily recognizable visual arrow patterns. Figure 6 It demonstrates the principles of SCP.

[0089] Let F be the characteristic map of SCP, F ij This is the response at position (i,j) in F. Using the H×W feature map, the pooled feature map F... SCP The activation response of the neuron at position (i,j) in the SCP is expressed by the following formula:

[0090]

[0091] Arrow key point prediction

[0092] In the arrow direction prediction network, heatmaps, offsets, and semantic embeddings are used to predict the head and tail keypoints of the arrow, as detailed below:

[0093] Let P cij It predicts the probability of class c at position (i,j) in the heatmap, y cij It's a ground-truth heatmap with C channels; then, the class loss of the arrow keypoints is estimated using the Focal Loss function.

[0094]

[0095] Where N is the number of objects in the image, and α is a hyperparameter that controls the contribution of each point;

[0096] The position of the arrow keypoint is slightly corrected by predicting the offset. Let (x, y) be the position in the image. This is its downsampling position in the heatmap, where s is the downsampling factor; estimate the deviation of the arrow key point k between these two positions:

[0097]

[0098] The prediction bias and ground-truth bias can be calculated using the Smooth-L1 Loss function:

[0099]

[0100] A diagram can include multiple arrows, thus allowing for the prediction of head and tail keypoints for multiple arrows. Therefore, it is necessary to determine which pair of head and tail keypoints belong to the same arrow. The method in this embodiment, also a semantic embedding used in CornerNet, matches the two keypoints with the highest similarity together; let e hk For the semantic embedding of the head keypoint of arrow k, e tk The semantic embeddings for the tail keypoints are all four-dimensional vectors; the network is trained using pull loss to match the keypoints, and push loss is used to remove keypoints that do not belong to the same object.

[0101]

[0102]

[0103]

[0104] Finally, the loss functions of all network branches are linearly combined to obtain the final loss function for the entire model. Here, loss functions with the same task properties share the same combination coefficients:

[0105]

[0106] here and It is the loss function for the arrow keypoint prediction task; while L det L push L pull and L off This is the loss function for the corner keypoint prediction task; here α, β, γ and λ are the coefficient weights of the sub-loss function.

[0107] Nested chart recognition

[0108] Besides the recognition of ordinary charts mentioned above, there is a special type of chart: nested charts. These are mainly program flowcharts, where nesting structures play a crucial role, typically representing program logic such as loops and selections. The most challenging aspect of recognizing nested charts lies in identifying the nested graphical elements. Nested graphical elements in a nested chart refer to a single graphical element containing one or more other graphical elements. These nested elements are called the nested graphical elements of the given element, reflecting complex semantic logic in a nested manner. From a digital image perspective, nested graphical elements appear as multiple elements "stacked" together, with the innermost layer being the nested graphical element and the outermost layer being the nesting graphical element, such as... Figure 7 As shown. Since the category of the outer nested primitives remains unchanged regardless of the number and category of the inner nested primitives and their logical structure, the existence of the inner primitives often brings great difficulties to the model's recognition of the outer nested primitives. The model needs to have a very strong generalization ability to ensure that the recognition of nested primitives is not affected by the changes of its inner primitives. To this end, this embodiment proposes a data augmentation algorithm for the training phase to improve the generalization of the model.

[0109] The feature pyramid operates on the decoder part of the backbone model, hourglass, with the top three feature maps serving as input to the feature pyramid. Let... It is the feature level of the i-th layer in the feature pyramid (e.g., Figure 8 As shown), and the sampling coefficient to this layer is s. k The ground-truth bounding box of the nested icon is defined as B = {B}. (1) B (2) ,…},here They are B (i) The coordinates of the top left corner and the bottom right corner.

[0110] The method used in this embodiment is to associate each primitive with multiple locations in multiple layers of the feature pyramid, thereby enabling each primitive to be predicted by one or more locations, specifically:

[0111] For any feature level P k any position on Use the following formula:

[0112]

[0113] Map it back to obtain its representation in the input image The corresponding position on the graph is called the mapping position, and it is close to l. k The center of the receptive field; for any two feature levels P k and P jany two positions l k and l j , l' k and l' j They will not overlap; therefore, each mapping location is globally unique in the input image; secondly, for If there exists a position l from the feature pyramid such that l' and b (i) The following conditions must be met:

[0114] 0<ξ(l′,b (i) (48) in

[0115]

[0116] Then l is b (i) Candidate predicted locations for the target; obviously, b (i) Multiple positions can be used, and all of these positions constitute b. (i) The set of candidate positions θ (i) Furthermore, if Make l i and b (i) The following conditions must be met:

[0117] γ k-1 <δ(l′,b (i) )≤γ k (50)

[0118] in

[0119]

[0120] Then l is b (i) At feature level P k The true predicted location, here γ k (k = 1, 4, ..., 5) is a hyperparameter; in P k Predicting b (i) All positions contain b (i) Location set and Reflects b (i) Capacity across the entire feature pyramid;

[0121] In addition, for and Will be responsible for predicting b (i) The feature hierarchy is determined by the discriminant function δ(l′,b) (i) The supremacy and infimum of b are determined, and they reflect the upper and lower bounds of b. (i) The regression range is calculated using the following formula:

[0122]

[0123]

[0124] Therefore, if Then there must be In other words, any primitive in a flowchart can be predicted at two adjacent feature levels; for primitive detection in a flowchart, this feature can effectively prevent nested primitives and their nested primitives from being assigned to the same feature level at the same time, thus avoiding target stacking.

[0125] For any two objectives b (i) and b (j) ,pass Determine if b is 1 (i) Is b nested? (j) ,in

[0126]

[0127] Now, let b (t) For a nested primitive, all its nested primitives form its nested set. The purpose of multi-level nested primitive detection is to detect b (t) and N (t) All nested primitives are appropriately associated with their positions on the feature pyramid, while maintaining high recall for each primitive. Therefore, for exist and Ω (t) There are two possible scenarios, as shown below:

[0128] 1) If but and b (t) The actual predicted positions come from two different feature levels, so they can both be effectively predicted by positive samples and thus learned by the model during training.

[0129] 2) If but The positions in the sample are fuzzy because they are all predictable. and b (t) This situation is attributed to the fact that the two objects are not significantly different in size. In this case, DrawnNet specifies that the primitives with the smallest predicted size at these locations are... therefore The actual location set is And b (t) The actual location set is also The value of b may be very close to 1, causing b to... (t) Number of positive samples Smaller, which will seriously affect b (t) The recall rate. Fortunately, in the nested graph dataset of this embodiment, there is a significant size difference between nested primitives and nested primitives, so this situation is unlikely to occur.

[0130] Therefore, for nested primitives and nested primitives, DrawnNet can assign them to appropriate feature level locations for prediction based on their own size. Since the feature pyramid has a rich variety of predictable locations, each primitive can be predicted at multiple locations, resulting in high recall for multi-level object detection in flowcharts. However, other non-dense prediction models such as YOLOv3 cannot handle clustered objects, leading to extremely low recall on dense images.

[0131] Data augmentation for nested graphs

[0132] To improve the diversity of nested chart datasets and more accurately identify nested primitives within nested charts, this embodiment proposes a data augmentation algorithm (Algorithm 1). Intuitively, the main function of this algorithm is to discard each primitive in a nested chart with a certain probability, thereby deriving a large number of different nested charts from a single nested chart. Specifically, the algorithm assumes that the probability of discarding each primitive follows a uniform distribution on the interval 0-1, meaning that the primitive will be discarded when the random variable X ≥ λ. Therefore, it can derive at most... Given a graph containing only k (0≤k≤n) primitives, a total of 2... n Each nested chart in the training set contains an average of 10 primitives, meaning each nested chart can generate a maximum of 1024 different nested charts. This is equivalent to expanding the training dataset by 1024 times, a huge number. Even if the dataset only contains a few hundred training samples, the data augmentation algorithm proposed in this embodiment can generate hundreds of thousands of training samples, which is more than sufficient for training convolutional network-based object detection models, since even the largest general-purpose object detection dataset, COCO-2017, has no more than 120,000 training samples.

[0133] Furthermore, the data augmentation algorithm proposed in this embodiment has an advantage over other data augmentation algorithms in the image domain. This algorithm has only one hyperparameter: the probability threshold λ for determining whether each primitive is discarded. The setting of λ is closely related to the number of primitives retained after each augmentation of each chart; the larger the value of λ, the more primitives are discarded each time data augmentation is performed, and vice versa. In the experiments of this embodiment, the value of λ is typically one of three possibilities: 0.25, 0.50, and 0.75. If the value of λ is too large (λ>0.75), the augmented chart will appear too empty due to the discarding of too many primitives, and detecting these empty charts would undoubtedly waste computational resources. Conversely, if the value of λ is too small (λ<0.25), too few primitives are discarded each time, which weakens the data augmentation effect. Therefore, when the value of λ is 0.5, a relatively good data augmentation effect can be achieved. It can be theoretically proven that when λ = 0.5, the probability of discarding half of the graphic elements in each chart is the highest, that is, the probability of retaining half of the graphic elements in each chart after data augmentation is the highest.

[0134]

[0135] Experimental evaluation

[0136] This embodiment evaluates DrawnNet on three publicly available hand-drawn graph datasets, two of which describe flowcharts (FC-A and FC-B) and one is a finite automata dataset (FA).

[0137] Released in 2011, FC-A serves as a benchmark database for hand-drawn flowcharts. It consists of 419 diagrams drawn by 35 authors from 28 predefined templates (248 for the training set and 171 for the test set). The biggest drawback of this dataset is the lack of annotations regarding diagram structure and time information; only single primitives are provided. Therefore, the quality of these diagrams is low, making it difficult to evaluate online recognition methods.

[0138] FC-B, a supplement to FC-A, was released in 2016 and contains 672 images (280 for training, 196 for testing, and 196 for validation). These samples come from 28 pattern templates drawn by 24 authors. Some templates reference FC-A and retain commonly used algorithmic features. In addition to chart structure annotations, its annotations also include arrow directions.

[0139] FA was released simultaneously with FC-B, containing a total of 300 charts (132 for the training set, 84 for the test set, and 84 for the validation set), generated from 12 templates drawn by 25 authors. The dataset has four categories: state (single circle), final state (two concentric circles), test, and arrows. The arrows are typically curved. Like FC-B, its annotations also include the direction of the arrows.

[0140] Furthermore, to evaluate the performance of the DrawnNet model in recognizing nested graphs, the authors' research group constructed a nested graph dataset, which is a program flowchart type. The authors' research group collaborated with a software company to develop software for drawing and annotating nested graphs. This embodiment first investigated the syntax mechanisms of mainstream programming languages ​​including C, C++, and Java, while also considering widely used programming languages ​​from the last century such as Fortain. From these languages, nine basic program structures were identified: begin, end, input / output, if-else, assign, process, while, for, and do-while. These nine predefined basic program structures were used as the basic primitives of the nested graph dataset, meaning the dataset contains only these nine types of primitives. Among them, if-else, while, for, and do-while are four nestable primitives, meaning any primitive among the nine can be nested within these four primitives; while begin, end, input / output, assign, and process are meta-primitives, meaning they cannot contain other primitives and can only be nested within nestable primitives.

[0141] In addition to defining the basic program structure, to construct a high-quality nested graph dataset so that the program algorithms described by the graphs can effectively reflect the business logic of existing programs in the current software engineering field, the authors of this embodiment conducted a sampling survey of code segments from open-source code repositories such as GitHub. From this, three metrics for characterizing code complexity were statistically analyzed: the number of basic program structures N, the nesting depth D, and the category distribution C of basic program structures. Based on the sampling results of these three metrics, this embodiment formulated corresponding quantitative standards as the overall distribution of the dataset to guide the graph drawing.

[0142] Ultimately, this embodiment constructs a nested graph dataset with a total of 600 samples. Each sample consists of two parts: the flowchart image and the annotation information for each element in the flowchart, which can be represented as a tuple.<image,label> The label represents the class and location of each graph element. The nested graph dataset is then divided into training and testing datasets in a 7:3 ratio.

[0143] In this embodiment, Precision and Recall are used to evaluate the performance of the proposed method on primitive recognition and graph recognition tasks. Furthermore, for nested graph recognition, to measure the accuracy of the model in object detection, this embodiment uses AP and mAP, referencing general object detection tasks, to evaluate the model's accuracy in nested graph recognition.

[0144] Precision: Precision measures the accuracy of a model in a task relative to the overall accuracy of all predictions.

[0145] Recall: Recall is the rate of recall, which measures the coverage of the model in a task when it correctly predicts the results compared to the total coverage of all results to be predicted.

[0146] AP: AP refers to "Average Precision," first introduced in VOC2007. It is defined as the average detection precision at different recall rates and is usually evaluated in a class-specific manner.

[0147] mAP: mAP is the "average accuracy" across all classes, and is widely used to compare the performance of all object classes in the evaluation of general object detection problems.

[0148] Table 1 shows a performance comparison of DrawnNet with other online and offline graph recognition systems. Online recognition systems rely on analyzing and modeling temporal stroke information to achieve offline graph recognition, largely utilizing primitive segmentation and classification. These low-level representations and local understandings are highly sensitive to benchmark datasets; low-quality or inaccurate annotations can negatively impact recognition accuracy. Conversely, deep learning-based object detection systems can be strategically designed to achieve robust and superior performance in graph recognition tasks.

[0149] Table 1 shows the chart recognition accuracy on various datasets.

[0150]

[0151] Figure 9 The results show the performance of DrawnNet on three benchmark sets, with the head and tail keypoints of the arrows marked.

[0152] This embodiment also illustrates the performance of DrwanNet on primitive recognition tasks across various benchmark datasets. Tables 2, 3, and 4 present the primitive recognition results for each benchmark dataset. Overall, DrwanNet achieves excellent recognition results for several primitive shapes, which can be explained by the fact that the differences in primitive shape and size are much smaller than those for arrows and text.

[0153] On the FC-A dataset (Table 2), DrawnNet exhibits higher primitive recognition recall and precision. However, DrawnNet performs slightly worse on the arrow category among all categories. Through examination of the FC-A training set, the authors found that some samples used circles instead of crosshairs to label the arrows. Figure 10 As shown, this can mislead the model's classification decisions, causing it to confuse the circle with other categories such as text, thus misleading the model's learning. Furthermore, another problem exists: some of these circles have too low a resolution, such as... Figure 10 As shown, after a series of downsampling operations on the feature map, the resolution becomes increasingly smaller. Therefore, as mentioned earlier, this severely hinders feature extraction performed by the corner pooling module in this embodiment.

[0154] Table 2 shows the accuracy and recall of FC-A primitive recognition for each category.

[0155]

[0156] Table 3 shows that DrawnNet can accurately identify primitives in graphs. Interestingly, Arrow R-CNN provides complementary results in terms of accuracy and recall for the Data and Process categories. Specifically, it achieves 100% accuracy and 94.9% recall for the Data category, while its accuracy and recall for the Process category are almost opposite. This is partly because, despite using FPN, the underlying network of Arrow R-CNN, Faster R-CNN, lacks a mechanism to learn fine-grained discriminative features between similar primitives of different categories. Both Data and Process primitives are quadrilaterals, but the two parallel sides of the Data primitive are slightly slanted, unlike the perpendicular sides of the Process primitive.

[0157] Table 3. Accuracy and Recall of FC-B Primitive Recognition in Each Category

[0158]

[0159] As shown in Table 4, DrawnNet completely recognizes the shapes of the State and Final State in the FA test set. Because this category is relatively small and the features of each category are very stable, it is not difficult to identify them.

[0160] Table 4. Accuracy and Recall of FA Primitive Recognition in Each Category

[0161]

[0162] ablation experiment

[0163] This embodiment conducted ablation studies on each benchmark dataset to further quantify the impact of the proposed keypoint pooling method on chart recognition. Table 5 shows that combining the CICP and SCP modules can significantly improve the accuracy of chart recognition. It should be noted that while this embodiment proposes an arrow direction branch network, it must be considered in the ablation study because its responsibility is to predict arrow keypoints, which is indispensable for chart recognition tasks. Therefore, in the ablation study, this embodiment only ablates the SCP module in the arrow direction branch network, rather than ablating the entire branch network.

[0164] Clearly, the use of the SCP module effectively improves the accuracy of chart recognition because its ingenious pooling method helps predict arrow keypoints. Whether or not arrow keypoints are predicted is crucial for subsequent chart recognition. Furthermore, the CICP module also helps DrawnNet perceive where corner keypoints within a rectangular outline might be located, but its potential might not be fully realized if used alone. After all, the accuracy of arrow keypoint prediction is critical to the correctness of chart understanding.

[0165] Table 5 Ablation Results

[0166]

[0167] Nested chart recognition experiment

[0168] The accuracy of this method was evaluated from two aspects: overall performance and the generalization of the data augmentation algorithm. To evaluate the overall performance, FCOS was trained on 100 epochs, and the trained model was evaluated in flowchart detection. Table 6 shows the results of the test split. As can be seen from Table 6, data augmentation significantly improved the accuracy of flowchart detection, with the highest mAP reaching 77.98, which is 3.65 percentage points higher than before data augmentation. Moreover, the improvement in AP across different IoU thresholds was very significant.

[0169] Table 6 shows the overall performance on the primitive recognition task.

[0170]

[0171] To further explore the effectiveness of data augmentation, this embodiment trains the model on the training set (interval of 10) with different λ values ​​and incremental iterations, and then observes the changes in mAP on test splits. Figure 11 shows the results. As analyzed above, mAP does indeed increase with the increase of epochs and the adoption of the data augmentation algorithm in this embodiment. This trend continuously enhances the model's generalization ability because the continuous expansion of training data diversifies its form, and with the increase of epochs, the model can witness more different training data, which is consistent with the expectations of this embodiment. In addition, as can be seen from Figure 11, different λ values ​​do have a certain impact on mAP. As analyzed above, the data augmentation effect is best when λ = 0.50, and the experimental results also verify this.

[0172] Therefore, with the increase in iterations and the adoption of data augmentation, the method in this embodiment performs surprisingly well in the flowchart detection task, and the corresponding metrics continue to rise. This sufficiently demonstrates the power of the data augmentation algorithm in this embodiment, which greatly helps DrwanNet achieve amazing results in these two tasks!

[0173] Data-enhanced generalization ability

[0174] To evaluate the overall performance of the model in this embodiment in recognizing each category and multi-level nested primitives, further experiments were conducted. Table 7 provides the mAP for each category. It can be seen that without data augmentation, the mAP for each category ranges from 55.45 to 88.89, with the mAP for the four nested primitives being significantly lower than that for the other five basic primitives. Clearly, data augmentation significantly improves the performance of the four nested primitives; for example, if-else statements and while loops all improve the performance by more than 5.49. This is because recognizing nested primitives is more challenging than recognizing basic primitives. Therefore, the benefits of data augmentation are mainly concentrated on nested primitives, which increases the average mAP of the four nested primitive categories to 65.01, 6.77 higher than without augmentation (as shown in Table 7).

[0175] Table 7 mAP for each category

[0176]

[0177] Table 8 shows the mAP of nestable primitives with different nesting depths. It can be seen that when the nesting depth is 1 or 2, the model's performance can maintain a high mAP value, while data augmentation has the most significant effect on improving the model's mAP when the nesting depth increases to 3, 4, or even 5. When the nesting depth is 3 or 4, the increment of mAP across all four nestable categories remains above 6.42, and their average mAP increases by 9.08 and 6.87 respectively across all four categories.

[0178] Table 8 Comparison of mAP for nesting categories with different nesting depths under data augmentation.

[0179]

[0180] Therefore, the data augmentation algorithm proposed in this embodiment generates a large amount of diverse training data for the model, which evolves with different internal structures. By learning these variants, the model can adapt to multi-level control flow, particularly the four nestable primitives with deeply nested sub-primitives shown in the figure.

[0181] The present invention and its embodiments have been described above illustratively. This description is not restrictive, and the figures shown are only one embodiment of the present invention; the actual structure is not limited thereto. Therefore, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the present invention, such designs should fall within the protection scope of the present invention.

Claims

1. A hand-drawn chart recognition method based on key point prediction, characterized in that: Includes the following steps:

1. Detect each graphic element in the chart as a pair of key points, that is, jointly determine the upper left and lower right corner points of the bounding box of the graphic element; 2. Perform pooling operations in parallel using Maximum Cross-Corner Pooling (MICP) and Cumulative Cross-Corner Pooling (CICP). In Cumulative Cross-Pooling (CICP), to determine whether the activation value of a primitive is the top-left corner, CICP searches horizontally to the right for the top boundary of the target and vertically downwards for the leftmost boundary of the target; let F t and F l The input feature map is used for corner pooling. and F respectively t and F l The response at position (i,j) is analyzed; using the H×W feature map, CICP determines whether the activation value of the neuron at position (i,j) is at the top left corner, and this is determined by parallel accumulation on F. t All responses in the horizontal distribution of (i,j) and (i,H) in F, and the vertical distribution in F l The sum of (i,j) and (W,j) in T is used as T ij and L ij Finally, they are added together to generate the feature map F. CICP The calculation process is expressed by the following formula: The CICP pooling calculation method for the bottom right corner is similar to that for the top left corner, that is, all responses vertically distributed in (0,j) and (i,j) and all responses horizontally distributed in (i,0) and (i,j) are accumulated in parallel, and then the pooled results are summed. Third, feature fusion is performed on the pooled feature maps. Referring to the design of the ResNet residual network, firstly, two parallel 128-channel 3×3 Conv-BN-ReLU layers are used to replace the original 3×3 convolutional module in the residual module. Then, MICP and CICP are used in parallel to perform pooling operations on these two feature maps, one for vertical pooling and the other for horizontal pooling. Finally, the pooled feature maps are summed to obtain F. CICP and F MICP ; Fourth, for each arrow connecting two primitives, the arrow direction prediction network is used to predict the head and tail key points of the arrow, and the snowflake corner pooling SCP module is used to enhance the arrow key point information; in this way, the recognition of the entire graph structure is completed. In the snowflake corner pooling SCP module: Let F be the characteristic map of SCP, F ij It is the response at position (i,j) in F; using the H×W feature map, the pooled feature map F SCP The activation response of the neuron at position (i,j) in the SCP is expressed by the following formula: In the arrow direction prediction network, heatmaps, offsets, and semantic embeddings are used to predict the head and tail keypoints of the arrow, as detailed below: Let P cij It predicts the probability of class c at position (i,j) in the heatmap, y cij It's a ground-truth heatmap with C channels; then, the class loss of the arrow keypoints is estimated using the Focal Loss function. Where N is the number of objects in the image, and α is a hyperparameter that controls the contribution of each point; The position of the arrow keypoint is slightly corrected by predicting the offset. Let (x, y) be the position in the image. This is its downsampling position in the heatmap, where s is the downsampling factor; estimate the deviation of the arrow key point k between these two positions: The prediction bias and ground-truth bias are calculated using the Smooth-L1 Loss function: To determine which pair of leading and trailing keypoints belong to the same arrow, semantic embedding is used to match the two keypoints with the highest similarity; let e hk For the semantic embedding of the head keypoint of arrow k, e tk The semantic embeddings for the tail keypoints are all four-dimensional vectors; a pull loss network is used to train the keypoints for matching, and a push loss network is used to remove keypoints that do not belong to the same object. Finally, the loss functions of all network branches are linearly combined to obtain the final loss function for the entire model. Here, loss functions with the same task properties share the same combination coefficients: here and It is the loss function for the arrow keypoint prediction task; while L det L push L pull and L off This is the loss function for the corner keypoint prediction task; here α, β, γ and λ are the coefficient weights of the sub-loss function.

2. The hand-drawn chart recognition method based on key point prediction according to claim 1, characterized in that: The arrow direction prediction network uses a skip connection method, with the SCP module inserted in the middle of the skip connection.

3. The hand-drawn chart recognition method based on key point prediction according to claim 1, characterized in that: If the chart is a nested chart, each primitive is associated with multiple locations across multiple layers in the feature pyramid, thus enabling each primitive to be predicted by one or more locations, specifically: For any feature level P k any position on Use the following formula: Map it back to obtain its representation in the input image The corresponding position on the graph is called the mapping position, and it is close to l. k The center of the receptive field; for any two feature levels P k and P j any two positions l k and l j ,l' k and l' j They will not overlap; therefore, each mapping location is globally unique in the input image; secondly, for If there exists a position l from the feature pyramid such that l' and b (i) The following conditions must be met: 0<ξ(l′,b (i) ) (12) in Then l is b (i) Candidate predicted locations for the target; obviously, b (i) Multiple positions can be used, and all of these positions constitute b. (i) The set of candidate positions θ (i) Furthermore, if Make l i and b (i) The following conditions must be met: c k-1 <δ(l′,b (i) )≤γ k (14) in Then l is b (i) At feature level P k The true predicted location, here γ k (k = 1, 4, ..., 5) is a hyperparameter; in P k Predicting b (i) All positions contain b (i) Location set and Reflects b (i) Capacity across the entire feature pyramid; In addition, for and Will be responsible for predicting b (i) The feature hierarchy is determined by the discriminant function δ(l′,b) (i) The supremacy and infimum of b are determined, and they reflect the upper and lower bounds of b. (i) The regression range is calculated using the following formula: Therefore, if Then there must be In other words, any primitive in a flowchart can be predicted at two adjacent feature levels; for primitive detection in a flowchart, this feature can effectively prevent nested primitives and their nested primitives from being assigned to the same feature level at the same time, thus avoiding target stacking. For any two objectives b (i) and b (j) ,pass Determine if b is 1 (i) Is b nested? (j) ,in Now, let b (t) For a nested primitive, all its nested primitives form its nested set. The purpose of multi-level nested primitive detection is to detect b (t) and N (t) All nested primitives are appropriately associated with their positions on the feature pyramid, while maintaining high recall for each primitive.

Images