A network graph data extraction method based on attention learning

Abstract

The present invention discloses a network graph data extraction method based on attention learning, which is characterized in that a text extraction deep neural network is used to extract text data in a graph, and a semantic segmentation network with an attention module is used to extract pixel features of a network graph visualization graph, and the data of nodes and connection relationships are calculated to restore the data structure of the original network graph. Compared with the prior art, the present invention has a better solution to the data mining problem of the high-level visual coding of the network graph, solves the identification problem of the connection line through the attention mechanism, improves the robustness of the model, and can extract data from the bitmap of the network graph visualization graph in various practical application scenarios such as data conversion, visual design style switching, and intellectual property protection, and has high practical value and good development prospects.

Description

Technical Field

[0001] The present invention relates to the technical field of graph data extraction, and in particular to a method for extracting raw data of a network graph based on attention learning. Background Art

[0002] Data visualization images can help people get data features faster. Most charts are stored in the form of bitmap images and published on various media. Getting their original data is a complex task. Restoring the original data of charts has become an important research because its development directly promotes the field of human-computer interaction.

[0003] Network diagrams refer to relationship diagrams with text information and different styles, including mind maps, modeling diagrams, flow charts, etc. They are high-level visual codes that are easy for humans to understand but difficult for machines to decode. Studies have shown that there are some methods to solve the problem of obtaining raw data, but these methods are only applicable to some simple charts, such as pie charts, bar charts, and line charts. Network diagrams have high dimensionality, complex data types, and difficult-to-identify connecting lines. There is also a problem of matching text and graphics. Summary of the invention

[0004] The purpose of the present invention is to design a network graph data extraction method based on attention learning to address the deficiencies of the prior art. The GraphDecoder framework is adopted to construct a text extraction neural network and an optical character recognition system to extract text from graphs, thereby automatically extracting raw data from network graph images. The method can be applied to many scenarios, such as mind maps, flow charts, ER diagrams, hierarchical structure diagrams, etc. Users can redesign and modify data on the system interface, which improves the robustness of the network and effectively solves the problem of extracting network graph data from bitmaps.

[0005] The object of the present invention is achieved by: a network graph data extraction method based on attention learning, which is characterized by using the framework of GraphDecoder to construct a text extraction neural network and an optical character recognition system to extract text from a graph, and the extraction of network graph data specifically includes the following steps:

[0006] Step a: Input a raster image I of a network graph of size H×W, wherein the raster image I is a pixel value matrix of size H×W, where H is the number of pixels per column in the vertical direction of the image I, and W is the number of pixels per row in the horizontal direction of the image I.

[0007] Step b: Construct a semantic dataset of the network graph.

[0008] Step c: training a semantic segmentation neural network model based on U-Net and with an attention module on the semantic data set of the network graph; the neural network includes a feature extraction part, an upsampling part, an attention module AttentionGate, and an output layer of a softmax model;

[0009] The feature extraction part of the semantic segmentation neural network includes: four convolution layers for extracting network graph visualization images, each convolution kernel uses a size of 3×3, the first two convolution layers are subjected to maximum pooling after two convolutions, and the last two convolution layers are subjected to maximum pooling after three convolutions; the upsampling part of the neural network uses the AttentionGate module to obtain an attention matrix α of the same size as the original resolution on feature maps of different scales, and then multiplies and connects it with the feature map corresponding to the feature extraction part, and finally a convolution layer with a 1×1 convolution kernel is used to obtain the semantic category value prediction results of all pixels.

[0010] The semantic segmentation network is collaboratively trained using the joint loss function defined by the following formula:

[0011]

[0012] Said The function defined by the following formula b is used:

[0013]

[0014] Said Use the following c-style defined function:

[0015]

[0016] Where: ω is the weight of each pixel class, the default weight of the background class is 0.8, the default weight of the connecting line class is 1.25, and the default weight of other node classes is 1; y i is the classification result of pixel i obtained by the segmentation network; y′ i is the true label of the pixel i category; in training, the loss function takes into account the balance between too many pixels in the background class and too few pixels in the connecting line class.

[0017] Step d: Train a neural network model for text extraction; the neural network includes a text positioning part CTPN and a text recognition part CRNN.

[0018] Step e: Apply the neural network model for text extraction in step d to the image I input in step a, locate the features of the text information from the CTPN, and then output the information array TextArr of each text box t from the output layer of the CRNN neural network t. The array contains {t x , t y , t w , t h , t a , Text, Confidence}, where: t x , t y is the coordinate of the center point of the text box; t w , t h is the width and height of the text box; t a is the inclination of the text box; Text is the text content of the text box; Confidence is the confidence of the text box. By default, values ​​higher than 0.95 are credible.

[0019] Step f: remove the text box portion in image I, fill it with the background color block of the text box, and perform dilation processing with Kernel = (2, 2) on it.

[0020] Step g: Input the image I preprocessed in step f into the semantic segmentation network in step c, and output a semantic map I of size H×W s , I s Each pixel value y in i Represents the category to which pixel i belongs, which is divided into 5 categories, and is represented by the following formula d: background class, connecting line class, rectangular node class, ellipse node class and diamond node class;

[0021]

[0022] Step h: Based on the input image I, the text information array TextArr obtained in step e t And the semantic graph I obtained in step g s , to perform data analysis of connected areas, the calculation steps are as follows:

[0023] h1: Change y i = 0 is assigned the color average of the pixels to C back ;

[0024] h2: Extract y on I i =1, 2, 3 connected components, these connected components are node classes, O p , C p , H p , W p , T p They are the coordinate value of the central pixel of the connected component, the color RGB value of the central pixel, the pixel height, the pixel width and the y value of the central pixel. i value;

[0025] h3: extract y on I i=1, 2, 3 connected components, these connected components are connected line types, and the y of the edge pixels of these connected components i Value, calculate the connection confidence between the connected component and each node The top two nodes are considered connected;

[0026] h4: TextArr t Text data with confidence higher than 0.95 is assigned to the nearest connected component, and the font size F is 0.75t h .

[0027] C back Represents the RGB value of the background color of the network diagram; p , H p , W p , C p , T p Indicates the center coordinate value, height value, width value, color RGB value and node shape type (rectangle, ellipse or diamond) of the pth node; F indicates the font size; Indicates the confidence that the node p and the connecting line q are connected. The initial value defaults to 0, and the confidence increases by 1 for each overlapping pixel.

[0028] Step i: Output the node information Node and connection information Link of the network diagram to obtain the original data extracted by network visualization. Users can redesign or modify the original data extracted by network visualization to obtain a visualization design diagram with a better layout or color scheme.

[0029] This method is also applicable to large-scale high-resolution network visualization graphs. The algorithm steps are as follows:

[0030] I-1: Input I into the semantic segmentation network to obtain the semantic graph y entire ;

[0031] I-2: Cut I into M pieces I m , m = 1, 2, ..., M;

[0032] I-3: Will I m Input into the semantic segmentation network to obtain the semantic graph y m , and y m Merge to y piece ;

[0033] I-4: The final semantic graph I s The y value of each pixel i The value is a function defined by the following formula:

[0034]

[0035] Where: α is the proportion of the complete semantic map or the semantic map after segmentation and splicing, the minimum weight is 0.5, and it is proportional to H×W; In the method, this function takes into account the different inherent characteristics of node-type and connection-line-type pixels in the convolutional neural network.

[0036] The present invention adopts the framework of GraphDecoder, which can automatically extract raw data from network graph images and solve the problem of extracting network graph data from bitmaps. A text extraction neural network and an optical character recognition system are used to extract text from the graph, and a neural network-based segmentation deep neural network is constructed. The backbone network and loss function are designed according to the characteristics of the graph, and a module with an attention mechanism is added to the decoder to improve the robustness of the network. In the process of learning to imitate human perception of network graphs, the extracted data is analyzed to obtain network relationship data. A complete and usable system is built. After the user uploads the raster image, the system extracts its underlying data. The user can redesign and modify the data on the system interface. The system can be applied to many scenarios, such as mind maps, flow charts, ER diagrams, hierarchical structure diagrams, etc.

[0037] Compared with the prior art, the present invention has better solved the data mining problem of network graphs, which are high-level visual codes. It solves the problem of identifying connecting lines through the attention mechanism, improves the robustness of the model, and can extract data from bitmaps of network graph visualization charts in various practical application scenarios such as data conversion, visual design style switching, and intellectual property protection. It has high practical value and good development prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of the process of the present invention;

[0039] Figure 2 A schematic diagram of an embodiment. DETAILED DESCRIPTION

[0040] See attached Figure 1 , the present invention performs data extraction of the network visualization graph according to the following steps:

[0041] Step 1: Input a network visualization bitmap I, i.e., a raster image I;

[0042] Step 2: Extract text information through the text extraction network and output text data TextArr t , and remove the text information pixels in I, and output the preprocessed image I;

[0043] Step 3: Input I into the semantic segmentation network and output the semantic graph I s ;

[0044] Step 4: I s ,I,TextArr t Perform connected component analysis algorithm;

[0045] Step 5: Output the network diagram node information Node and connection information Link.

[0046] The present invention is further described in detail below by taking the extraction of a mind map as an example.

[0047] Example 1

[0048] Step 1: Input a mind map I of size H×W, where the image I is a pixel value matrix of H×W, where H is the number of pixels per column in the vertical direction of the image I, and W is the number of pixels per row in the horizontal direction of the image I.

[0049] Step 2: Input the image I input in step 1 into the text extraction model, locate the features of the text information from the CTPN, and output the information array TextArr of each text box t from the output layer of the CRNN neural network. t . The array contains {t x , t y , t w , t h , t a , Text, Confidence}, where: t x , t y is the coordinate of the center point of the text box, t w , t h is the width and height of the text box, t a is the inclination of the text box, Text is the text content of the text box, Confidence is the confidence of the text box, and the default confidence value higher than 0.95 is credible.

[0050] Step 3: Remove the text box part in image I, fill it with the background color block of the text box, and then perform a dilation process with Kernel = (2, 2).

[0051] Step 4: Input the preprocessed image I in step 3 into the semantic segmentation network and output a semantic map I of size H×W s , I s Each pixel value y in i Represents the category to which pixel i belongs, which is divided into five categories: background class, connecting line class, rectangular node class, ellipse node class and diamond node class.

[0052] Step 5: Based on the input image I, the text information array TextArr obtained in step 2 t , the semantic graph I obtained in step 4s , and conduct data analysis of connected areas.

[0053] Step 6: Output the node information Node and connection information Link of the network diagram.

[0054] Step 7: Users can redesign or modify the original data extracted from network visualization to obtain a visualization design with a better layout or color scheme.

[0055] The present invention uses a text detection module to extract text data from a chart, improves the performance of semantic segmentation, obtains the context and position of the text through an optical character recognition system, removes the text area in the image, and fills it with a color block to obtain a network graph image without text. The present invention constructs a semantic segmentation neural network with an attention mechanism, which can accurately locate the pixels where nodes and edges are located and classify various types of nodes. By adding an attention module and improving the objective function, the network is robust to continuous curves and broken lines. The data analysis module analyzes the relevant components of the data obtained in the first two modules and matches text and graphics to finally obtain complete data.

[0056] The above is only a further explanation of the present invention and is not intended to limit this patent. Equivalent implementations that do not deviate from the spirit and scope of the present invention should be included in the scope of the claims of this patent.

Claims

1. A network graph data extraction method based on attention learning, characterized in that The extraction of network graph data specifically includes the following steps: Step a: Input a raster image I of a network graph of size H×W; Step b: construct the semantic dataset of the network graph; Step c: Train a semantic segmentation neural network based on u-Net and with an attention module on the semantic data set of the network graph, wherein the neural network includes a feature extraction part, an upsampling part, an attention module AttentionGate, and an output layer of a softmax model; Step d: training a neural network model for text extraction, wherein the neural network includes a text positioning part CTPN and a text recognition part CRNN; Step e: Apply the neural network model for text extraction in step d to the raster image I input in step a, locate the features of the text information from the CTPN, and then output the information array TextArr of each text box t from the output layer of the CRNN neural network t , the information array TextArr t Contains {t x , t y , t w , t h , t a , Text, Confidence}, where: t x , t y is the coordinate of the center point of the text box; t w , t h is the width and height of the text box; t a is the inclination of the text box; Text is the text content of the text box; Confidence is the confidence of the text box. By default, values ​​higher than 0.95 are credible. Step f: remove the text box part in the raster image I, fill it with the background color block of the text box, and perform a dilation process of Kernel = (2, 2); Step g: Input the image I preprocessed in step f into the semantic segmentation network in step c, and output a semantic map I of size H×W s , the semantic graph I s Each pixel value y in i represents the category to which pixel i belongs, and is represented by the background class, connecting line class, rectangular node class, ellipse node class and diamond node class as shown in the following formula d; Step h: Based on the input raster image I and text information array TextArr t and semantic graph I s Follow the steps below to perform data analysis and calculation of connected areas: h1: Change y i = 0 is assigned the color average of the pixels to C back ; h2: Extract y from raster image I i = 1, 2, 3 are connected components as node classes, and O p , H p , W p , C p , T p They are the coordinate value of the central pixel of the connected component, the color RGB value of the central pixel, and the pixel height, pixel width, and y value of the central pixel. i value; h3: Extract y from raster image I i = 1, 2, 3 are connected line types, and the y values ​​of the edge pixels of these connected components are i Value, calculate the connection confidence between the connected component and each node h4: TextArr array of text information t The text data with confidence higher than 0.95 is assigned to the nearest connected component, whose font size F is 0.75t h ; C back Represents the RGB value of the background color of the network diagram; p , H p , W p , C p , T p Indicates the center coordinate value, height value, width value, color RGB value and node shape type of the pth node; F indicates the font size; Indicates the confidence that the node p and the connecting line q are connected. The initial value defaults to 0, and the confidence increases by 1 for each overlapping pixel. Step i: Output the node information Node and connection information Link of the network diagram to obtain the original data extracted by network visualization. The user can redesign or modify the original data extracted by network visualization to obtain a visualization design diagram with a better layout or color scheme.

2. According to the network graph data extraction method based on attention learning according to claim 1, it is characterized in that The feature extraction part of the neural network includes: four convolution layers for extracting network graph visualization images, each convolution kernel uses a size of 3×3, the first two convolution layers are subjected to maximum pooling after two convolutions, and the last two convolution layers are subjected to maximum pooling after three convolutions; the upsampling part of the neural network uses the AttentionGate module to obtain an attention matrix α of the same size as the original resolution on feature maps of different scales, and then multiplies and connects it with the feature map corresponding to the feature extraction part, and finally a convolution layer with a 1×1 convolution kernel is used to obtain the semantic category value prediction results of all pixels.

3. According to the network graph data extraction method based on attention learning according to claim 1, it is characterized in that The semantic segmentation neural network based on U-Net and with attention module is collaboratively trained using the joint loss function defined by the following formula a: in: The function defined by the following formula b is used: Use the following c-style defined function: Where: ω is the weight of each pixel class, the default weight of the background class is 0.8, the default weight of the connecting line class is 1.25, and the default weight of other node classes is 1; y i is the classification result of pixel i obtained by the segmentation network, y′ i is the true label of pixel i category; During training, the loss function takes into account the balance between too many pixels in the background class and too few pixels in the connecting line class.

4. According to the network graph data extraction method based on attention learning according to claim 1, it is characterized in that The node shape type is rectangle, ellipse or rhombus.

5. According to the network graph data extraction method based on attention learning according to claim 1, it is characterized in that The grating image I is a pixel value matrix of H×W, wherein H is the number of pixels in each column of the image I in the longitudinal direction, and W is the number of pixels in each row of the image I in the transverse direction.

6. According to the network graph data extraction method based on attention learning according to claim 2, it is characterized in that The network diagram visualization images are different types of diversified network diagrams selected from open source visualization libraries, specifically including: D3, ECharts, Matplotlib and Scipy open source visualization libraries.

Images