A graphic recognition method and system for rich format documents

Abstract

The present invention discloses a method and system for graphic recognition of rich-format documents, belonging to the field of graphic recognition technology. The method comprises: identifying the area and category of graphics in a document page; obtaining objects in the graphic area based on the graphic category, wherein the objects include shape objects, text blocks, and lines; establishing an association between the text block and the shape object or line based on the location of the text block; finding adjacent objects of the line based on the endpoints of the line; integrating adjacent lines based on the line category to obtain an integrated line; establishing an association between the integrated line, adjacent shape objects, and the text block, and obtaining structured information. Different graphic categories have different object forms and connection forms. Graphic recognition based on graphic categories can effectively improve recognition accuracy. According to the line category, the individual lines of the line are integrated to obtain an integrated line, which is conducive to identifying the connection relationship between objects in complex graphics.

Description

Technical Field

[0001] The present invention relates to the technical field of graphic recognition, and in particular to a graphic recognition method and system for rich-format documents. Background Art

[0002] With the development of digital and intelligent technologies, the demand for automatic processing of rich-format documents with both text and images used in various industries is becoming more and more urgent. For example, various financial-related announcements and analysis reports contain equity structure charts and organizational charts; patent documents contain system framework diagrams and flow charts; scientific and technological papers and reports contain a large number of framework diagrams and flow charts.

[0003] In order to achieve deep structuring of rich-format document information and build domain expertise graphs, identifying and understanding the various figures contained in documents is an important step. The figures contain a large amount of structured information that can be converted into knowledge graphs and stored in databases.

[0004] Currently, deep learning-based image recognition methods are used for image recognition, including traditional methods based on image processing and methods based on object detection. For example, the following papers use traditional image processing-based methods to recognize flowcharts: Flowchart recognition for non-textual information retrieval in patent search, Information Retrieval, Universitat Autònoma de Barcelona, Visual structural analysis of flowcharts in patent images, JOANNEUMRESEARCH Forschungsgesel Ischaft mbH, DIGITAL-Institute for Information and Communication Technologies, Austria, and Textual summarization of flowcharts in patent drawings for CLEF-IP 2012, INRIA-Humboldt Universitat zu Berlin-Institut fur Deutsche Sprache und Linguistik.The following papers use object detection-based methods for graphic recognition: Symbol detection in online handwritten graphics using Faster R-CNN (DAS-2018, University of Sao Paulo), Arrow R-CNN for handwritten diagram recognition (International Journal on Document Analysis and Recognition (IJDAR), 2021 / 02, SAP and University of Mannheim), Sketch2BPMN: Automatic Recognition of Hand-drawn BPMN Models (33rd International Conference on Advanced Information Systems Engineering (CAiSE 2021), SAP and University of Mannheim), and DiagramNet: Hand-drawn Diagram Recognition using Visual Arrow-relation Detection (ICDAR-2021, SAP and University of Mannheim). Mannheim).

[0005] However, the above detection methods still have some problems: they only rely on graphic visual features for processing, usually focusing on the processing of a specific type of graphics; and usually deal with relatively simple direct connection relationships between some simple graphic objects, and cannot handle complex connection relationships such as curves, oblique lines, broken lines, and dotted lines. Summary of the Invention

[0006] In view of the above technical problems existing in the prior art, the present invention provides a method and system for graphic recognition of rich-format documents, which is suitable for processing the connection relationship between objects in complex graphics.

[0007] The present invention discloses a method for recognizing graphics in rich-format documents, the method comprising: identifying the area and category of graphics in a document page; obtaining objects in the graphic area based on the graphic category, the objects including shape objects, text blocks and lines; establishing an association between the text block and the shape object or line according to the location of the text block; finding adjacent objects of the line according to the endpoints of the line, the adjacent objects including adjacent lines and adjacent shape objects; integrating adjacent lines according to the category of the line to obtain an integrated line; establishing an association between the integrated line, the adjacent shape object and the text block, and obtaining structured information.

[0008] Graphic categories can be identified through machine learning methods:

[0009] Extract visual features of document pages using CNN+FPN or SwinTransformer methods;

[0010] Get the candidate box of the object in the page in the visual features;

[0011] After the candidate boxes are pooled, the bounding boxes of various objects and preliminary rough classification (distinguishing between Diagram and non-Diagram) are obtained through a neural network;

[0012] Sending the feature map of the candidate box to the mask prediction module to obtain the mask of the candidate box;

[0013] Obtaining fine features corresponding to the border from a second feature map obtained by directly upsampling or fusing FPN multi-scale features and then upsampling;

[0014] For the fine features, extract text features within a corresponding range of the graphic;

[0015] Obtaining a visual vector and a text vector respectively according to the fine features and the text features;

[0016] After the visual vector and the text vector are concatenated, they are classified by a multi-layer perceptron to obtain more detailed categories of the graphics.

[0017] The above candidate boxes can be obtained through Region Proposal Network (RPN);

[0018] After the candidate box is pooled by RoIAlign, the corrected border and rough classification are obtained by MLP;

[0019] Upsample the feature map to restore it to the original size to obtain a finer second feature map;

[0020] Obtaining corresponding fine features from the second feature map according to the position and size of the border;

[0021] Perform text detection, obtain the location of the text box, and recognize the text in it;

[0022] Use simple word vectors or context-related methods to obtain word vectors for each word (token) in the text content;

[0023] Perform pooling operation on the word vectors in the text block to obtain the text block vector;

[0024] Performing a pooling operation on all text block vectors in the graphic to obtain a text vector of the graphic;

[0025] The fine features corresponding to the graphics are reduced in dimension by 1x1 convolution and then converted into visual vectors through a multi-layer perceptron.

[0026] The detailed categories of the graphics vary depending on the specific application, including, for example, equity structure charts, organizational charts, framework diagrams, flow charts, engineering drawings, circuit diagrams, and molecular structure diagrams, etc.

[0027] The types of the connection line or integrated connection line include: bifurcation line, broken line, curve and straight line;

[0028] The bifurcated line includes a bus portion and a branch line portion, and the method for integrating the bifurcated line includes:

[0029] Further identifying endpoints of the line according to the line mask, the endpoints including a start point and an end point;

[0030] According to the type of the connection, a bus part and a branch part are obtained;

[0031] Connecting the branch line portion to the bus portion based on the endpoints to obtain a bifurcated line;

[0032] The polyline integration method includes: obtaining each segment of the polyline according to the type of the connecting line or the border; connecting the segments according to the endpoints to obtain the polyline;

[0033] The method for identifying the curve or oblique line comprises obtaining the rotation angle of the curve or oblique line frame, and the starting point and the ending point thereof.

[0034] If the closest object to two or more text blocks is the same object (shape or line), these text blocks can be merged together as a text annotation for this object;

[0035] If so, the text blocks are merged and used as text annotations for the shape object or line.

[0036] The present invention also includes a method for generating training data:

[0037] According to the type or field of graphics, establish object library, connection relationship, constraint and text information standard library,

[0038] The object library includes shape objects in the corresponding field, the connection relationship includes the connection lines of the shape objects and their directions and shapes, the constraints include the regulations and constraints between shape objects in a specific field, and the text information standard library includes the corpus and text rules of the field;

[0039] Based on the object library, connection relationships, constraints and text information standard library, a random algorithm can be used to generate graphic data;

[0040] Wherein, the random algorithm includes:

[0041] Set the number of layers of graphics;

[0042] Set the number of objects per layer, and the type, color, shadow, size, or font of the objects;

[0043] Set the type and distribution ratio of different connection methods between adjacent layers, and mark the fonts;

[0044] Set the connection type and distribution ratio between layers that are not directly adjacent;

[0045] Based on the constraints, check and modify the generated graph.

[0046] The present invention also provides a system for implementing the above-mentioned graphic recognition method, comprising a graphic category recognition module, an object recognition module, an object integration module and an aggregation module.

[0047] The graphic category recognition module is used to recognize the area and category of the graphic in the document page;

[0048] Based on the graphic category, the object recognition module is used to recognize objects in the graphic area, the objects including: shape objects, text blocks and connecting lines;

[0049] The object integration module is used to: establish an association between a text block and a shape object or a line according to the location of the text block; find adjacent objects of the line according to the endpoints of the line, wherein the adjacent objects include adjacent lines and adjacent shape objects; and integrate adjacent lines according to the type of the line to obtain an integrated line;

[0050] The aggregation module is used to establish an association between the integrated line, the adjacent shape objects and the text block.

[0051] The system also includes a training data generation module and a structuring module,

[0052] The structuring module is used to establish structured information or knowledge graphs based on the association between shape objects, integrated lines and text blocks;

[0053] The training data generation module is used to establish an object library, connection relationship, constraint and text information standard library according to the type or field of the graphic, and use a random algorithm to generate graphic data based on the object library, connection relationship, constraint and text information standard library.

[0054] Compared with the prior art, the beneficial effects of the present invention are as follows: different graphic categories have different object forms and connection forms, and graphic recognition based on graphic categories can effectively improve the recognition accuracy; according to the category of the connection, the various borders of the connection are integrated to obtain integrated connections, which is conducive to identifying the connection relationship between objects in complex graphics, and establishing structured information based on the association between text blocks and objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a flow chart of the graphic recognition method of rich format documents of the present invention;

[0056] Figure 2 It is a machine learning network structure diagram for identifying graphic categories;

[0057] Figure 3 It is a graphical diagram with a curved line;

[0058] Figure 4 is a graphical diagram with a broken line;

[0059] Figure 5 It is a graphical diagram generated by an automatic algorithm;

[0060] Figure 6 It is a system logic block diagram of the present invention. DETAILED DESCRIPTION

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0062] The present invention is described in further detail below with reference to the accompanying drawings:

[0063] A graphic recognition method for rich format documents, such as Figure 1 As shown, the method includes:

[0064] Step 101: Identify the area and category of a diagram in a rich document page. Typical categories include, but are not limited to, equity structure diagrams, organizational charts, framework diagrams, flow charts, engineering drawings, circuit diagrams, and molecular structure diagrams.

[0065] Step 102: Based on the graphic category, objects in the graphic area are obtained, wherein the objects include shape objects, connecting lines, and text blocks. The objects can be identified by visual features.

[0066] Step 103: Establish an association between the text block and the shape object or line based on the location of the text block.

[0067] Step 104: Find the adjacent objects of the line according to the starting point and the ending point of the line, ie, the endpoints. The adjacent objects include adjacent lines and adjacent shape objects.

[0068] Step 105: According to the type of the connection, adjacent connection lines are integrated to obtain integrated connection lines. In addition to straight lines, the types of integrated connection lines also include curves, broken lines, and bifurcated lines.

[0069] Step 106: Establish associations among the integrated lines, adjacent shape objects, and text blocks to obtain structured information.

[0070] Different graphic categories have different object forms and connection forms. Graphic recognition based on graphic categories can effectively improve recognition accuracy. According to the category of the connection, the various parts of the connection are integrated to obtain integrated connections, which is conducive to identifying the connection relationship between objects in complex graphics and establishing structured information based on the association between text blocks and objects.

[0071] In step 101, Figure 2 As shown, the graphic category can be identified through machine learning methods:

[0072] Step 201: Identify the visual features (Feature Map) of the diagram in the input document page (Input Image). Methods for obtaining visual features are known in the art and will not be described in detail in this disclosure. For example, visual features can be obtained using CNN+FPN or SwinTransformer methods.

[0073] Step 202: Obtain a proposal for the object in the image using the region proposal net (RPN). The proposal is the range of the candidate object in the original input image.

[0074] Step 203: The candidate box is pooled using RolAlign to obtain a fixed-size feature map. This is then predicted using a neural network or prediction module to obtain the corresponding corrected bounding box and coarse classification (object class). For example, the prediction module (prediction head) based on an MLP neural network performs the prediction. The coarse classification may include whether it is a graphic object, thereby filtering out the bounds or objects within the graphic, but is not limited to this. The bounds of an object define the boundaries of the object, and the object includes not only the bounds but also the features within the bounds.

[0075] Step 204: After convolution, the candidate boxes from step 202 are fed into a mask prediction module (mask head) to obtain a mask for the candidate boxes. A mask represents image segmentation, including the classification of each pixel. The mask acquisition method is well-known in the art and will not be further described in this disclosure.

[0076] Step 205: The visual features are directly upsampled (up-sampling) or the FPN multi-scale features are fused and then upsampled to restore the original image size to obtain a second feature map. The refined features corresponding to the border are obtained from the second feature map (Refined Feature Map). For example, based on the position and size of the object border, the refined features are deducted from the second feature map.

[0077] Step 206: Perform text detection on the fine features to extract text features. First, perform text detection to obtain text blocks. For example, the DBNET method (see the paper "Real-time Scene Text Detection with Differentiable Binarization," Minghui Liao, Zhaoyi Wan, et al., AAAI 20 November 2019) is used. This method uses a fully convolutional network architecture and can process feature maps of any size without the need for pooling to a fixed size. After obtaining text blocks through text detection, text recognition can be used to further obtain the text content, i.e., text features. The text features include the text content and the location of the text block.

[0078] Step 207: Obtain a visual vector and a text vector respectively according to the fine feature map and the text features.

[0079] The word vectors for text vectors can be generated using the following methods: simple word vector representations, such as word2vec or Glove; contextual vector representations, such as BERT (Bidirectional Encoder Representation from Transformers) or RoBERTa. Pooling operations are performed on the characters or words (tokens) of each text block to obtain a representation for the entire text block, such as average pooling. Pooling operations are performed on all text blocks within a page to obtain a text vector representation for the entire page, such as average pooling.

[0080] The fine features are reduced in dimension by 1x1 convolution and then transformed into a visual vector representation through MLP (multi-layer perceptron).

[0081] Step 208: After the visual vector and the text vector are concatenated, they are classified using a multi-layer perceptron (MLP) to obtain a detailed classification of the graphic, such as determining whether it is a shareholding structure chart or an organizational chart.

[0082] In step 105 , the connection line includes not only straight lines but also bifurcated lines, broken lines, and curved lines. The bifurcated line includes a main line portion and a branch line portion.

[0083] The detection results of the lines in the diagram are divided into regular straight lines and curves, as well as special buses. A bus and multiple regular straight lines or curves can form a bifurcation line; multiple straight lines can form a polyline.

[0084] According to the mask, endpoints of the line are identified, where the endpoints include a start point and an end point.

[0085] For straight lines and curves, we can identify the endpoints of the line, including the start and end points. This problem can be transformed into a classification problem for each pixel within the line mask, identifying the pixel as the start and end point of the current line. This is similar to the key point recognition of human poses in Mask-RCNN, but here there are only two keypoint types: start and end points.

[0086] The method for integrating the bifurcation lines includes:

[0087] Step 301: Based on the type of the connection, obtain the main line portion and the branch line portion connected thereto, where the branch line portion is a branched straight line or curve. The various parts of the branch line can be classified based on the type of the endpoint or border.

[0088] Step 302: Connect the branch line portion to the bus portion based on the endpoints to obtain a bifurcated line.

[0089] Polyline integration method:

[0090] Step 311: Obtain the straight lines of each segment of the polyline according to the type of the connection line;

[0091] Step 312: Connect the straight lines of the polyline according to the endpoints to obtain a polyline.

[0092] Figure 4 An example of the first bifurcation line is shown, which includes bus 1 and three straight lines input into the bus, and one straight line output from the bus. Finally, the lines can be connected to obtain three pairs of shape objects. The connection relationship of the first bifurcation line is: GZ Company---→HX Technology; RK Company---→HX Technology; ZY Company---→HX Technology.

[0093] Figure 4 An example of a second bifurcation line is also shown, which includes bus 2, 1 straight line input into the bus, and 4 straight lines output from the bus. Finally, the lines can be connected to obtain 4 pairs of shape objects. The connection relationship of the second bifurcation line is: HX Technology----→MD Company; HX Technology----→YT Company; HX Technology----→ZK Company; HX Technology----→HT Company.

[0094] Figure 4 There is also an example of a broken line, which is divided into three straight lines, and finally connected together to determine the connection relationship of the entire broken line.

[0095] Method for identifying a curve or oblique line: obtain the rotation angle of the connecting line frame, as well as the starting point and ending point of the connecting line. Figure 3 The recognition of diagonal lines is shown. By rotating the bounding rectangles of diagonal and curved lines at a certain angle, we can avoid complete overlap of the bounding rectangles of multiple curves and diagonal lines, thus preventing missed recognition. Therefore, in the recognition of curves and diagonal lines, in addition to predicting the position and size of the bounding rectangles, we also need to predict the rotation angle of the bounding rectangles. Methods for detecting objects with rotation angles, such as the ICCV-2021 paper "Oriented R-CNN for Object Detection," have been proposed.

[0096] The bounding box of the shape object can be obtained in step 203. Its size, proportion, and outline are relatively common, so it can be easily identified using conventional object detection methods. Since the current detection object is a shape object, the loss function calculated during training only calculates the classification prediction error and the coordinate prediction error of the bounding box. A method similar to Mask-RCNN can be used for detection.

[0097] In step 106, each text block coordinate can be used to find the closest object (shape object or line) to it, and its mask or bounding box can be referenced. For example, a text block is located inside a shape object; a text block overlaps a line; or a text block is close to a line.

[0098] If two or more text blocks are closest to the same object (shape or line), you can merge these text blocks together to form a text annotation for that object. For example, if a annotation is represented by two text blocks on two lines, you can merge the two text blocks.

[0099] Based on the integration of lines and the connection relationship between lines and shape objects, the relationship between associated objects (corresponding entities) is obtained, for example: (object 1, relationship, object 2); (object 1, relationship, object 3); (object 3, relationship, object 4); etc.

[0100] That is, the structured information of the graph is identified and stored in a knowledge graph. For example, it can be converted into a series of RDF triples and stored in an RDF database to construct a knowledge graph.

[0101] It is also possible to further identify the attributes of the relationship (such as the proportion of equity in the equity structure diagram) based on the accompanying text annotations on the line connecting the two objects. Therefore, it is more convenient to store them in a graph database based on the property graph structure, such as Neo4j. A shape object corresponds to a node (node, containing the attributes of the object), and the line between the two shape objects corresponds to the relationship (relation, containing the corresponding attributes).

[0102] The present invention also provides a method for generating training data:

[0103] Step 401: According to the type or field of the graphic, establish an object library, connection relationship, constraint and text information standard library.

[0104] The object library includes shape objects in corresponding fields. Some graphics in these fields contain relatively simple shape objects, such as equity structure diagrams, organizational charts, and flow charts, which typically only have simple switch boxes such as rectangular boxes, diamond boxes, and circular boxes. Other graphics in other fields contain more complex shape objects and a wider range of object types, such as system structure diagrams, engineering drawings, circuit diagrams, and molecular structure diagrams.

[0105] The connection relationships include the lines connecting shape objects, their directions, and their shapes. Some are simple one-to-one connections between two shape objects, while others are one-to-many connections with forks in the middle. Some connections have directions (with arrows), while others have no direction. Shapes include vertical or horizontal lines, broken lines, forked lines, oblique lines, and curved lines.

[0106] The constraints include regulations and restrictions between shape objects in a specific field. In some fields, there are no special constraints between objects, and connections can be generated relatively freely, while in other fields, there are some constraints, such as constraints between molecules or electronic components.

[0107] The standard text information library includes the corpus and text rules for the domain. Some domains have no specific requirements for text annotation and can be ignored; while other domains require the establishment of a corresponding text annotation information library based on the specific characteristics of the domain. This includes collecting relevant corpora, such as a database of company names for equity structure charts; and collecting relevant rules and specifications, such as the numeric percentage of equity ratios or the naming conventions for electronic components.

[0108] Step 402: Generate graph data using a randomized algorithm based on the object library, connection relationships, constraints, and text information standard library. Given a graph's topology, topological sorting or breadth-first search can be used to organize and visualize the graph hierarchically. Many practical documents also display graphs in a hierarchical topology.

[0109] Wherein, the random algorithm includes:

[0110] Step 411: Setting the number of layers of the graphics;

[0111] Step 412: Set the distribution of the number of objects in each layer, and set the type, color, shadow, size, and annotation font of the objects.

[0112] Step 413: Set the types and distribution ratios of different connection modes between adjacent layers, and mark the fonts.

[0113] Step 414: Set some connection types and distribution ratios between layers that are not directly adjacent.

[0114] Step 415: Check and modify the generated graph according to the constraints. Check whether the generated graph meets the set constraints.

[0115] The generated graphics can be stored in various formats (such as SVG) to store detailed information of the graphics, including: the position coordinates of the shape object, the coordinates of the connecting lines, and the coordinates of the annotation text. Figure 5An example of a graph generated by a random algorithm is shown, but the present invention is not limited thereto.

[0116] Automatically generating graphics through random algorithms reduces a large amount of manual labeling workload, is beneficial to the training of classification methods or models such as graphic classification and object classification, and is beneficial to the training of prediction models.

[0117] The present invention also provides a system for implementing the above-mentioned pattern recognition method, such as Figure 6 As shown, it includes: graphic category recognition module 1, object recognition module 2, object integration module 3 and aggregation module 4,

[0118] The graphic category recognition module 1 is used to recognize the area and category of the graphic in the document page;

[0119] The object recognition module 2 is used to obtain objects in the graphic area based on the graphic category, and the objects include: shape objects, text blocks and connecting lines;

[0120] The object integration module 3 is used to: establish an association between the text block and the shape object or line according to the location of the text block; find the adjacent objects of the line according to the endpoints of the line, wherein the adjacent objects include adjacent lines and adjacent shape objects; and integrate the adjacent lines according to the type of the line to obtain an integrated line;

[0121] The aggregation module 4 is used to establish associations among the integrated lines, adjacent shape objects and text blocks.

[0122] The system of the present invention further comprises a training data generating module 5 and a structuring module 6,

[0123] The structuring module 6 is used to establish structured information or knowledge graphs based on the association between objects, integrated links and texts;

[0124] The training data generation module 5 is used to establish an object library, connection relationship, constraint and text information standard library according to the type or field of the graphic, and use a random algorithm to generate graphic data based on the object library, connection relationship, constraint and text information standard library.

[0125] To facilitate understanding of the present invention, relevant technical terms are explained as follows:

[0126] Feature Pyramid Network (FPN) is a method that uses a conventional CNN model to efficiently extract features of various scales in images.

[0127] The Swin Transformer is adaptable to computer vision and has shown promising results on certain tasks, notably image classification and joint vision-language modeling.

[0128] RoIAlign is a pooling method based on bilinear interpolation, which avoids the loss of information of the original feature map in the process, making the obtained small feature map more accurate and complete.

[0129] Word2Vec is a language model that learns semantic knowledge from large amounts of text in an unsupervised manner. It is widely used in natural language processing. Word2Vec is a tool used to generate word vectors.

[0130] BERT (Bidirectional Encoder Representation from Transformers) is a pre-trained language representation model.

[0131] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A method for pattern recognition of rich format documents, characterized in that: The method comprises: Identify regions and categories of graphics within document pages; Based on the graphic category, obtaining objects in a graphic area, the objects including: shape objects, text blocks and connecting lines; According to the location of the text block, establish the association between the text block and the shape object or line; Finding adjacent objects of the line according to the endpoints of the line, wherein the adjacent objects include adjacent lines and adjacent shape objects; According to the category of the connection, the adjacent connections are integrated to obtain integrated connections; Establishing associations among the integrated lines, adjacent shape objects, and text blocks, and obtaining structural information; Among them, the machine learning method is used to identify graphic categories, including: Identify visual features of document pages; Get the candidate box of the object in the page in the visual features; After the candidate frames are pooled, the bounding boxes and rough classification are obtained through a neural network; According to the rough classification, obtaining a border or object of the graphic; Upsampling the visual features of the graphic or upsampling after fusing multi-scale visual features to obtain a second feature map; Obtaining fine features corresponding to the border; For the fine features, extract text features within a corresponding range of the graphic; Obtaining a visual vector and a text vector respectively according to the fine features and the text features; After the visual vector and the text vector are concatenated, they are classified by a multi-layer perceptron to obtain the graphic category.

2. The pattern recognition method according to claim 1, characterized in that: Extract visual features of graphics using CNN+FPN or SwinTransformer methods; Obtain a candidate box from the visual features through RPN; After the candidate box is pooled by RoIAlign, it is corrected by MLP to obtain the border and rough classification; Obtaining fine features from the second feature map according to the position and size of the border; Performing text detection on the fine features or borders to obtain the position and text features of the text box; Use simple word vectors or context-related methods to obtain word vectors for text features; Perform pooling operation on word vectors to obtain text block vectors; Performing a pooling operation on all text block vectors in the graphic to obtain a text vector for the page or graphic; After the fine features are reduced in dimension by 1x1 convolution, they are converted into visual vectors through a multi-layer perceptron.

3. The pattern recognition method according to claim 1, wherein: The graphic categories include equity structure charts, organizational charts, framework diagrams, flow charts, engineering drawings, circuit diagrams and molecular structure diagrams.

4. The pattern recognition method according to claim 1, wherein: The feature map of the candidate box is sent to the mask prediction module to obtain the mask of the candidate box.

5. The pattern recognition method according to claim 4, characterized in that: The types of the connection line or integrated connection line include: straight line, bifurcated line, broken line and curve; The bifurcated line includes a bus portion and a branch line portion, and the method for integrating the bifurcated line includes: According to the mask, identifying endpoints of the line, the endpoints including a start point and an end point; According to the type of the connection, a bus part and a branch part are obtained; Connecting the branch line portion to the bus portion based on the endpoints to obtain a bifurcated line; The polyline integration method includes: obtaining each segment of the polyline according to the category of the frame; connecting the segments according to the endpoints to obtain the polyline; The method for identifying the curve or oblique line comprises obtaining a rotation angle, a starting point, and an ending point of the curve or oblique line frame.

6. The pattern recognition method according to claim 1, characterized in that: If the closest object to two or more text blocks is the same shape object or line, these text blocks are merged together and the whole is used as the text label of the switch object or line.

7. The pattern recognition method according to claim 1, characterized in that: It also includes methods for generating training data: According to the type or field of graphics, establish object library, connection relationship, constraint and text information standard library, The object library includes shape objects in the corresponding field, the connection relationship includes the connection lines of the shape objects and their directions and shapes, the constraints include the regulations and constraints between shape objects in a specific field, and the text information standard library includes the corpus and text rules of the field; Generating graphic data using a random algorithm according to the object library, connection relationships, constraints and text information standard library; Wherein, the random algorithm includes: Set the number of layers of graphics; Set the number of objects per layer, and the type, color, shadow, size, or font of the objects; Set the type and distribution ratio of different connection methods between adjacent layers, and mark the fonts; Set the connection type and distribution ratio between layers that are not directly adjacent; Based on the constraints, check and modify the generated graph.

8. A system for implementing the pattern recognition method according to any one of claims 1 to 7, characterized in that: It includes graphic category recognition module, object recognition module, object integration module and aggregation module. The graphic category recognition module is used to recognize the area and category of the graphic in the document page; The object recognition module is used to obtain objects in the graphic area based on the graphic category, and the objects include: shape objects, text blocks and connecting lines; The object integration module is used to: establish an association between a text block and a shape object or a line according to the location of the text block; find adjacent objects of the line according to the endpoints of the line, wherein the adjacent objects include adjacent lines and adjacent shape objects; and integrate adjacent lines according to the type of the line to obtain an integrated line; The aggregation module is used to establish an association between the integrated line, the adjacent shape objects and the text block.

9. The system according to claim 8, characterized in that It also includes training data generation module and structured module, The structuring module is used to establish structured information or knowledge graphs based on the association between shape objects, integrated lines and text blocks; The training data generation module is used to establish an object library, connection relationship, constraint and text information standard library according to the type or field of the graphic, and use a random algorithm to generate graphic data based on the object library, connection relationship, constraint and text information standard library.

Images