Document information extraction method and device

Abstract

The invention provides an innovative solution, on the basis of an original second-order method of OCR (Optical Character Recognition) textbox detection and character recognition extraction, the high-order processing capability of relation logic recognition extraction of information in a picture document sub-layout is expanded, visual features and semantic features of the document can be fully utilized, and the recognition efficiency is improved. A model with a good effect is trained under the condition of few samples, and the method has strong generalization ability, so that the requirement of complex layout document information extraction in an enterprise is quickly met.

Description

Technical Field

[0001] The present application belongs to the technical field of text optical character image recognition, and in particular, the present application provides a document information extraction method and device. Background Art

[0002] Document processing is an indispensable part of daily business operations and personal work. For example, there are a large number of business scenarios in enterprises that require information to be extracted from documents and processed, such as extracting required fields from different types of documents such as invoices, expense reports, and delivery notes, and performing operations such as input, verification, and comparison. Therefore, document information extraction is one of the core capabilities of the intelligent document processing platform.

[0003] However, traditional documents may be in the form of PDF, PNG images, etc., and the processing method often relies on manual transcription of the text content on the graphics, which is inefficient and prone to errors. Especially when processing a large number of documents, it is necessary to extract and summarize the key information in the documents. The defects of manual recognition and transcription of text operations are becoming more and more obvious, making it difficult to meet the needs of efficient and accurate document processing.

[0004] In recent years, the rise of intelligent document processing (IDP) technology has provided a new idea for solving this problem. IDP technology combines multiple technologies to achieve automatic extraction, classification, processing and analysis of document content in image format. IDP technology is mainly based on image preprocessing based on OCR optical character recognition, and parses the text sections in the image based on algorithms (including layout detection, directory detection, table detection, text box detection, etc.) and extracts information, and finally recognizes information based on semantic analysis and reorganizes output.

[0005] IDP (Intelligence Document Processing, IDP intelligent document processing) technology is a package of technologies for automatic processing and analysis of document content. In addition to natural language processing technology, it also includes related underlying technologies such as computer vision and document parsing. Therefore, compared with NLP technology, IDP technology is more complex. IDP is a specific application field of multimodal data processing in intelligent document processing. Among IDP technologies, Optical Character Recognition (OCR) technology, Document Parsing (DP) technology and Information Extraction (IE) technology are the most core and widely used.

[0006] Among them, OCR (Optical Character Recognition) technology mainly solves the problem of presenting text in the form of images. A lot of text information is presented in the form of graphics in documents, and even many documents are saved in the form of images. Therefore, OCR technology is needed to extract all the text and its position in the document for analysis. DP (DocumentParsing) technology implements different types of file protocol parsing, unified representation of document content, layout analysis technology, table parsing technology, etc. The purpose is to use the same set of protocols to represent the document structure and content of different types of documents and various element information therein, including images and semantic information. IE (Information Extraction) technology refers to the use of different algorithms to extract information from different document elements based on document structure and content information, merge the extracted information results, and output them according to business needs.

[0007] However, document images usually contain a large amount of non-text content such as pictures and tables. For example, in addition to text information, the layout of an ordinary paper or manuscript often contains headers, footers, tables, QR codes, formulas, pictures and other elements. However, OCR's built-in "second-order text recognition and text box recognition" are not flexible in extracting information. In OCR second-order processing, it can only return in a fixed order from left to right and from top to bottom, and cannot achieve flexible text block recognition and correct annotation sorting according to the document structure in specific business scenarios. Figure 4 In the tax payment certificate example, there are many line breaks in the tax details. The order of the text blocks output by OCR is as follows: Figure 4 The numbers marked on the top (①, ②, ③, ... ⑦, ⑧, ...) are obviously not in the reading order, and will also disrupt the semantic and layout information. If no adjustment is made, the model will not be able to accurately obtain the correct semantic and structural information in the future.

[0008] Therefore, a certain amount of labeled data is usually required to assist in model training to ensure that the extracted document information can accurately represent the original semantic and structural information of the document. However, this also poses two challenges to training machine learning models for document information extraction tasks.

[0009] First, there are many types of documents, and even the same type of documents may have multiple formats. For example, for regular tax payment certificates, the formats are usually very different, especially in the detailed table part, where the number of fields and the order of fields are different. This means that if the traditional IDP method is used, multiple machine learning models need to be trained to meet the needs of extracting information from different formats.

[0010] Second, many document types are enterprise-specific, which means that it is difficult to collect data and train models in advance. Instead, it is necessary to train a usable model based on a small sample of enterprise-specific document data. Otherwise, the cost of training the model will be too high. Summary of the invention

[0011] In view of the above-mentioned problems in the prior art, the purpose of this application is to provide a document information extraction method and device, which can ensure that the extracted information conforms to the layout of the graphic document, improve the semantic accuracy of the extracted information, and can train a model with good results using fewer samples and have strong generalization ability.

[0012] In order to solve the above technical problems, the specific technical solutions of this application are as follows:

[0013] On the one hand, the present application provides a document information extraction method, characterized in that the method includes: performing text block detection on a document image source to be processed to obtain text block features of the document image source to be processed, and the text block features are used to characterize the text information and adjacent connection relationship information of the text blocks in the document image source to be processed; using the text block features as input, performing layout recognition on the document image source to be processed to obtain document typesetting features of the document image source to be processed, and the document typesetting features are at least used to characterize the layout area to which the text block belongs and the layout type of the layout area; based on the text block features and document typesetting features of the document image source to be processed, performing information extraction on the document image source to be processed to obtain document structured information of the document image source to be processed.

[0014] In some embodiments, the text block features are used as input to perform layout recognition on the document image source to be processed, including: using the text block features as input to perform layout area segmentation and classification on the document image source to be processed, and obtaining document layout features for characterizing the layout area to which the text block belongs and the layout type of the layout area; for layout areas with a tabular layout type, using an image semantic segmentation algorithm to convert the wireframe graphic information of the tabular layout area into document layout features.

[0015] In some embodiments, the text block detection is performed on the document image source to be processed to obtain the text block features of the document image source to be processed, including: extracting text features of the document image source to be processed to obtain text features; performing slice detection on the document image source to be processed to generate slice features of the document image source to be processed; performing same-layer neighbor detection and cross-layer neighbor detection on the slices of the document image source to be processed to obtain neighbor relationship features; and splicing the slice features with the neighbor relationship features to obtain text box splicing detection features of the document image source to be processed.

[0016] In some implementations, the layout area segmentation and classification of the document image source to be processed includes: using point-wise convolution to segment and classify the document image source to be processed; wherein the point-wise convolution configures convolution parameters based on the text features, slice features and text box stitching detection features of the document image source to be processed, wherein the convolution parameters include the number of feature maps, and the filter size, weight parameters and sliding step size of the convolution kernel.

[0017] In some embodiments, the method further includes: extracting multiple document image sources with differences in document typesetting as sample document image sources, performing text quick annotation and document typesetting annotation on the sample document image sources to obtain a sample set, and performing text block detection and layout recognition on the sample document image sources based on the sample set.

[0018] In some implementations, the document image source to be processed is segmented and classified in layout areas, including: using point-wise convolution to segment and classify the sample document image source in layout areas; wherein the point-wise convolution configures convolution parameters based on the text features, slice features, and text box splicing detection features of the sample document image source, as well as the annotated slice features and annotated text box splicing detection features, wherein the convolution parameters include the number of feature maps, and the filter size, weight parameters, and sliding step size of the convolution kernel.

[0019] In some embodiments, the image semantic segmentation algorithm includes table area segmentation and recognition and table frame line detection; wherein, the table area segmentation and recognition is used to divide cells based on the table frame lines, and use the cell information as the document layout feature; the table frame line detection refers to regressing the graphic boundaries based on Faster-RCNN, detecting the horizontal and vertical line frames of the table, and using the horizontal and vertical line frame information of the table as the document layout feature.

[0020] In some embodiments, the information extraction of the document image source to be processed based on the text block features and document layout features of the document image source to be processed includes: encoding and serializing the text block features and the document layout features based on a transformer algorithm to obtain a coding sequence of the document image source to be processed; wherein the transformer algorithm includes an encoder for text block features and an encoder for document layout features; and information extraction of the document image source to be processed based on the coding sequence of the document image source to be processed.

[0021] In some embodiments, the information extraction of the document image source to be processed based on the coding sequence of the document image source to be processed includes: based on the coding sequence of the document image source to be processed, using a semantic feature encoder of a predefined information extraction structure to extract information from the document image source to be processed to obtain document structured information that satisfies the information extraction structure.

[0022] On the other hand, the present application also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method described in any of the above embodiments is implemented.

[0023] This application proposes an innovative solution. Based on the original second-order method of OCR text box detection and text recognition and extraction, it expands the high-level processing capabilities of relational logic recognition and extraction of information within the layout of image documents. It can make full use of the visual and semantic features of the document, train a very effective model with fewer samples, and has strong generalization ability, thereby quickly meeting the needs of enterprises for information extraction from complex layout documents.

[0024] In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the following specifically cites preferred embodiments and describes them in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 A schematic diagram of the model architecture of the text serialization model of an embodiment of the present application is shown;

[0027] Figure 2 A schematic diagram of the processing flow of the text serialization model of an embodiment of the present application is shown;

[0028] Figure 3 A schematic diagram of document information extraction based on a text serialization model in an embodiment of the present application is shown;

[0029] Figure 4 A schematic diagram of a document image source in an embodiment of the present application is shown;

[0030] Figure 5 A schematic diagram of text information extracted in an embodiment of the present application is shown;

[0031] Figure 6A schematic diagram of the internal structure of a computer device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0034] The present application pre-builds a text serialization model, uses the text serialization model to perform regional segmentation processing on the overall image document layout, and automatically parses the text position and semantic logic association of each region, thereby extracting the original regional structure information on the image document together with the text symbol information, restoring the regional layout on the original image in the output document, and placing the extracted text information in the correct position in the newly generated document.

[0035] The text serialization model has little to do with the specific file type, but is mainly related to the structural position arrangement of the specific document on the image layout. Therefore, a general text block serialization model can be trained through the text block annotation data on the image layout, and the model can be directly used in other different types of document processing projects, reducing the initial cost of starting each IDP project.

[0036] Therefore, based on the original second-order method of OCR text box detection and text recognition and extraction, this application expands the high-level processing capabilities of relational logic recognition and extraction of information within the layout of image documents. It can make full use of the visual and semantic features of the document, train a very effective model with fewer samples, and has strong generalization capabilities, thereby quickly meeting the needs of enterprises for information extraction from complex layout documents.

[0037] Accordingly, the embodiment of the present application first provides a method for building a text serialization model. This specification provides method operation steps as described in the embodiment or flowchart, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the order of many steps and does not represent the only execution order. When the system or device product is executed in practice, it can be executed in the order of the method shown in the embodiment or the accompanying drawings or in parallel. The method may include:

[0038] Before the model is built, the samples can be annotated in the following manner so that the samples can correctly reflect the text position and semantic logic of each area of ​​the document source, guide the adjustment of model parameters, and accurately and quickly complete the model training. Accordingly, in some embodiments, multiple document sources with different document layouts can be extracted as sample document sources, and the sample document sources can be annotated with text and document layout to obtain a sample set, so as to perform text block detection and layout recognition on the sample document source based on the sample set.

[0039] Step 1. Sample annotation

[0040] Step 1-1. Text block Value annotation

[0041] For the document layout of the "Tax Payment Certificate" above, you can circle the text segments that have line breaks but belong to the same information block with the "Text Block Value Marking".

[0042] Step 1-2. Layout classification field labeling

[0043] Each text block value is labeled with the corresponding field domain label. These circled text segments will be used as sample data for text block segmentation and as sampling for the seglink algorithm and FCN neural network model.

[0044] The sample data based on which the model construction mentioned below is based can refer to this embodiment. Of course, other types of annotations can also be used, which is not limited here.

[0045] refer to Figures 1 to 5 , the model construction mainly includes the contents described in steps 2 to 4 below.

[0046] Step 2. Text block detection based on regression method

[0047] Step 2-1. Text feature extraction

[0048] Conventional algorithms can be used to extract text features. Text features can be, for example, OCR character optical detection features, such as image processing of document image sources, text positioning and segmentation, character feature extraction, character recognition and other algorithms to obtain OCR character optical detection features.

[0049] The following sub-steps 2-2 to 2-4 can use the Seglink algorithm of "slice connection integration" based on regression thinking to extract feature information of text blocks.

[0050] Seg detection is multi-layer dense sampling. Its feature layer extraction uses VGG16 (an international standard open source deep convolutional neural network framework) as the main backbone of the network. There is no fc fully connected layer, and 6 groups of convolutional layers conv1-conv6 are provided for feature map extraction. The feature map sizes of these 6 layers are different, and the size of each layer is only half of the previous layer (1 / 2, 1 / 4, 1 / 8, 1 / 16, 1 / 32, 1 / 64). Segment slices and link adjacency relationships are obtained from these 6 layers of different sizes, and the detection of text lines of different sizes can be realized (large feature map sizes are good at detecting small objects, and small feature map sizes are good at detecting large objects).

[0051] Step 2-2. Segment detection

[0052] For the scanned image document, the text blocks on the image are operated to generate slices one by one and form image annotation boxes. These slices are part of the text line (or word), which may be a character, a word, or several characters.

[0053] In segment detection, based on the "sample frame annotation" pre-training method, the results are regressed. The number of channels output by each feature map after convolution is 7, of which 2 represent the positive and negative sample confidence values ​​of whether the segment is text (0,1), and the remaining 5 are the five offsets (X, Y, W, H, O) of the default box of the frame annotation sample relative to the corresponding position of the slice.

[0054] X and Y are the offsets from the midpoint of the default box and the midpoint of the ground truth of the training dataset. W and H are the width and height of the image of the sample with the box annotation. O represents the image deflection angle of the sample with the box annotation.

[0055] Step 2-3. Link adjacency detection

[0056] Based on the detected segments, the connection relationship (link) between segments is further determined. Link is used to indicate which segments belong to the same text line, thereby helping to connect scattered text fragments to form a complete text line.

[0057] Neighbor detection is to identify the continuous neighbor relationship between adjacent slices. There are two types of links: one is same-layer neighbor detection, and the other is cross-layer neighbor detection.

[0058] For the same-layer neighbor detection, it is defined between adjacent slices in the same layer. Because of the slice distribution shape of the tic-tac-toe grid, a slice has 8 adjacent slices in the same layer, that is, there are 8 neighbor relationships. The score of the logistic regression (using the softmax function) of 2 dimensions at each neighbor relationship mapping, a total of 16 dimensions.

[0059] Cross-layer neighbor connections are used to solve the problem of redundant slice detection of the same text area on feature maps of different layers. In order to improve the recall rate, the range of each slice prediction overlaps. However, the result is that for the same word, feature maps of different layers can perform slice detection, which will generate a lot of redundant repeated detections.

[0060] In the cross-layer neighbor connection processing, except for the first layer, each layer will output cross-layer neighbor connections. The scale of the feature maps of two adjacent layers differs by 2 times. In addition, according to the 2-dimensional regression operation, the adjacent position relationship between the same layer and the previous layer is 1:4, that is, the current layer will be adjacent to 4 slices of the previous layer. For these 4 adjacent connections, each of the current layer also outputs a 2-dimensional regression score.

[0061] Overall, the output of a conv convolution filter includes: 2-dimensional word sample confidence + 5-dimensional sample offset + 16-dimensional same-layer neighbor regression + 8-dimensional cross-layer link regression, a total of 31-dimensional neighbor relationship features.

[0062] Step 2-4. Seg slicing and Link adjacent splicing

[0063] The goal of combining Seg and Link splicing is to fit the features of slice detection and link detection and the results of regression operations, so as to output a merged text box feature vector, obtain the location information of the text area (such as bounding box coordinates, rotation angle, etc.), and the confidence of the text area.

[0064] Take out the slice segment of the same row.

[0065] Perform the least squares linear regression on the center points of these segments to obtain a straight line.

[0066] The center point of each segment is projected vertically onto this line.

[0067] Take the two points with the farthest distance from all the projection points and record them as (xp, yp) and (xq, yq).

[0068] Then the final merged text box has (1) a center point at ((xp+xq) / 2, (yp+yq) / 2), (2) a width equal to the distance between the two farthest points (xp, yp) and (xq, yq) plus half the width of the segment (Wp / 2+Wq / 2), and (3) a height equal to the average height of all segments.

[0069] Through the above scheme, at least the feature information of the text block can be obtained: OCR character optical detection features, text box seg slice detection features and text box seg+link splicing detection features.

[0070] Of course, other regression methods may also be used to extract feature information of text blocks, which is not limited here.

[0071] Step 3. Layout serialization based on CNN convolutional network

[0072] Step 3-1. Layout feature extraction based on CNN

[0073] The text block features may be used as input to segment and classify the layout areas of the document image source to be processed, thereby obtaining document layout features for characterizing the layout area to which the text block belongs and the layout type of the layout area.

[0074] For example, after completing the preprocessing of text block feature recognition, the original generalized feature map information of the document image source can be cyclically enhanced in the directional recognition of layout classification features based on the CNN convolutional neural algorithm, and finally a feature sequence that meets the requirements of layout recognition can be formed through vector extraction and combination.

[0075] The point-wise convolution method of deep learning can be used. Based on the basic characteristic of the traditional convolution method that "the number of output features is equal to the number of input channels", the point-wise convolution operation will perform a weighted combination of the feature map of the previous step in the depth direction to generate a new text block feature map.

[0076] The point-wise convolution method makes up for the defect of traditional convolution that "each channel performs convolution operation independently and does not effectively utilize the feature information of different channels at the same spatial position". The characteristics of the point-wise convolution method are particularly suitable for extracting feature maps of subtle text graphics, thereby achieving higher accuracy in recognizing dense text areas.

[0077] The specific implementation process is described as follows:

[0078] (1) After the conventional processing of "text serialization" in the previous step 2, three types of source feature maps are output: conventional text feature maps, slice feature maps, and text box splicing feature maps. Therefore, the subsequent point-wise convolution kernel filter size is set to 1*1*3. Channel weighting of the depth of text features, slices, and text box splicing features is achieved.

[0079] (2) The number of point-wise convolution feature maps is designed based on the two input image source objects that need to be "serialized". The two input image sources are: one is the sample document image source of the "framed annotation sample" in step 1; the other is the document image source to be processed using the model to perform document information extraction after the model is built. For the two image source objects, the focus of the required extracted features and the corresponding weight parameters are different.

[0080] According to the current best experience after adjusting the convolutional layer parameters, the number of feature emphasis (i.e., the number of feature maps) of the sample document source output is designed to be 5, including:

[0081] Feature focus: 1. The “base image” OCR character optical detection features of the source object.

[0082] Feature focus 2: Seg slice detection feature of the "base map" text box of the image source object.

[0083] Feature focus 3. seg+link stitching detection feature of the "base map" text box of the image source object.

[0084] Feature focus 4, "Manually marked frame on the image" text box seg slice detection feature of the image source object.

[0085] Feature focus 5: seg+link splicing detection feature of “manually marked frame on the image” of the image source object.

[0086] The output feature emphasis number (i.e. the number of feature maps) of the document source to be processed is 3, including:

[0087] Feature focus: 1. The “base image” OCR character optical detection features of the source object.

[0088] Feature focus 2: Seg slice detection feature of the "base map" text box of the image source object.

[0089] Feature focus 3. seg+link stitching detection feature of the "base map" text box of the image source object.

[0090] (3) In the design of the weight parameters and sliding step size of the point-wise convolution kernel, according to the focus of feature extraction of the above two different image source objects, the best effect is set empirically after adjusting the parameters of the convolution layer. In this 1*1*3 size, the initialization weight parameters can be:

[0091] Feature focus 1. The initialization weight parameters of the source object OCR character optical detection feature are designed to be {[4], [1], [1]}.

[0092] Feature focus 2. The initialization weight parameters of the source object text box seg slice detection feature are designed to be {[2], [4], [3]}.

[0093] Feature focus 3. The initialization weight parameters of the source object text box seg+link splicing detection feature are designed to be {[2], [3], [4]}.

[0094] The sliding step size of the convolution kernel filter is designed, and the value is set to 3 based on the best experience after adjusting the parameters of the convolution layer. The number of convolution layers also uses the 6-layer convolution of the VGG16 backbone network, and all output features are processed by pooling convergence without FC fully connected layer processing.

[0095] Step 3-2. FCN-based table wireframe graphic feature extraction

[0096] After extracting the layout features, for the table wireframe graphics (i.e. the table-type layout area), the table and list wireframe graphic information in the image source can be converted into "semantic vector information of the layout pattern" based on the FCN image semantic segmentation algorithm, that is, the wireframe graphic information of the table-type layout area is converted into document layout features, so as to perform text division and semantic coherence based on the table wireframe graphics to avoid semantic confusion.

[0097] In this application, based on the FCN algorithm, feature recognition of table graphics is performed in two directions:

[0098] (1) Table area segmentation and recognition: Similar to the above-mentioned layout area segmentation and classification method, each cell divided by the table frame is segmented and recognized as the corresponding local layout area in the table, forming a document layout feature with clear blocks.

[0099] (2) For table frame line detection, it is mainly based on the high-precision target detection characteristics of Faster-RCNN. It maps the feature vector obtained by convolution processing based on the candidate area of ​​the RPN layer. After the classification processing of the traditional CNN, it forms a regression processing of the graphic boundary, and finally achieves the target detection of the table horizontal and vertical wireframe graphic object, forming an accurate table wireframe target feature vector, that is, using the table horizontal and vertical wireframe information as the document layout feature.

[0100] (3) These feature vectors will be used as input sequences and then enter Tranformer for fitting with other feature vector sequences to generate a table parameter sequence of cell text + table wireframe.

[0101] Step 3-3. Transformer-based information extraction model training

[0102] After serializing the text as above, the feature vector of the document source (including sample document source and document source to be processed) will be obtained. This step is based on the Transformer modeling framework to map the image layout feature vector to semantic classification, thereby forming a layout recognition and information extraction model.

[0103] Accordingly, in some implementations, the text block features and document layout features can be encoded and serialized based on the transformer algorithm to obtain a coding sequence of the document image source to be processed; wherein the transformer algorithm includes an encoder for text block features and an encoder for document layout features, and then based on the coding sequence of the document image source to be processed, information extraction is performed on the document image source to be processed.

[0104] The implementation process is outlined as follows:

[0105] (1) For the sample document image source of the "framed annotation sample" and the document image source to be processed for information extraction, multiple corresponding feature maps are obtained through the above-mentioned deep convolution and point convolution. Since the output feature map has not been processed by pooling and full connection, the dimensionality is still relatively large and cannot be directly used to train layout classification and information recognition processing models. It also needs to be further serialized into vectors that can be used for model training and model inference.

[0106] This application uses a "compression fusion" mechanism to compress each feature map dimension into a two-dimensional sequence, and then further fuses the feature maps of multiple compressed dimensions corresponding to a source object into a final two-dimensional sequence. The core method used in this patent is that the Tranformer encoder serializes the multi-dimensional feature map.

[0107] The encoder of Tranformer can provide a function similar to the pooling and fully connected layers in convolutional networks to achieve feature dimension convergence. The patent sets up two corresponding encoder mechanisms for text information and layout information - to achieve (i) text block feature vector encoding and (ii) layout element feature vector encoding.

[0108] (2) In addition to converting the source object feature map into a serialized vector by setting up two corresponding encoders, the annotation data samples generated by the "text block value annotation" and "layout classification field annotation" of the previous text serialization sample are serialized by Transformer. After serialization, these annotation samples can describe the format structure (Json structure) followed by subsequent information extraction. Therefore, the patent sets up a third encoder mechanism here - semantic feature encoding of the information extraction structure.

[0109] (3) After having the above-mentioned convolutional feature map extraction and encoding serialization processing module, the feature vector dataset with manually labeled text block Value and layout classification field is used as a sample and input into the transformer framework for model training to realize the modeling link of layout recognition and information extraction.

[0110] The purpose of layout serialization is to use it for further Transformer processing, thereby forming an integrated processing model that can semantically classify and extract information from image layout information.

[0111] Step 4. Structured information extraction

[0112] Based on the coding sequence of the document image source to be processed, a semantic feature encoder of a predefined information extraction structure is used to extract information from the document image source to obtain document structured information that meets the information extraction structure.

[0113] The information extraction model formed by the above steps, after inputting a document source of the same type or similar layout (such as a tax payment certificate, pure table and other graphic documents or PDF), will be combined with the model to operate on the two types of vectors of the document source, text and layout, and infer the structured text block sequence and layout information sequence in the sample document. Based on the result sequence output by the model, this step provides the assembly of the extraction results according to the specified pattern, and finally generates the structured information output in Json format, and provides a standardized API interface for the application layer system to call the Json file to realize the structured presentation of document information extraction.

[0114] This application is different from the traditional second-order processing method based on OCR text box + text recognition. It makes up for the deficiency that it can only extract text serialization and cannot accurately restore the layout of each part of the graphic. It expands the detection and extraction of layout structures of complex document forms such as text segments, tables, drawings, and irregular text segments. Finally, on the Word document, the original graphic document layout is accurately restored, and the corresponding text is formatted according to the corresponding paragraph attributes. It can also train a model with good results with fewer samples and has strong generalization ability, so as to quickly meet the needs of complex layout document information extraction in enterprises.

[0115] Accordingly, in some embodiments, the method for processing the document source to be processed based on the above-constructed text serialization model to obtain text information is as follows:

[0116] S22: Perform text block detection on the document source to be processed to obtain text block features of the document source to be processed, wherein the text block features are used to characterize text information and adjacent connection relationship information of the text blocks in the document source to be processed.

[0117] The text feature extraction of the document source to be processed can be performed to obtain text features; the slice detection of the document source to be processed can be performed to generate the slice features of the document source to be processed; the slices of the document source to be processed can be subjected to same-layer adjacency detection and cross-layer adjacency detection to obtain adjacency relationship features; the slice features and adjacency relationship features can be spliced ​​to obtain the text box splicing detection features of the document source to be processed. The specific processing method can refer to the process of building the above model. Through this preprocessing method, the text features, text position features and position relationship features of the text that characterizes the document source to be processed can be preliminarily obtained to assist in the extraction of text block features and document typesetting features.

[0118] Of course, other methods can also be used to perform text block detection, such as CNN convolutional neural network, etc.

[0119] S24: Using the text block feature as input, performing layout recognition on the document source to be processed to obtain document layout features of the document source to be processed, wherein the document layout features are at least used to characterize the layout area to which the text block belongs and the layout type of the layout area.

[0120] The text block features can be used as input, and the layout recognition steps in the model can be used to perform layout area segmentation and layout area classification, cut out multiple layout areas, and obtain the type corresponding to each layout area. The types of layout areas may include regular text areas, table areas, irregular text areas, non-text images (such as attached figure masks, etc.). And, for the table area, table parsing and recognition are performed based on the table wireframe graphic feature extraction step. Based on the file layout feature sequence obtained by the above processing, the layout information of the text in the document source to be processed can be more accurately characterized, so that the semantic information of the text information extracted subsequently is more accurate. For the specific implementation method, reference can be made to the embodiment of the above model construction, which will not be repeated here. Of course, other algorithms can also be used to perform layout recognition, which is not limited here.

[0121] Accordingly, in some embodiments, the text block features are used as input to segment and classify the layout area of ​​the document source to be processed, and the document layout features used to characterize the layout area to which the text block belongs and the layout type of the layout area are obtained; for the layout area of ​​the table type, the wireframe graphic information of the table type layout area is converted into document layout features using an image semantic segmentation algorithm. By further extracting layout features from the table type layout area, the information extraction of table-type documents such as tax payment certificates can be made more in line with the special layout format of the document, further avoiding confusion in the extracted information.

[0122] In some embodiments, the image semantic segmentation algorithm includes table area segmentation recognition and table frame line detection; wherein the table area segmentation recognition is used to divide cells based on table frame lines, and the cell information is used as the document layout feature; the table frame line detection refers to regressing the graphic boundary based on Faster-RCNN, detecting the horizontal and vertical line frames of the table, and using the horizontal and vertical line frame information of the table as the document layout feature. Through the method of this embodiment, by converting the table information into layout information to guide the extraction of text information, the semantics of the extracted text information can be made more accurate.

[0123] Based on the above model building process, in some embodiments, point-wise convolution can be used to perform layout area segmentation and classification on the document image source to be processed; wherein, the point-wise convolution configures convolution parameters based on the text features, slice features and text box stitching detection features of the document image source to be processed, wherein the convolution parameters include the number of feature maps, and the filtering size, weight parameters and sliding step size of the convolution kernel.

[0124] Correspondingly, when multiple document image sources with differences in document typesetting are extracted as sample document image sources, and the sample document image sources are subjected to text fast annotation and document typesetting annotation to obtain a sample set, point-wise convolution can be used to perform layout area segmentation and classification on the sample document image sources; wherein, the point-wise convolution configures convolution parameters based on the text features, slice features and text box splicing detection features of the sample document image sources as well as the annotated slice features and annotated text box splicing detection features, wherein the convolution parameters include the number of feature maps, and the filtering size, weight parameters, and sliding step size of the convolution kernel.

[0125] The point-wise convolution method makes up for the defect of traditional convolution that "each channel performs convolution operation independently and does not effectively utilize the feature information of different channels at the same spatial position". The characteristics of the point-wise convolution method are particularly suitable for extracting feature maps of subtle text graphics, thereby achieving higher accuracy in recognizing dense text areas and further improving the accuracy of document layout information extraction.

[0126] S26: Based on the text block features and document layout features of the document source to be processed, information is extracted from the document source to be processed to obtain document structured information of the document source to be processed.

[0127] The text block features and document layout features can be encoded and serialized based on the transformer algorithm to obtain the encoding sequence of the document image source to be processed; wherein the transformer algorithm includes an encoder for text block features and an encoder for document layout features; based on the encoding sequence of the document image source to be processed, information extraction is performed on the document image source to be processed. The specific implementation method can refer to the model construction and will not be described here. By encoding and serializing the text block features and document layout features, the feature maps of multiple dimensions corresponding to a source can be fused into a final two-dimensional sequence, thereby reducing the data processing burden for the subsequent text information extraction algorithm and reducing the difficulty of text information extraction.

[0128] In some embodiments, based on the coding sequence of the document source to be processed, a semantic feature encoder of a predefined information extraction structure can be used to extract information from the document source to be processed, so as to obtain document structured information that satisfies the information extraction structure. For example, the annotation data samples generated by the "text block value annotation" and "layout classification field annotation" of the previous text serialization sample can be serialized by Transformer. After the serialization, these annotation samples can describe the format structure followed by the subsequent information extraction. Constructing a semantic feature encoder based on the predefined information extraction structure and then processing the coding sequence can make the final output text information more neatly presented and improve the user experience.

[0129] This embodiment provides a computer device, whose internal structure diagram can be shown as follows: Figure 6 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection.

[0130] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0131] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0132] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0133] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0134] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0135] It should also be understood that in the embodiments of the present application, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present application generally indicates that the associated objects before and after are in an "or" relationship.

[0136] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0137] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0138] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.

[0139] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0140] Specific embodiments are used in this application to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core idea of ​​this application. At the same time, for those skilled in the art, according to the idea of ​​this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on this application.

Claims

1. A document information extraction method, characterized in that: The method comprises: Perform text block detection on the document source to be processed to obtain text block features of the document source to be processed, wherein the text block features are used to characterize at least text information and adjacent connection relationship information of the text blocks in the document source to be processed; Using the text block feature as input, performing layout recognition on the document image source to be processed, and obtaining the document layout feature of the document image source to be processed, wherein the document layout feature is at least used to characterize the layout area to which the text block belongs and the layout type of the layout area; Based on the text block features and document layout features of the document image source to be processed, information is extracted from the document image source to be processed to obtain document structured information of the document image source to be processed.

2. The method according to claim 1, characterized in that The step of taking the text block feature as input and performing layout recognition on the document source to be processed includes: Taking the text block feature as input, segmenting and classifying the layout area of ​​the document image source to be processed, and obtaining the document layout feature for characterizing the layout area to which the text block belongs and the layout type of the layout area; For a layout area whose layout type is a table type, an image semantic segmentation algorithm is used to convert the wireframe graphic information of the table type layout area into document layout features.

3. The method according to claim 2, characterized in that The performing text block detection on the image source of the document to be processed to obtain the text block features of the image source of the document to be processed includes: Extract text features from the image source of the document to be processed to obtain text features; Performing slicing detection on the image source of the document to be processed, and generating slicing features of the image source of the document to be processed; Performing same-layer neighbor detection and cross-layer neighbor detection on the slices of the document image source to be processed to obtain neighbor relationship features; The slice feature and the adjacency relationship feature are spliced ​​together to obtain the text box splicing detection feature of the document source to be processed.

4. The method according to claim 3, characterized in that The layout area segmentation and classification of the document image source to be processed includes: Point-wise convolution is used to segment and classify the layout area of ​​the document source to be processed; wherein, the point-wise convolution configures convolution parameters based on the text features, slice features and text box stitching detection features of the document source to be processed, wherein the convolution parameters include the number of feature maps, and the filter size, weight parameter and sliding step size of the convolution kernel.

5. The method according to claim 3, characterized in that: The method further comprises: A plurality of document image sources with different document layouts are extracted as sample document image sources, and the sample document image sources are annotated with text blocks and document layouts to obtain a sample set, so as to perform text block detection and layout recognition on the sample document image sources based on the sample set.

6. The method according to claim 5, characterized in that The layout area segmentation and classification of the document image source to be processed includes: Point-wise convolution is used to segment and classify the layout area of ​​the sample document source; wherein the point-wise convolution configures convolution parameters based on the text features, slice features and text box stitching detection features of the sample document source as well as the annotated slice features and annotated text box stitching detection features, wherein the convolution parameters include the number of feature maps, and the filter size, weight parameter and sliding step size of the convolution kernel.

7. The method according to claim 2, characterized in that The image semantic segmentation algorithm includes table area segmentation and recognition and table frame line detection; wherein, the table area segmentation and recognition is used to divide cells based on the table frame line, and the cell information is used as the document layout feature; the table frame line detection refers to regressing the graphic boundary based on Faster-RCNN, detecting the horizontal and vertical line frames of the table, and using the horizontal and vertical line frame information of the table as the document layout feature.

8. The method according to claim 1, characterized in that The extracting information from the image source of the document to be processed based on the text block features and document layout features of the image source of the document to be processed includes: Based on the transformer algorithm, the text block features and the document layout features are coded and serialized to obtain the coding sequence of the document image source to be processed; wherein the transformer algorithm includes an encoder for text block features and an encoder for document layout features; Based on the coding sequence of the document image source to be processed, information is extracted from the document image source to be processed.

9. The method according to claim 8, characterized in that The extracting information from the image source of the document to be processed based on the coding sequence of the image source of the document to be processed includes: Based on the coding sequence of the document image source to be processed, a semantic feature encoder of a predefined information extraction structure is used to extract information from the document image source to obtain document structured information that meets the information extraction structure.

10. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the method according to any one of claims 1 to 9 when executing the computer program.

Images