Document Element Extraction Method, Device, Equipment and Storage Medium

Abstract

The present application discloses a document element extraction method, apparatus, device, and storage medium, including: obtaining layout structure information of a document; encoding each word in the document according to the layout structure information; and determining the element tag to which each word belongs based on the encoding result of each word. This application incorporates the layout structure information of the document when encoding each word in the document, and determines the element tag to which each word belongs based on the word encoding result that incorporates the document encoding structure information, thereby improving the accuracy of document element extraction.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and more specifically, to a document element extraction method, apparatus, device and storage medium. Background Art

[0002] Feature extraction, a crucial subfield of natural language processing, primarily involves extracting structured information from unstructured text. Currently available document feature extraction methods primarily rely on deep learning-based models, but these methods suffer from poor accuracy. Summary of the Invention

[0003] In view of this, the present application provides a document element extraction method, apparatus, device and storage medium to improve the accuracy of document element extraction.

[0004] In order to achieve the above objectives, the following solutions are proposed:

[0005] A document element extraction method, comprising:

[0006] Obtaining layout structure information of the document;

[0007] encoding each word in the document according to the layout structure information;

[0008] The element label to which each character belongs is determined based on the encoding result of each character.

[0009] In the above method, preferably, obtaining the layout structure information of the document includes:

[0010] Processing the image containing the document to obtain semantic features of each word in the document and position features corresponding to each text line;

[0011] For each text line, the semantic features of each word in the text line and the corresponding position features are fused to obtain the encoding features of the text line;

[0012] The coding features of each text line are decoded to obtain the layout structure information of the document.

[0013] In the above method, preferably, the process of obtaining the coding features of each text line and the layout structure information of the document includes:

[0014] Inputting the semantic features and corresponding position features of each word in each text line into a layout analysis model in a document element extraction model, obtaining the layout analysis model fusing the semantic features and corresponding position features of each word in the text line for each text line to obtain encoding features of the text line, and decoding the encoding features of each text line to output layout structure information;

[0015] The layout analysis model is trained by taking the semantic features and corresponding position features of each word in each text line of a sample image as input, the labeled layout structure information of the sample image as a sample label, and aiming to ensure that the layout structure information output by the layout analysis model is close to the sample label.

[0016] In the above method, preferably, the processing of the image containing the document to obtain the semantic features of each word in the document and the position features corresponding to each text line includes:

[0017] Performing optical character recognition on the image using a character recognition model in the document element extraction model to obtain each text line in the document and the coordinates of the text line;

[0018] Performing a first encoding on each word in each text line using the context representation model in the document element extraction model to obtain a semantic feature of each word;

[0019] The first feature extraction module of the text line position feature extraction model in the document element extraction model is used to extract features of the image to obtain a feature map; the second feature extraction module of the text line position feature extraction model is used to extract the position features corresponding to each text line in the feature map according to the coordinates of each text line; the first feature extraction module is a feature extraction module of a pre-trained text line boundary detection model.

[0020] In the above method, preferably, the text line boundary detection model is trained in the following manner:

[0021] Inputting a sample image into the text line boundary detection model, and performing feature extraction on the input sample image through a feature extraction module of the text line boundary detection model to obtain a feature map of the sample image;

[0022] Processing the feature map of the sample image through the output module of the text line boundary detection model to obtain the text line boundary coordinates in the sample image;

[0023] Updating the parameters of the text line boundary detection model with the goal of ensuring that the text line boundary coordinates output by the text line boundary detection model are close to the label of the sample image;

[0024] The label of the sample image is: the boundary coordinates of each text line marked for the sample image.

[0025] In the above method, preferably, the text line boundary detection model is trained in the following manner:

[0026] Inputting a sample image into the text line boundary detection model, and performing feature extraction on the input sample image through a feature extraction module of the text line boundary detection model to obtain a feature map of the sample image;

[0027] Processing the feature map of the sample image through the output module of the text line boundary detection model to obtain text line boundary coordinates in the sample image and the category of the area corresponding to each text line boundary coordinate;

[0028] Updating the parameters of the text line boundary detection model with the text line boundary coordinates output by the text line boundary detection model and the category of the area corresponding to each text line boundary coordinate being close to the label of the sample image;

[0029] The labels of the sample images are: the boundary coordinates of each text line annotated for the sample image, and the category of the area corresponding to each text line boundary coordinate.

[0030] In the above method, preferably, encoding each word in the document according to the layout structure information, and determining the element label to which each word belongs according to the encoding result of each word, includes:

[0031] The extraction model in the document element extraction model is used to encode each word in the document according to the layout structure information, and the element label to which each word belongs is determined according to the encoding result of each word; the extraction model is trained in the following way:

[0032] Inputting the layout structure information and each text line in the document into the extraction model, obtaining that the extraction model encodes each word in the input text line according to the input layout structure information, and determining the element label to which each word belongs based on the encoding result of each word;

[0033] With the goal of making the element labels of each word output by the extraction model close to the labels of the sample image, the parameters of the extraction model are updated;

[0034] The labels of the sample images are element labels to which each word annotated on the sample images belongs.

[0035] In the above method, preferably, the layout structure information at least includes: paragraph division, title level, header, and footer.

[0036] In the above method, preferably, encoding each word in the document according to the layout structure information includes:

[0037] According to the layout structure information, a heterogeneous graph is constructed based on the document, wherein the nodes in the heterogeneous graph include word nodes, title nodes, and text segment nodes; the edges in the heterogeneous graph include: relationships between words, relationships between words and text segments, and relationships between text segments and titles;

[0038] Performing graph convolution on the heterogeneous graph to obtain encoding results of each node;

[0039] The encoding results of each word are merged with the encoding results of the corresponding title nodes to obtain the encoding results of each word.

[0040] In the above method, preferably, the title nodes in the heterogeneous graph are the titles in the document; the text segment nodes in the heterogeneous graph are the text segments in the document;

[0041] or,

[0042] The title node in the heterogeneous graph is the target title in the document, and the level of the target title is higher than the target level; the text segment node in the heterogeneous graph includes the non-target titles and each text segment in the document; the level of the non-target title is lower than or equal to the target level.

[0043] In the above method, preferably, the initial value of each node in the heterogeneous graph is determined by:

[0044] Taking text units in the document as units, performing a second encoding on each word in each text unit to obtain a context feature representation of each word in each text unit in the text unit, which serves as an initial value of each byte in the heterogeneous graph; each text unit is a title or a text segment;

[0045] For any title node, the context feature representations of each word in the title of the title node are fused to obtain the initial value of the title node;

[0046] For any text segment node, the context feature representations of the title of the text segment node or each word in the text segment are fused to obtain the initial value of the text segment node.

[0047] A document element extraction device, comprising:

[0048] An obtaining unit, configured to obtain the layout structure information of the document;

[0049] an encoding unit, configured to encode each word in the document according to the layout structure information;

[0050] The extraction unit is used to determine the element label to which each word belongs according to the encoding result of each word.

[0051] A document element extraction device includes a memory and a processor;

[0052] The memory is used to store programs;

[0053] The processor is used to execute the program to implement each step of the document element extraction method as described in any one of the above items.

[0054] A computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, each step of the document element extraction method as described in any one of the above items is implemented.

[0055] As can be seen from the above technical solutions, the document element extraction method, apparatus, device, and storage medium provided in the embodiments of the present application obtain the layout structure information of a document; encode each word in the document based on the layout structure information of the document; and determine the element label to which each word belongs based on the encoding result of each word. When encoding each word in a document, the present application incorporates the layout structure information of the document, and determines the element label to which each word belongs based on the word encoding result that incorporates the document encoding structure information, thereby improving the accuracy of document element extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative work.

[0057] Figure 1 A flowchart of an implementation of the document element extraction method disclosed in an embodiment of the present application;

[0058] Figure 2 This is an example diagram of the element recognition result disclosed in the embodiment of this application;

[0059] Figure 3 A flowchart for obtaining document layout structure information disclosed in an embodiment of the present application;

[0060] Figure 4 A schematic diagram of the structure of the document element extraction model disclosed in the embodiment of this application;

[0061] Figure 5 A flowchart for implementing encoding of each word in a document according to layout structure information disclosed in an embodiment of the present application;

[0062] Figure 6 Another schematic diagram of the document element extraction model disclosed in the embodiment of this application;

[0063] Figure 7 A schematic diagram of the structure of the document element extraction device disclosed in the embodiment of the present application;

[0064] Figure 8 This is a hardware structure block diagram of the document element extraction device disclosed in the embodiment of this application. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0066] Current document feature extraction methods based on deep learning models obtain vector representations of each word in a document, input these vector representations into a pre-trained neural network, and obtain the feature labels associated with each word. This feature extraction method suffers from low accuracy. To improve the accuracy of document feature extraction, this application proposal is proposed.

[0067] like Figure 1 As shown, a flowchart of an implementation method of the document element extraction method provided in an embodiment of the present application may include:

[0068] Step S101: Obtaining document layout structure information.

[0069] Optionally, the documents in this application can be documents in any field. For example, they can be documents in the financial field (such as contract texts), or documents in other fields, such as documents in the legal field (such as judgment documents), or documents in the medical field (such as medical records), etc.

[0070] As an example, the layout structure information of the document can be obtained by processing a picture containing the document (ie, a document in picture format, which is referred to as a picture document for ease of description).

[0071] As an example, the layout structure information may include, but is not limited to, paragraph division, title level, header, footer and other information.

[0072] Step S102: Encode each word in the document according to the layout structure information of the document.

[0073] The present application encodes each word in a document based on the layout structure information of the document, so that the encoding result of each word integrates the layout structure information of the document.

[0074] Step S103: Determine the element label to which each character belongs based on the encoding result of each character.

[0075] As an example, the encoding results of each word can be decoded through a conditional random field (CRF) model to obtain the element label to which each word belongs.

[0076] like Figure 2 As shown, it is an example diagram of the element recognition result provided by the embodiment of the present application. In this example, the element <Party A> is recognized, that is, "Zhang San" is recognized as the element: Party A.

[0077] Because the location of elements in a document is closely related to the document's layout structure, for example, if a heading is titled "1. Basic Information of Party A in the Contract," the paragraphs under this heading are likely to contain the element "Party A." Therefore, determining the element label for each word based on word encoding that incorporates information about the document's layout structure can improve the accuracy of document element extraction.

[0078] In an optional embodiment, a flowchart of obtaining the layout structure information of a document is as follows: Figure 3 As shown, this may include:

[0079] Step S301: Process the image containing the document to obtain the semantic features of each word in the document and the position features corresponding to each text line.

[0080] For an English document, each word in the document is a character.

[0081] As an example, optical character recognition (OCR) can be performed on an image document to obtain each text line in the document and the coordinates of each text line. The coordinates of each text line (recorded as the i-th text line for ease of description and distinction) can be represented by the coordinates of the four vertices of the rectangular box covering the i-th text line.

[0082] Each character in the i-th text line is encoded (referred to as the first encoding for ease of description and distinction) to obtain semantic features of each character in the i-th text line. i = 1, 2, 3, ..., N, where N is the total number of text lines in the document. Optionally, the i-th text line can be input into a pre-trained contextual representation model to obtain contextual feature representations for each character in the i-th text line. As an example, the pre-trained contextual representation model can be a pre-trained BERT model.

[0083] The position feature of the i-th text line is determined based on the coordinates of the i-th text line. As an example, the position feature of the i-th text line can be determined in any of the following two ways:

[0084] Method 1: The coordinates of the i-th text line are determined as the position features of the i-th text line.

[0085] Method 2: The first feature extraction module of the text line position feature extraction model is used to extract features of the image document to obtain a feature map; the second feature extraction module of the text line position feature extraction model is used to extract the position features corresponding to the i-th text line in the feature map according to the coordinates of the i-th text line.

[0086] The first feature extraction module is a feature extraction module of a pre-trained text line boundary detection model. As an example, the text line boundary detection model can be implemented by a cascade region-based convolutional neural network (Cascade-RCNN) based on pre-selected regions.

[0087] In an optional implementation, the input of the text line boundary detection model is a picture containing a document (i.e., a picture document), and the output of the text line boundary detection model is the coordinates of the text line boundaries in the input picture document (which can be the vertex coordinates of the rectangular box covering the text line). The samples used to train the text line boundary detection model are pictures containing documents (which can be called sample pictures), and the sample labels are: the boundary coordinates of each text line annotated for the sample picture. The text line boundary detection model can be trained in the following way:

[0088] The sample image is input into the text line boundary detection model, and the text line boundary detection model extracts features from the input sample image through the feature extraction module to obtain a feature map, and processes the feature map through the output module to output the text line boundary coordinates; with the goal of making the text line boundary coordinates output by the text line boundary detection model approach the sample label, the parameters of the text line boundary detection model are updated until the training end conditions are met.

[0089] In another optional implementation, the input of the text line boundary detection model is a picture containing a document (i.e., a picture document), and the output of the text line boundary detection model is the text line boundary coordinates in the input picture document (which can be the vertex coordinates of the rectangular box covering the text line), and the category of the area corresponding to each text line boundary coordinate (for example, text line, table, etc.). The sample used to train the text line boundary detection model is a picture containing a document, and the sample label is: the boundary coordinates of each text line annotated for the sample picture, and the category of each text line area. The text line boundary detection model can be trained in the following way:

[0090] The sample image is input into the text line boundary detection model, and the text line boundary detection model extracts features from the input sample image through the feature extraction module to obtain a feature map. The feature map is processed by the output module to output the text line boundary coordinates and the category of the area corresponding to the text line boundary coordinates; the parameters of the text line boundary detection model are updated with the goal that the text line boundary coordinates and the area category output by the text line boundary detection model are close to the sample label until the training end conditions are met.

[0091] Optionally, in the second method, one implementation method of extracting the position feature corresponding to the i-th text line in the feature map according to the coordinates of the i-th text line may be:

[0092] The bounding box corresponding to the i-th text line is determined according to the coordinates of the i-th text line.

[0093] Get the scale of the feature map relative to the image (i.e., the image containing the document).

[0094] The bounding box of the i-th text line is scaled according to the obtained scaling ratio, so that the scaling ratio of the bounding box after scaling relative to the bounding box before scaling is equal to the scaling ratio of the feature map relative to the image.

[0095] In the feature map, the features within the scaled bounding box area are extracted as the position features corresponding to the i-th text line. In other words, the features within the scaled bounding box area in the feature map are taken as the position features corresponding to the i-th text line.

[0096] Step S302: For each text line, the semantic features of each word in the text line and the corresponding position features are fused to obtain the coding features of the text line.

[0097] As an example, the semantic features of each word in the text line and the corresponding position features can be fused through the feature fusion module in the pre-trained layout analysis model.

[0098] The layout analysis model can use the feature fusion module to average the semantic features of each character in the i-th text line to obtain the semantic features of the i-th text line (that is, sum the semantic features of each character, divide the sum by the number of characters in the i-th text line, and obtain the semantic features of the i-th text line), splice the semantic features of the i-th text line with the position features of the i-th text line, and encode the spliced features through a bidirectional long short-term memory network (Long Short-Term Memory, LSTM) or RNN (Recurrent Neural Network) to obtain the encoded features of the i-th text line.

[0099] Step S303: Decode the coding features of each text line to obtain the layout structure information of the document.

[0100] The encoding features of each text line can be decoded by the decoding module in the pre-trained layout analysis model (for the convenience of distinction, it is recorded as the first decoding module). As an example, the first decoding module can be implemented based on a two-layer unidirectional GRU network. The two layers of single-item GRU are connected in series. Among them, the parent GRU is used to fuse the encoding features of the current text line to be decoded (for the convenience of distinction and description, it is recorded as the i-th text line) and the hidden layer features of the previous text line (i.e., the i-1-th text line) to obtain the target fusion feature (for the convenience of distinction and description, it is recorded as the first target fusion feature), decode the first target fusion feature, and obtain a partial decoding result (recorded as the first decoding result) of the current text line to be decoded (i.e., the i-th text line). The child GRU is used to fuse the encoding features of the current text line (i.e., the i-th text line) and the first target fusion feature output by the parent GRU to obtain another target fusion feature (for the convenience of distinction and description, it is recorded as the second target fusion feature), decode the second target fusion feature, and obtain another partial decoding result (recorded as the second decoding result) of the current text line to be decoded.

[0101] The first decoding result is the attribute of the i-th text line. The attribute of the text line is one of the following attributes: title level, text paragraph, header, and footer.

[0102] The second decoding result is the associated text line of the ith text line, and the relationship between the ith text line and the associated text line.

[0103] That is to say, in the present application, for each text line, the first decoding module has three outputs, namely: the attributes of the text line, the associated text lines of the text line, and the relationship between the text line and the associated text lines.

[0104] The associated text line of the i-th text line may be the previous line of the i-th text line, or may be a line of a title before the i-th text line.

[0105] The relationship between the i-th text line and the associated text line is one of the following relationships: a parallel relationship (for example, there are two subheadings under a first-level title, and these two subheadings both belong to the second-level title, then the text lines in these two subheadings (belonging to different subheadings) are in a parallel relationship; the last line of the previous text segment and the first line of the next text segment in two adjacent text segments are in a parallel relationship), a progressive relationship (for example, the text line of the aforementioned second-level subheading and the text line in a first-level title are in a progressive relationship), and a connection relationship (for example, two adjacent text lines in the same text segment are in a connection relationship).

[0106] Optionally, one implementation method of fusing the encoding features of the current text line to be decoded (i.e., the i-th text line) and the hidden features of the previous text line (i.e., the i-1-th text line) can be: splicing the encoding features of the current text line to be decoded and the hidden features of the previous text line, performing dimensionality transformation on the spliced features, and obtaining the first target fusion features.

[0107] Optionally, one implementation method of fusing the encoding features of the current text line (i.e., the i-th text line) and the first target fusion features output by the parent GRU can be: splicing the encoding features of the current text line to be decoded with the first target fusion features, performing dimensionality transformation on the spliced features, and obtaining the second target fusion features.

[0108] Optionally, the hidden layer feature of the i-1th text line is the second target fusion feature at the previous moment, that is, the second target fusion feature obtained when the sub-GRU decodes the associated text line of the i-1th text line and the relationship between the i-1th text line and the associated text line.

[0109] Optionally, when the i-th text line is the first text line, the hidden feature of the previous text line may be the average of the encoding features of each text line in the document.

[0110] Optionally, the input of the layout analysis model is the semantic features and corresponding positional features of each character in each text line obtained by processing the image containing the document in step S301. The output of the layout analysis model is the attributes of each text line (text segment, title, header, or footer), as well as the associated text lines of each text line and the relationship between each text line and its associated text lines.

[0111] The samples used to train the layout analysis model are images containing documents, and the sample labels are: the attributes of each text line annotated for the sample image, the associated text lines of each text line, and the relationship between each text line and the associated text lines.

[0112] If the attribute of a text line is a text paragraph, header or footer, the associated text line of the text line is the previous line of the text line. In this case, the relationship between the text line and the associated text line is a parallel relationship or a connection relationship.

[0113] If the attribute of a text line is a first-level title, then the associated text line of the text line belongs to the text line of the title. Among them, if the previous title of the first-level title is also a first-level title, then the associated text line of the text line belongs to the text line of the previous title, and the relationship between the text line and the associated text line is a parallel relationship; if the previous title of the first-level title is a second-level title, and the second level is higher than the first level, then the associated text line of the text line belongs to the text line of the previous title, and the relationship between the text line and the associated text line is a progressive relationship; if the previous title of the first-level title is a second-level title, and the second level is lower than the first level, then the associated text line of the text line is the text line of the first-level title closest to the text line, and the relationship between the text line and the associated text line is a parallel relationship.

[0114] The training process of the layout analysis model may include:

[0115] The sample image is processed through step S301 to obtain the semantic features and corresponding position features of each character in each text line in the sample image, and the semantic features and corresponding position features of each character in each text line are input into the layout analysis model to obtain the attributes of each text line output by the layout analysis model, the associated text lines of each text line, and the relationship between each text line and the associated text lines; with the goal of making the attributes of each text line output by the layout analysis model, the associated text lines of each text line, and the relationship between each text line and the associated text lines approach the sample label, the parameters of the layout analysis model are updated until the training end conditions are met.

[0116] Optionally, the document element extraction method provided in the embodiment of the present application can be implemented by a document element extraction model. Figure 4 As shown, a schematic diagram of the structure of the document element extraction model provided in an embodiment of the present application may include:

[0117] Character recognition model 401, context representation model 402, text line position feature extraction model 403, layout analysis model 404, extraction model 405; wherein,

[0118] The character recognition model 401 is used to perform optical character recognition on a picture containing a document (ie, a picture document) to obtain each text line in the document and the coordinates of the text line. As an example, the character recognition model 401 may be an OCR model.

[0119] The context representation model 402 is used to perform a first encoding on each character in each text line output by the character recognition model 401 to obtain semantic features of each character. The context representation model 402 may be a pre-trained BERT model.

[0120] The text line position feature extraction model 403 is used to extract features from the image containing the document using a first feature extraction module to obtain a feature map. A second feature extraction module then extracts the position features corresponding to each text line from the feature map based on the coordinates of each text line output by the character recognition model 401. The first feature extraction module is a feature extraction module for a pre-trained text line boundary detection model. The training process for the text line boundary detection model has been described in the previous embodiment and will not be further elaborated here.

[0121] The layout analysis model 404 is used to fuse the semantic features of each character in each text line output by the character recognition model 401 and the corresponding position features output by the text line position feature extraction model 403 through the feature fusion module to obtain the encoding features of the text line; and decode the encoding features of each text line through the first decoding module to obtain the layout structure information of the document.

[0122] Obviously, step S301 is implemented by character recognition model 401, context representation model 402, and text line position feature extraction model 403. Steps S302-S303 are implemented by layout analysis model 404. In other words, step S101 is implemented by character recognition model 401, context representation model 402, text line position feature extraction model 403, and layout analysis model 404.

[0123] The extraction model 405 is used to encode each character in each text output by the character recognition model 401 according to the layout structure information output by the layout analysis model 404, and determine the element label to which each character belongs based on the encoding result of each character.

[0124] Obviously, steps S102 - S103 are implemented by the extraction model 405 .

[0125] Optionally, the character recognition model 401, the context representation model 402, the text line boundary detection model, the layout analysis model 404 and the extraction model 405 can be trained independently.

[0126] The training process of the character recognition model 401 and the context representation model 402 can refer to existing implementation solutions and will not be described in detail here.

[0127] The training process of the text line boundary detection model and the layout analysis model 404 can be found in the aforementioned embodiment and will not be described again here.

[0128] The samples used to train the extraction model 405 are documents and the layout structure information of the documents. The layout structure information in the samples can be manually annotated or obtained by processing the image containing the document through the aforementioned character recognition model 401, context representation model 402, text line position feature extraction model 403 and layout analysis model 404; the sample labels are: the element labels of each word annotated for each word in the document.

[0129] The input of the extraction model 405 is the document and its layout structure information, and the output of the extraction model 405 is the element label to which each word in the input document belongs.

[0130] The extraction model 405 may be trained as follows:

[0131] The document serving as a sample and the layout structure information of the document are input into the extraction model 405 , and the element label to which each word outputted by the extraction model 405 belongs is obtained.

[0132] The parameters of the extraction model 405 are updated with the goal of making the element labels of each word output by the extraction model 405 close to the sample labels until the training end condition is met.

[0133] Optionally, the character recognition model 401, the context representation model 402, and the text line boundary detection model can be trained separately, and the layout analysis model 404 and the extraction model 405 can be trained jointly.

[0134] The samples for joint training of the layout analysis model 404 and the extraction model 405 are pictures containing documents (i.e., picture text), and the sample labels are: the attributes of each text line annotated for the picture document, the associated text lines of each text line, the relationship between each text line and the associated text lines, and the element labels to which each word in the document belongs.

[0135] The process of jointly training the layout analysis model 404 and the extraction model 405 may include:

[0136] The character recognition model 401 performs optical character recognition on the sample image to obtain each text line in the document contained in the sample image and the coordinates of the text line.

[0137] The context representation model 402 performs a first encoding on each character in each text output by the character recognition model 401 to obtain a semantic feature of each character.

[0138] The text line position feature extraction model 403 is used to extract features from the sample image to obtain a feature map; and the position features corresponding to each text line are extracted from the feature map according to the coordinates of each text line output by the character recognition model 401.

[0139] The semantic features of each word in each text line in the sample image and the position features corresponding to the text line are input into the layout analysis model 404 to obtain the layout structure information output by the layout analysis model 404 (including the attributes of each text line in the sample, the associated text lines of each text line, and the relationship between each text line and the associated text lines).

[0140] The layout structure information output by the layout analysis model 404 and the text lines in the sample output by the character recognition model 401 are input into the extraction model 405 to obtain the element labels to which each character in the text line output by the extraction model 405 belongs.

[0141] With the goal of making the layout structure information output by the layout analysis model 404 and the element labels of each word output by the extraction model 405 approach the sample labels, the parameters of the layout analysis model 404 and the extraction model 405 are updated until the training end conditions are met.

[0142] In an optional embodiment, a flowchart for implementing encoding of each word in a document according to the layout structure information is shown below: Figure 5 As shown, this may include:

[0143] Step S501: According to the layout structure information of the document, a heterogeneous graph is constructed based on the document. The nodes in the heterogeneous graph include word nodes, title nodes, and text segment nodes; the edges in the heterogeneous graph include: the relationship between words, the relationship between words and text segments, and the relationship between text segments and titles.

[0144] This application determines titles and text segments from a document based on layout structure information, and then constructs a heterogeneous graph based on the words, titles, and text segments in the document.

[0145] When determining the title, the titles of each level are determined.

[0146] The graph convolution module in the extraction model 405 can construct a heterogeneous graph based on the document according to the layout structure information of the document.

[0147] In this application, the nodes in the heterogeneous graph include three categories: byte nodes, title nodes, and text segment nodes. The edges in the heterogeneous graph reflect the relationships between nodes, which also include three types: the relationship between words, the relationship between words and text segments, and the relationship between text segments and titles. The relationship between words can be the co-occurrence frequency or the number of co-occurrences, etc. The relationship between words and text segments can be the importance of words to the text segment, such as the tf-idf (term frequency–inverse document frequency) of the words. The relationship between text segments and titles can be whether the text segment and the title are related. If the text segment belongs to the content under the title, then the text segment is related to the title; otherwise, the text segment and the title are not related.

[0148] As an example, the title nodes in the heterogeneous graph are the titles in the document, that is, each title is a title node; the text segment nodes in the heterogeneous graph are the text segments in the document.

[0149] As an example, the title node in the heterogeneous graph is the target title in the document, and the hierarchy of the target title is higher than the target hierarchy; the text segment nodes in the heterogeneous graph include non-target titles and text segments in the document; the hierarchy of non-target titles is lower than or equal to the target hierarchy. In other words, this application only uses some titles (i.e., titles with a hierarchy higher than the target hierarchy) as title nodes in the heterogeneous graph, and the other part of the titles (i.e., titles with a hierarchy lower than or equal to the target hierarchy) as some text segment nodes in the heterogeneous graph. Then, the text segment nodes in the heterogeneous graph include two parts: one part of the text segment nodes is the text segment in the document that is not a title, and the other part of the text segment nodes is a title.

[0150] Step S502: Perform graph convolution on the heterogeneous graph to obtain the encoding result of each node.

[0151] Graph convolutional networks can be used to perform graph convolution on heterogeneous graphs to obtain the encoding results of each node. The specific implementation process is not the focus of this application. You can refer to existing implementation methods and will not be described in detail here.

[0152] The graph convolution module (i.e., graph convolution network) in the extraction model 405 can perform graph convolution on the heterogeneous graph.

[0153] Step S503: The encoding result of each character is merged with the encoding result of the corresponding title node to obtain the encoding result of each character.

[0154] The title node corresponding to a word W is the title node of the text segment where the word W is located. For example,

[0155] Assume that there are two sub-headings T11 and T12 under a first-level heading T1, and each sub-heading has a text segment. Assume that the text segment under the sub-heading T11 is D1, and the text segment under the sub-heading T12 is D2. Then,

[0156] If the first-level title T1, the second-level titles T11 and T12 are all title nodes, and W belongs to the text segment D1, then the title node corresponding to the word W is T11.

[0157] If only the first-level title T1 is a title node, and the second-level titles T11 and T12 are both text segment nodes, then the title node corresponding to word W is T1.

[0158] As an example, the encoding result of the title node can be linearly transformed so that the dimension of the transformed encoding result is the same as the dimension of the encoding result of the word W. Then, the transformed encoding result of the title node and the encoding result of the word W are spliced or averaged to obtain the encoding result of the word W.

[0159] The graph convolution module in the extraction model 405 can fuse the encoding results of each character with the encoding results of the corresponding title node to obtain the encoding results of each character.

[0160] After obtaining the encoding results for each character, the decoding module of the extraction model 405 (referred to as the second decoding module for ease of distinction) can decode the encoding results for each character output by the graph convolution module to obtain the element label to which each character belongs. The second decoding module can decode the encoding results for each character using a conditional random field (CRF) model to obtain the element label to which each character belongs.

[0161] In an optional embodiment, the initial value of each node in the heterogeneous graph can be determined as follows:

[0162] Taking the text units in the document as units, each word in each text unit (denoted as the jth text unit for ease of distinction and description) is encoded (denoted as the second encoding for ease of description and distinction). The contextual feature representation of each word in each text unit is obtained and used as the initial value of each byte in the heterogeneous graph. j = 1, 2, 3, ..., M, where M is the total number of text units. Each text unit is a title or a text segment.

[0163] As an example, the j-th text unit may be processed by the context representation module in the extraction model 405 to obtain the context feature representation of each word in the j-th text unit in the j-th text unit.

[0164] For any title node, the context feature representations of each word in the title of the title node are fused to obtain the initial value of the title node.

[0165] As an example, the context feature representations of the respective words in the title of the title node may be averaged to obtain the initial value of the title node.

[0166] For any text segment node, the contextual feature representations of the title or each word in the text segment of the text segment node are fused to obtain the initial value of the text segment node. That is, if the text segment node is a title, the contextual feature representations of each word in the title of the text segment node are fused to obtain the initial value of the text segment node. If the text segment node is a text segment, the contextual feature representations of each word in the text segment of the text segment node are fused to obtain the initial value of the text segment node.

[0167] As an example, the context feature representations of each word in the title of a text segment node may be averaged to obtain the initial value of the text segment node. Alternatively, the context feature representations of each word in the text segment of a text segment node may be averaged to obtain the initial value of the text segment node.

[0168] like Figure 6 FIG. 1 is another structural diagram of the document element extraction model provided in the embodiment of the present application. Figure 4 The main difference is that a structural diagram of the extraction model 405 is given. In this example,

[0169] The extraction model 405 may first perform a second encoding on each character in each text unit according to the layout structure information using the text unit in the document as a unit, thereby obtaining a context feature representation of each character in each text unit, and then fuse the context feature representations of each character belonging to the same text unit to obtain a feature representation of the text unit.

[0170] Then the extraction model 405 constructs a heterogeneous graph based on the layout structure information through the graph convolution module, and uses the context feature representation of the characters obtained by the context representation module and the feature representation of each text unit as the initial value of the node of the heterogeneous graph, performs graph convolution on the heterogeneous graph, and obtains the encoding result of each node; the encoding result of each character is fused with the encoding result of the corresponding title node to obtain the encoding result of each character.

[0171] Finally, the extraction model 405 decodes the encoding results of each word through the conditional random field decoding module to obtain the element label to which each word belongs.

[0172] Corresponding to the method embodiment, the present application also provides a document element extraction device. A structural diagram of the document element extraction device provided in the embodiment of the present application is shown as follows: Figure 7 As shown, this may include:

[0173] Acquisition unit 701, encoding unit 702 and extraction unit 703; wherein

[0174] The obtaining unit 701 is used to obtain the layout structure information of the document;

[0175] The encoding unit 702 is used to encode each word in the document according to the layout structure information;

[0176] The extraction unit 703 is used to determine the element label to which each word belongs according to the encoding result of each word.

[0177] The document element extraction device provided in the embodiment of the present application incorporates the layout structure information of the document when encoding each word in the document, and determines the element label to which each word belongs based on the word encoding result that integrates the document encoding structure information, thereby improving the accuracy of document element extraction.

[0178] In an optional embodiment, the obtaining unit 701 includes:

[0179] a feature extraction unit, configured to process the image containing the document to obtain semantic features of each word in the document and position features corresponding to each text line;

[0180] The first fusion unit, for each text line, the user fuses the semantic features of each word in the text line and the corresponding position features to obtain the encoding features of the text line;

[0181] The decoding unit is used to decode the coding features of each text line to obtain the layout structure information of the document.

[0182] In an optional embodiment, when the first fusion unit obtains the encoding features of each text line and the decoding unit obtains the layout structure information of the document, it is configured to:

[0183] Inputting the semantic features and corresponding position features of each word in each text line into a layout analysis model in a document element extraction model, obtaining the layout analysis model fusing the semantic features and corresponding position features of each word in the text line for each text line to obtain encoding features of the text line, and decoding the encoding features of each text line to output layout structure information;

[0184] The layout analysis model is trained by taking the semantic features and corresponding position features of each word in each text line of a sample image as input, the labeled layout structure information of the sample image as a sample label, and aiming to ensure that the layout structure information output by the layout analysis model is close to the sample label.

[0185] In an optional embodiment, when the feature extraction unit processes the image containing the document to obtain the semantic features of each word in the document and the position features corresponding to each text line, it is used to:

[0186] Performing optical character recognition on the image using a character recognition model in the document element extraction model to obtain each text line in the document and the coordinates of the text line;

[0187] Performing a first encoding on each word in each text line using the context representation model in the document element extraction model to obtain a semantic feature of each word;

[0188] The first feature extraction module of the text line position feature extraction model in the document element extraction model is used to extract features of the image to obtain a feature map; the second feature extraction module of the text line position feature extraction model is used to extract the position features corresponding to each text line in the feature map according to the coordinates of each text line; the first feature extraction module is a feature extraction module of a pre-trained text line boundary detection model.

[0189] In an optional embodiment, a training module is further included for training the text line boundary detection model, including:

[0190] Inputting a sample image into the text line boundary detection model, and performing feature extraction on the input sample image through a feature extraction module of the text line boundary detection model to obtain a feature map of the sample image;

[0191] Processing the feature map of the sample image through the output module of the text line boundary detection model to obtain the text line boundary coordinates in the sample image;

[0192] Updating the parameters of the text line boundary detection model with the goal of ensuring that the text line boundary coordinates output by the text line boundary detection model are close to the label of the sample image;

[0193] The label of the sample image is: the boundary coordinates of each text line marked for the sample image.

[0194] In an optional embodiment, a training module is further included for training the text line boundary detection model, including:

[0195] Inputting a sample image into the text line boundary detection model, and performing feature extraction on the input sample image through a feature extraction module of the text line boundary detection model to obtain a feature map of the sample image;

[0196] Processing the feature map of the sample image through the output module of the text line boundary detection model to obtain text line boundary coordinates in the sample image and the category of the area corresponding to each text line boundary coordinate;

[0197] Updating the parameters of the text line boundary detection model with the text line boundary coordinates output by the text line boundary detection model and the category of the area corresponding to each text line boundary coordinate being close to the label of the sample image;

[0198] The labels of the sample images are: the boundary coordinates of each text line annotated for the sample image, and the category of the area corresponding to each text line boundary coordinate.

[0199] In an optional embodiment, the encoding unit 702 encodes each word in the document according to the layout structure information, and the extraction unit 703 determines the element tag to which each word belongs based on the encoding result of each word, and is used to:

[0200] The extraction model in the document element extraction model is used to encode each word in the document according to the layout structure information, and the element label to which each word belongs is determined according to the encoding result of each word; the extraction model is trained in the following way:

[0201] Inputting the layout structure information and each text line in the document into the extraction model, obtaining that the extraction model encodes each word in the input text line according to the input layout structure information, and determining the element label to which each word belongs based on the encoding result of each word;

[0202] With the goal of making the element labels of each word output by the extraction model close to the labels of the sample image, the parameters of the extraction model are updated;

[0203] The labels of the sample images are element labels to which each word annotated on the sample images belongs.

[0204] In an optional embodiment, the layout structure information includes at least: paragraph division, title level, header, and footer.

[0205] In an optional embodiment, the encoding unit 702 includes:

[0206] Extraction unit, the user extracts text units from the document according to the layout structure information; each text unit is a title or a text segment;

[0207] A graph construction unit, configured to construct a heterogeneous graph based on the document, wherein the nodes in the heterogeneous graph include word nodes, title nodes, and text segment nodes; and the edges in the heterogeneous graph include relationships between words, relationships between words and text segments, and relationships between text segments and titles.

[0208] A graph convolution unit, configured to perform graph convolution on the heterogeneous graph to obtain an encoding result for each node;

[0209] The second fusion unit is used to fuse the encoding results of each character with the encoding results of the corresponding title node to obtain the encoding results of each character.

[0210] In an optional embodiment, the title nodes in the heterogeneous graph are the titles in the document; the text segment nodes in the heterogeneous graph are the text segments in the document;

[0211] or,

[0212] The title node in the heterogeneous graph is the target title in the document, and the level of the target title is higher than the target level; the text segment node in the heterogeneous graph includes the non-target titles and each text segment in the document; the level of the non-target title is lower than or equal to the target level.

[0213] In an optional embodiment, the graph convolution unit determines the initial value of each node in the heterogeneous graph by:

[0214] Taking text units in the document as units, performing a second encoding on each word in each text unit to obtain a context feature representation of each word in each text unit in the text unit, which serves as an initial value of each byte in the heterogeneous graph; each text unit is a title or a text segment;

[0215] For any title node, the context feature representations of each word in the title of the title node are fused to obtain the initial value of the title node;

[0216] For any text segment node, the context feature representations of each word in the title of the text segment node are fused to obtain the initial value of the text segment node.

[0217] The document element extraction device provided in the embodiment of the present application can be applied to document element extraction devices, such as PC terminals, cloud platforms, servers and server clusters. Figure 8 The hardware structure diagram of the document element extraction device is shown. Figure 8,The hardware structure of the document element extraction device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;

[0218] In the embodiment of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 communicate with each other through the communication bus 4;

[0219] The processor 1 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention;

[0220] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0221] The memory stores a program, and the processor can call the program stored in the memory, wherein the program is used to:

[0222] Obtaining layout structure information of the document;

[0223] encoding each word in the document according to the layout structure information;

[0224] The element label to which each character belongs is determined based on the encoding result of each character.

[0225] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0226] An embodiment of the present application further provides a storage medium, which may store a program suitable for execution by a processor, wherein the program is used to:

[0227] Obtaining layout structure information of the document;

[0228] encoding each word in the document according to the layout structure information;

[0229] The element label to which each character belongs is determined based on the encoding result of each character.

[0230] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0231] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0232] In the several embodiments provided herein, it should be understood that the disclosed systems, devices, and methods may be implemented in other ways. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0233] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0234] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0235] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0236] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0237] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0238] The above description of the disclosed embodiments will enable those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is to be construed in the widest manner consistent with the principles and novel features disclosed herein.

Claims

1. A document element extraction method, characterized in that: include: Obtaining layout structure information of the document, wherein the layout structure information includes at least paragraph division, title level, header, and footer; encoding each word in the document according to the layout structure information; Determine the element label to which each character belongs based on the encoding result of each character; The step of encoding each word in the document according to the layout structure information includes: Determining titles and text segments from the document based on the layout structure information, and constructing a heterogeneous graph based on the characters, titles, and text segments in the document, wherein the nodes in the heterogeneous graph include character nodes, title nodes, and text segment nodes; and the edges in the heterogeneous graph include relationships between characters, relationships between characters and text segments, and relationships between text segments and titles; Performing graph convolution on the heterogeneous graph to obtain encoding results of each node; The encoding results of each word are merged with the encoding results of the corresponding title nodes to obtain the encoding results of each word.

2. The method according to claim 1, characterized in that The obtaining of the layout structure information of the document includes: Processing the image containing the document to obtain semantic features of each word in the document and position features corresponding to each text line; For each text line, the semantic features of each word in the text line and the corresponding position features are fused to obtain the encoding features of the text line; The coding features of each text line are decoded to obtain the layout structure information of the document.

3. The method according to claim 2, characterized in that The process of obtaining the coding features of each text line and the layout structure information of the document includes: Inputting the semantic features and corresponding position features of each word in each text line into a layout analysis model in a document element extraction model, obtaining the layout analysis model fusing the semantic features and corresponding position features of each word in the text line for each text line to obtain encoding features of the text line, and decoding the encoding features of each text line to output layout structure information; The layout analysis model is trained by taking the semantic features and corresponding position features of each word in each text line of a sample image as input, the labeled layout structure information of the sample image as a sample label, and aiming to ensure that the layout structure information output by the layout analysis model is close to the sample label.

4. The method according to claim 2, characterized in that The processing of the image containing the document to obtain the semantic features of each word in the document and the position features corresponding to each text line includes: Performing optical character recognition on the image using a character recognition model in a document element extraction model to obtain each text line in the document and the coordinates of the text line; Performing a first encoding on each word in each text line using the context representation model in the document element extraction model to obtain a semantic feature of each word; The first feature extraction module of the text line position feature extraction model in the document element extraction model is used to extract features of the image to obtain a feature map; the second feature extraction module of the text line position feature extraction model is used to extract the position features corresponding to each text line in the feature map according to the coordinates of each text line; the first feature extraction module is a feature extraction module of a pre-trained text line boundary detection model.

5. The method according to claim 4, characterized in that The text line boundary detection model is trained in the following way: Inputting a sample image into the text line boundary detection model, and performing feature extraction on the input sample image through a feature extraction module of the text line boundary detection model to obtain a feature map of the sample image; Processing the feature map of the sample image through the output module of the text line boundary detection model to obtain the text line boundary coordinates in the sample image; Updating the parameters of the text line boundary detection model with the goal of ensuring that the text line boundary coordinates output by the text line boundary detection model are close to the label of the sample image; The label of the sample image is: the boundary coordinates of each text line marked for the sample image.

6. The method according to claim 4, characterized in that The text line boundary detection model is trained in the following way: Inputting a sample image into the text line boundary detection model, and performing feature extraction on the input sample image through a feature extraction module of the text line boundary detection model to obtain a feature map of the sample image; Processing the feature map of the sample image through the output module of the text line boundary detection model to obtain text line boundary coordinates in the sample image and the category of the area corresponding to each text line boundary coordinate; Updating the parameters of the text line boundary detection model with the text line boundary coordinates output by the text line boundary detection model and the category of the area corresponding to each text line boundary coordinate being close to the label of the sample image; The labels of the sample images are: the boundary coordinates of each text line annotated for the sample image, and the category of the area corresponding to each text line boundary coordinate.

7. The method according to claim 4, characterized in that The step of encoding each word in the document according to the layout structure information and determining the element tag to which each word belongs according to the encoding result of each word includes: The extraction model in the document element extraction model is used to encode each word in the document according to the layout structure information, and the element label to which each word belongs is determined according to the encoding result of each word; the extraction model is trained in the following way: Inputting the layout structure information and each text line in the document into the extraction model, obtaining that the extraction model encodes each word in the input text line according to the input layout structure information, and determining the element label to which each word belongs based on the encoding result of each word; With the goal of making the element labels of each word output by the extraction model close to the labels of the sample image, the parameters of the extraction model are updated; The labels of the sample images are element labels to which each word annotated on the sample images belongs.

8. The method according to claim 1, characterized in that The title nodes in the heterogeneous graph are the titles in the document; the text segment nodes in the heterogeneous graph are the text segments in the document; or, The title node in the heterogeneous graph is the target title in the document, and the level of the target title is higher than the target level; the text segment node in the heterogeneous graph includes the non-target titles and each text segment in the document; the level of the non-target title is lower than or equal to the target level.

9. The method according to claim 1, characterized in that The initial value of each node in the heterogeneous graph is determined as follows: Taking text units in the document as units, performing a second encoding on each word in each text unit to obtain a context feature representation of each word in each text unit in the text unit, which serves as an initial value of each byte in the heterogeneous graph; each text unit is a title or a text segment; For any title node, the context feature representations of each word in the title of the title node are fused to obtain the initial value of the title node; For any text segment node, the context feature representations of the title of the text segment node or each word in the text segment are fused to obtain the initial value of the text segment node.

10. A document element extraction device, characterized in that: include: An obtaining unit, configured to obtain layout structure information of the document, wherein the layout structure information includes at least paragraph division, title level, header, and footer; an encoding unit, configured to encode each word in the document according to the layout structure information; An extraction unit, configured to determine the element label to which each character belongs based on the encoding result of each character; The encoding unit is specifically used for: Determining titles and text segments from the document based on the layout structure information, and constructing a heterogeneous graph based on the characters, titles, and text segments in the document, wherein the nodes in the heterogeneous graph include character nodes, title nodes, and text segment nodes; and the edges in the heterogeneous graph include relationships between characters, relationships between characters and text segments, and relationships between text segments and titles; Performing graph convolution on the heterogeneous graph to obtain encoding results of each node; The encoding results of each word are merged with the encoding results of the corresponding title nodes to obtain the encoding results of each word.

11. A document element extraction device, characterized in that: including memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the document element extraction method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the document element extraction method according to any one of claims 1 to 9 is implemented.

Images