Information extraction model construction method, information extraction method and device

Abstract

Provided are a method for constructing an information extraction model, an information extraction method and device. The method for constructing an information extraction model comprises: obtaining a classification training sample set and classification annotation information; training a text box classification model for classifying text boxes based on the classification training sample set and the classification annotation information; obtaining an extraction training sample set and extraction annotation information; training a text extraction model for extracting text from text boxes based on the extraction training sample set and the extraction annotation information; and constructing an information extraction model for extracting information entities from images based on the trained text box classification model and the trained text extraction model. The method for constructing an information extraction model, the information extraction method and device according to the present disclosure solve the problems of long text being difficult to process and information being lost due to the destruction of the relevance between texts in the existing information extraction method, and can avoid the problem of excessively long text while learning the relationship between text boxes.

Description

Technical Field

[0001] This disclosure relates to the field of information processing technology, and specifically to a method for constructing an information extraction model, an information extraction method, and an apparatus. Background Technology

[0002] With the development of information technology, the ways in which information is disseminated are becoming increasingly diverse, and the amount of information to be disseminated is also growing. When processing rich text documents, in order to enable people to obtain information more quickly and intuitively, it is usually necessary to extract important text information from the document in the form of key-value pairs for easier reading and subsequent structured processing. Under this trend, it may be necessary to detect, recognize, and extract text information already generated in images.

[0003] In the process of information recognition and extraction, it is necessary to detect text boxes in the image, identify the text content in the text boxes, and then perform structured processing on the text in order to extract information entities.

[0004] However, existing information extraction methods require concatenating all the identified text before inputting it into the information extraction model. In one case, when the concatenated text is long, the model may be unable to process the excessively long text and thus fail to extract information. In another case, the concatenated text can be forcibly divided into multiple segments, and information can be extracted from each segment separately. However, since the information of each segment is independent of the others, the connection between the texts may be destroyed, leading to information loss. Summary of the Invention

[0005] The exemplary embodiments disclosed herein may at least solve the above-described problems, or may not solve the above-described problems.

[0006] According to a first aspect of this disclosure, a method for constructing an information extraction model is provided. The method includes: acquiring a classification training sample set and classification annotation information, wherein each classification training sample in the classification training sample set includes the position of a classification training text box in an image and the text within the classification training text box, and the classification annotation information indicates which category of a preset text box category the classification training text box belongs to; training a text box classification model for classifying text boxes based on the classification training sample set and the classification annotation information; acquiring an extraction training sample set and extraction annotation information, wherein each extraction training sample in the extraction training sample set is an extraction training text sequence including one or more characters, and the extraction annotation information indicates the position of each character in the extraction training text sequence within a preset reference position of an information entity; training a text extraction model for extracting text from text boxes based on the extraction training sample set and the extraction annotation information; and constructing an information extraction model for extracting information entities from an image based on the trained text box classification model and the trained text extraction model.

[0007] Optionally, the step of obtaining the classification labeling information includes: obtaining the positions of multiple pre-labeled text boxes in the image and the categories of the pre-labeled text boxes; determining the area overlap between the classification training text box and each of the multiple pre-labeled text boxes based on the positions of the classification training text box and the multiple pre-labeled text boxes, and labeling the classification training text box based on the area overlap to obtain the classification labeling information, wherein the area overlap refers to the ratio of the overlapping area of ​​the classification training text box and the pre-labeled text box to the area of ​​the pre-labeled text box; when the area overlap between the classification training text box and the pre-labeled text box is greater than a preset threshold, the classification training text box is labeled as the same category as the corresponding pre-labeled text box.

[0008] Optionally, the step of obtaining the classification labeling information further includes: when the area overlap between the classification training text box and each of the plurality of pre-labeled text boxes is less than or equal to the preset threshold, labeling the classification training text box as a category different from the categories of the plurality of pre-labeled text boxes.

[0009] Optionally, the step of training a text box classification model for classifying text boxes based on the classification training sample set and the classification annotation information includes: dividing the classification training text box of each classification training sample into at least one classification training sub-text box; determining the position information of each classification training sub-text box based on the position of the classification training text box of each classification training sample, wherein the position information includes the position of at least one pixel of the classification training sub-text box in the image and the height and width of the classification training sub-text box or the positions of the pixels at the two vertices on the diagonal of the classification training sub-text box in the image; determining the text identifier vector corresponding to each classification training sub-text box based on the text of the classification training text box of each classification training sample, wherein the text identifier vector represents the semantics represented by the text in the classification training sub-text box; and training a text box classification model for classifying text boxes based on the position information of the classification training sub-text boxes, the text identifier vector, and the classification annotation information.

[0010] Optionally, the text box classification model includes a first neural network and a second neural network. The step of training the text box classification model for classifying text boxes based on the position information of the training sub-text boxes, text identifier vectors, and the classification annotation information includes: inputting the position information and text identifier vectors of the training sub-text boxes into the first neural network to obtain sample identifier vectors output from the first neural network that correspond one-to-one with the training text boxes; arranging the sample identifier vectors corresponding to the training text boxes according to their position order in the image to obtain a sample identifier vector sequence; inputting the sample identifier vector sequence into the second neural network to obtain category prediction results output from the second neural network that correspond one-to-one with the training text boxes; and adjusting the parameters of the first neural network and the second neural network based on the category prediction results and the classification annotation information to obtain the trained text box classification model.

[0011] Optionally, the step of obtaining the extraction annotation information includes: obtaining a pre-annotated text sequence and text annotation information corresponding to each extraction training text sequence, wherein the text annotation information represents the position of each text in the pre-annotated text sequence in the information entity; determining the minimum edit distance of the extraction training text sequence to the corresponding pre-annotated text sequence; and annotating each text in the extraction training text sequence according to the minimum edit distance and the text annotation information, according to preset annotation conditions, wherein the preset annotation conditions include: when the text in the extraction training text sequence is the same as the text in the corresponding pre-annotated text sequence at the minimum edit distance, the text in the extraction training text sequence is annotated to the same position as the corresponding text in the corresponding pre-annotated text sequence.

[0012] Optionally, the preset annotation conditions further include: when the text in the extracted training text sequence is different from the text in the corresponding pre-annotated text sequence at the minimum edit distance, annotation is performed according to the operation that achieves the minimum edit distance, wherein when the operation that achieves the minimum edit distance is a replacement operation, the text in the extracted training text sequence is annotated at the same position as the corresponding text in the corresponding pre-annotated text sequence; when the operation that achieves the minimum edit distance is a deletion operation or an addition operation, the text in the extracted training text sequence is annotated at a position that is different from the position of each text in the corresponding pre-annotated text sequence.

[0013] Optionally, the text extraction model includes a third neural network, a fourth neural network, and a fifth neural network. The step of training the text extraction model for extracting text from a text box based on the extraction training sample set and the extraction annotation information includes: inputting the extraction training sample set into the third neural network to obtain sample identifier vectors output from the third neural network that correspond one-to-one with the extraction training samples; inputting the sample identifier vectors into the fourth neural network to obtain a first position prediction probability of each character in the extraction training text sequence appearing at each preset reference position, output from the fourth neural network; inputting the first position prediction probability of each character in the extraction training text sequence into the fifth neural network to obtain a second position prediction probability of each character in the extraction training text sequence appearing at each preset reference position, output from the fifth neural network, wherein the second position prediction probability is determined based on the first position prediction probability between adjacent characters in the extraction training text sequence; and training the third neural network, the fourth neural network, and the fifth neural network based on the second position prediction probability and the extraction annotation information to obtain a trained text extraction model.

[0014] Optionally, the preset reference position includes the position of the first character of the information entity and the positions of the entity characters other than the first character position. The extraction training text sequence is obtained by concatenating multiple extraction training text boxes. The step of obtaining the pre-labeled text sequence and text annotation information corresponding to each extraction training text sequence includes: combining the characters contained in each preset information entity in the extraction training text sequence into a pre-labeled text sequence; determining the extraction training text box corresponding to the pre-labeled text sequence among the multiple extraction training text boxes, wherein when there is more than one extraction training text box corresponding to the pre-labeled text sequence, the first character in the pre-labeled text sequence is labeled as the first character position, and all characters in the pre-labeled text sequence other than the first character are labeled as the entity character positions.

[0015] Optionally, the step of constructing an information extraction model for extracting information entities from an image based on a trained text box classification model and a trained text extraction model includes: combining the text in the text boxes classified into the same category into a text sequence according to the text box classification output by the trained text box classification model; inputting the text sequence corresponding to each category into the trained text extraction model to obtain the text position of each text in the text sequence output by the trained text extraction model in the information entity; and extracting the information entity under each category from the image according to the predicted text position corresponding to each category, so as to construct the information extraction model.

[0016] According to a second aspect of this disclosure, an information extraction method is provided, the information extraction method comprising: acquiring image data of an image about which information to be extracted is to be acquired, wherein the image data includes the positions of text boxes in the image and the text in each text box; and extracting information from the image using an information extraction model based on the image data, wherein the information extraction model is constructed according to the information extraction model construction method described in this disclosure.

[0017] According to a third aspect of this disclosure, an information extraction method is provided, the information extraction method comprising: acquiring image data of an image from which information to be extracted, wherein the image data includes the positions of text boxes in the image and the text in each text box; classifying the text boxes according to a preset text box category based on the positions of the text boxes and the text in the text boxes, wherein the preset text box category represents the category of the information entity represented by the text in the text box; combining the text in the text boxes classified into the same category into a text sequence; for each category, predicting the text position of each text in the text sequence in the information entity corresponding to the corresponding category based on the text sequence; and extracting the information entity under each category from the image data according to the predicted text position, for use in determining the information in the image based on the information entity.

[0018] According to a fourth aspect of this disclosure, an apparatus for constructing an information extraction model is provided. The apparatus includes: a first acquisition unit configured to acquire a classification training sample set and classification labeling information, wherein each classification training sample in the classification training sample set includes the position of a classification training text box in an image and the text within the classification training text box, and the classification labeling information indicates which category of a preset text box category the classification training text box of the classification training sample belongs to; a first training unit configured to train a text box classification model for classifying text boxes based on the classification training sample set and the classification labeling information; and a second... The acquisition unit is configured to acquire an extraction training sample set and extraction annotation information, wherein each extraction training sample in the extraction training sample set is an extraction training text sequence including one or more characters, and the extraction annotation information indicates the position of each character in the extraction training text sequence within a preset reference position of the information entity; the second training unit is configured to train a text extraction model for extracting text from text boxes based on the extraction training sample set and the extraction annotation information; and the construction unit is configured to construct an information extraction model for extracting information entities from an image based on the trained text box classification model and the trained text extraction model.

[0019] Optionally, the first acquisition unit is further configured to: acquire the positions of multiple pre-labeled text boxes in the image and the categories of the pre-labeled text boxes; determine the area overlap between the classification training text box and each of the multiple pre-labeled text boxes based on the positions of the classification training text box and the multiple pre-labeled text boxes, and label the classification training text box based on the area overlap to obtain the classification labeling information, wherein the area overlap refers to the ratio of the overlapping area of ​​the classification training text box and the pre-labeled text box to the area of ​​the pre-labeled text box, and when the area overlap between the classification training text box and the pre-labeled text box is greater than a preset threshold, the classification training text box is labeled as the same category as the corresponding pre-labeled text box.

[0020] Optionally, the first acquisition unit is further configured to: when the area overlap between the classification training text box and each of the plurality of pre-labeled text boxes is less than or equal to the preset threshold, label the classification training text box as a category different from the categories of the plurality of pre-labeled text boxes.

[0021] Optionally, the first training unit is further configured to: divide the classification training text box of each classification training sample into at least one classification training sub-text box; determine the position information of each classification training sub-text box based on the position of the classification training text box of each classification training sample, wherein the position information includes the position of at least one pixel of the classification training sub-text box in the image and the height and width of the classification training sub-text box or the positions of the pixels at the two vertices of the classification training sub-text box on the diagonal in the image; determine the text identifier vector corresponding to each classification training sub-text box based on the text of the classification training text box of each classification training sample, wherein the text identifier vector represents the semantics represented by the text in the classification training sub-text box; and train a text box classification model for classifying text boxes based on the position information of the classification training sub-text boxes, the text identifier vector, and the classification labeling information.

[0022] Optionally, the text box classification model includes a first neural network and a second neural network, wherein the first training unit is further configured to: input the position information and text label vector of the classification training sub-text box into the first neural network to obtain sample label vectors output from the first neural network that correspond one-to-one with the classification training text box; arrange the sample label vectors corresponding to the classification training text box according to the position order of the classification training text box in the image to obtain a sample label vector sequence; input the sample label vector sequence into the second neural network to obtain the category prediction result output from the second neural network that corresponds one-to-one with the classification training text box; and adjust the parameters of the first neural network and the second neural network based on the category prediction result and the classification label information to obtain a trained text box classification model.

[0023] Optionally, the second acquisition unit is further configured to: acquire a pre-annotated text sequence and text annotation information corresponding to each extracted training text sequence, wherein the text annotation information represents the position of each text in the pre-annotated text sequence in the information entity; determine the minimum edit distance of the extracted training text sequence to the corresponding pre-annotated text sequence; and, based on the minimum edit distance and the text annotation information, annotate each text in the extracted training text sequence according to preset annotation conditions, wherein the preset annotation conditions include: when the text in the extracted training text sequence is the same as the text in the corresponding pre-annotated text sequence at the minimum edit distance, annotate the text in the extracted training text sequence to the same position as the corresponding text in the corresponding pre-annotated text sequence.

[0024] Optionally, the preset annotation conditions further include: when the text in the extracted training text sequence is different from the text in the corresponding pre-annotated text sequence at the minimum edit distance, annotation is performed according to the operation that achieves the minimum edit distance, wherein when the operation that achieves the minimum edit distance is a replacement operation, the text in the extracted training text sequence is annotated at the same position as the corresponding text in the corresponding pre-annotated text sequence; when the operation that achieves the minimum edit distance is a deletion operation or an addition operation, the text in the extracted training text sequence is annotated at a position that is different from the position of each text in the corresponding pre-annotated text sequence.

[0025] Optionally, the text extraction model includes a third neural network, a fourth neural network, and a fifth neural network. The second training unit is further configured to: input the extraction training sample set into the third neural network to obtain sample identifier vectors output from the third neural network that correspond one-to-one with the extraction training samples; input the sample identifier vectors into the fourth neural network to obtain a first position prediction probability of each character in the extraction training text sequence appearing at each preset reference position, output from the fourth neural network; input the first position prediction probability of each character in the extraction training text sequence into the fifth neural network to obtain a second position prediction probability of each character in the extraction training text sequence appearing at each preset reference position, output from the fifth neural network, wherein the second position prediction probability is determined based on the first position prediction probability between adjacent characters in the extraction training text sequence; and train the third neural network, the fourth neural network, and the fifth neural network based on the second position prediction probability and the extraction annotation information to obtain a trained text extraction model.

[0026] Optionally, the preset reference position includes the first character position of the information entity and the positions of the entity characters other than the first character position. The extracted training text sequence is obtained by concatenating multiple extracted training text boxes. The second acquisition unit is further configured to: combine the text contained in each preset information entity in the extracted training text sequence into a pre-labeled text sequence; determine the extracted training text box corresponding to the pre-labeled text sequence among the multiple extracted training text boxes, wherein when there is more than one extracted training text box corresponding to the pre-labeled text sequence, the first character in the pre-labeled text sequence is labeled as the first character position, and all characters in the pre-labeled text sequence other than the first character are labeled as the entity character positions.

[0027] Optionally, the building unit is further configured to: combine the text in text boxes classified into the same category into a text sequence according to the text box classification output from the trained text box classification model; input the text sequence corresponding to each category into the trained text extraction model to obtain the text position of each character in the text sequence output from the trained text extraction model in the information entity; and extract the information entity under each category from the image according to the predicted text position corresponding to each category, so as to construct the information extraction model.

[0028] According to a fifth aspect of this disclosure, an information extraction apparatus is provided, the information extraction apparatus comprising: an image acquisition unit configured to acquire image data of an image about which information to be extracted is to be acquired, wherein the image data includes the positions of text boxes in the image and the text in each text box; and an information extraction unit configured to extract information from the image based on the image data using an information extraction model, wherein the information extraction model is constructed according to the information extraction model construction method described in this disclosure.

[0029] According to a sixth aspect of this disclosure, an information extraction apparatus is provided, comprising: an acquisition unit configured to acquire image data of an image from which information to be extracted, wherein the image data includes the positions of text boxes in the image and the text in each text box; a classification unit configured to classify the text boxes according to a preset text box category based on the positions of the text boxes and the text in the text boxes, wherein the preset text box category represents the category of the information entity represented by the text in the text box; a combination unit configured to combine the text in text boxes classified into the same category into a text sequence; a prediction unit configured to predict, for each category, the text sequence, the text position of each text in the text sequence in the information entity corresponding to the corresponding category; and an extraction unit configured to extract the information entity under each category from the image data according to the predicted text position, for determining the information in the image based on the information entity.

[0030] According to a seventh aspect of this disclosure, an electronic device is provided, the electronic device comprising: a processor; and a memory for storing processor-executable instructions, wherein, when executed by the processor, the processor-executable instructions cause the processor to perform a method for constructing an information extraction model according to this disclosure or an information extraction method according to a second or third aspect of this disclosure.

[0031] According to an eighth aspect of this disclosure, a computer-readable storage medium is provided that stores instructions which, when executed by at least one computing device, cause the at least one computing device to perform a method for constructing an information extraction model according to this disclosure or an information extraction method according to a second or third aspect of this disclosure.

[0032] According to a ninth aspect of this disclosure, a system is provided that includes at least one computing device and at least one storage device for storing instructions, wherein when the instructions are executed by the at least one computing device, the at least one computing device causes the at least one computing device to perform a method for constructing an information extraction model according to this disclosure or an information extraction method according to a second or third aspect of this disclosure.

[0033] The information extraction method and apparatus according to the exemplary embodiments of the present disclosure can perform text box classification and text extraction processing on the text data to be extracted. It solves the problems of existing information extraction methods, such as the difficulty in processing excessively long texts and the loss of information due to the destruction of the correlation between texts. It can avoid the problem of excessively long texts and learn the relationship between text boxes. Attached Figure Description

[0034] These and / or other aspects and advantages of this disclosure will become clearer and more readily understood from the following description of the embodiments, taken in conjunction with the accompanying drawings, wherein:

[0035] Figure 1 This is a flowchart illustrating a method for constructing an information extraction model according to an exemplary embodiment of the present disclosure.

[0036] Figure 2 This is a schematic diagram illustrating the area overlap described in the method for constructing an information extraction model according to an exemplary embodiment of the present disclosure.

[0037] Figure 3 This is a flowchart illustrating the steps of training a text box classification model in a method for constructing an information extraction model according to an exemplary embodiment of the present disclosure.

[0038] Figure 4 This is a schematic diagram illustrating the structure of a text box classification model in a method for constructing an information extraction model according to an exemplary embodiment of the present disclosure.

[0039] Figure 5 This is a flowchart illustrating the steps of training a text extraction model in a method for constructing an information extraction model according to an exemplary embodiment of the present disclosure.

[0040] Figure 6 This is a schematic diagram illustrating the structure of a text extraction model in a method for constructing an information extraction model according to an exemplary embodiment of the present disclosure.

[0041] Figure 7 This is a flowchart illustrating the steps of constructing an information extraction model based on a text box classification model and a text extraction model in a method for constructing an information extraction model according to an exemplary embodiment of the present disclosure.

[0042] Figure 8 This is a flowchart illustrating an information extraction method according to an exemplary embodiment of the present disclosure.

[0043] Figure 9 This is a schematic block diagram illustrating an apparatus for constructing an information extraction model according to an exemplary embodiment of the present disclosure.

[0044] Figure 10This is a schematic block diagram illustrating an example of an information extraction apparatus according to an exemplary embodiment of the present disclosure.

[0045] Figure 11 This is a schematic block diagram illustrating another example of an information extraction apparatus according to exemplary embodiments of the present disclosure. Detailed Implementation

[0046] The following description, provided with reference to the accompanying drawings, is intended to aid in a full understanding of embodiments of the present disclosure as defined by the claims and their equivalents. Various specific details are included to aid understanding, but these details are to be considered exemplary only. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Furthermore, for clarity and brevity, descriptions of well-known functions and structures are omitted.

[0047] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. As another example, "performing at least one of step one and step two" indicates the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0048] The following description, with reference to the accompanying drawings, describes a method for constructing an information extraction model, an information extraction method, an apparatus for constructing an information extraction model, an information extraction apparatus, a computer-readable storage medium, and a system including at least one computing device and at least one storage instruction, according to exemplary embodiments of the present disclosure.

[0049] It is worth noting that prior to this disclosure, some information extraction methods required concatenating all the identified text and inputting it into the information extraction model. This made it impossible for the model to process excessively long texts and thus fail to extract information. Furthermore, this method did not utilize the positional information of the text within the document containing the text information identified from the image, resulting in a lack of correlation between the texts.

[0050] To address this issue, the LayoutLM model was proposed. It considers text, layout, and image information; however, it concatenates the entire document's text as input. When the input is too long, it segments the long text into multiple segments. However, since each segment cannot be correlated with information from other segments, this causes serious information loss. Furthermore, when dealing with a single value distributed across multiple text boxes, the LayoutLM model splits a consecutive value into multiple independent values, leading to information extraction errors.

[0051] In addition, there are some information extraction methods that can classify text boxes, but they neglect to further extract key information from the text within the text boxes to obtain accurate information for each category.

[0052] In view of the above problems, in a first aspect, exemplary embodiments of the present disclosure propose a method for constructing an information extraction model to at least address one of the problems in the related art.

[0053] Figure 1 This is a flowchart illustrating a method for constructing an information extraction model according to an exemplary embodiment of the present disclosure.

[0054] like Figure 1 As shown, the method for constructing an information extraction model may include the following steps:

[0055] In step S10, the classification training sample set and classification labeling information can be obtained.

[0056] In this step, each classification training sample in the classification training sample set may include the position of the classification training text box in the image and the text information in the classification training text box. The classification label information can indicate which category of the preset text box category the classification training text box of the classification training sample belongs to.

[0057] Here, multiple images can be acquired, each image including one or more text boxes. The position of each text box in the corresponding image and the text information within the text box are determined. As an example, the position of the text box and the text information within the text box can be obtained using existing text detection and text recognition models. This disclosure does not specifically limit the process of obtaining the position of the text box and the text information within the text box. Here, the position information of the text box can be used to locate the position of the text box within the image and to determine the size of the text box. For example, the position of at least one pixel of the text box in the image, as well as the height and width of the text box, can be determined. Thus, since the position of the at least one pixel in the text box is determined, the position of the text box within the image can be located if the position of the at least one pixel in the image is known. Alternatively, the positions of the pixels at the two vertices on the diagonal of the classification training sub-text box in the image can be known, thereby determining the height and width of the text box based on the positions of the pixels at the two vertices in the image.

[0058] The preset text box categories can be pre-defined according to the application scenario of information extraction. For example, they can be text elements in HTML, such as title, address, body, etc. Here, each text box can belong to one or more categories.

[0059] As an example, the steps to obtain classification labeling information may include: obtaining the positions and categories of multiple pre-labeled text boxes in the image; determining the area overlap between the classification training text box and each of the multiple pre-labeled text boxes based on the positions of the classification training text box and the multiple pre-labeled text boxes, and labeling the classification training text box based on the area overlap to obtain classification labeling information.

[0060] In this step, text boxes in the image can be pre-labeled, and their positions and categories within the image can be recorded. These pre-labeled text boxes can be obtained by selecting content of interest for information extraction, and their categories can be preset categories based on the application scenario of information extraction, as described above. Thus, the category of the training text boxes can be determined by comparing the area overlap between the training text boxes and the pre-labeled text boxes.

[0061] Here, area refers to the pixel area occupied, and area overlap can be defined as the ratio of the overlapping area of ​​the training text box and the pre-labeled text box to the area of ​​the pre-labeled text box. The greater the area overlap, the higher the overlap between the training text box and the pre-labeled text box, and the closer their text content is. When their text content is relatively similar, the training text box can be labeled based on the known category of the pre-labeled text box.

[0062] Specifically, when the overlap between the area of ​​the training text box and the pre-labeled text box exceeds a preset threshold, the training text box is labeled with the same category as the corresponding pre-labeled text box. The preset threshold can be set according to actual needs, for example, it can be 0.5.

[0063] When the area overlap between the training text box and each of the multiple pre-labeled text boxes is less than or equal to a preset threshold, the training text box is labeled as a category different from the categories of all the pre-labeled text boxes. Here, since the pre-labeled text boxes can be obtained by selecting content of interest for information extraction, when the area overlap is less than or equal to the preset threshold, the training text box may not belong to the content of interest for information extraction; for example, it may be background text in an image. Therefore, it does not need to be extracted and can be labeled as a category different from the categories of all the pre-labeled text boxes, such as "background".

[0064] Here, a classification training text box can have an area overlap greater than a preset threshold with one or more of the pre-labeled text boxes. That is, a classification training text box can match one or more pre-labeled text boxes. Therefore, a classification training text box can correspond to one or more categories.

[0065] Figure 2 An exemplary schematic diagram of area overlap is shown. For example... Figure 2 As shown, the solid-lined boxes represent the obtained classification training text boxes, and the dashed-lined boxes represent the pre-labeled text boxes. Given that the area of ​​the pre-labeled text box labbox_j is area_j, we can calculate the overlap area area area_iou between the classification training text box detbox_i and the pre-labeled text box labbox_j. We can also set a threshold thre. When (area_iou) / (area_j)>thre, we can consider the classification training text box detbox_i to match the pre-labeled text box labbox_j. We can then label the classification training text box detbox_i with the category corresponding to the pre-labeled text box labbox_j.

[0066] In step S20, a text box classification model for classifying text boxes can be trained based on the classification training sample set and classification labeling information.

[0067] In this step, the classification training sample set can also be preprocessed to train the text box classification model based on the preprocessed classification training sample set.

[0068] Specifically, step S20 may include the following steps:

[0069] In step S21, the classification training text box of each classification training sample can be divided into at least one classification training sub-text box.

[0070] For example, existing word segmentation models can be used to segment the text within each text box into one or more tokens, and then the text box can be divided into sub-text boxes corresponding to each token, with equal widths. Here, a token refers to a single character in Chinese and a sub-word (part of a word) in English.

[0071] In step S22, the position information of each classification training sub-text box can be determined based on the position of the classification training text box of each classification training sample.

[0072] As mentioned above, since the position of the classification training text box in the image is determined, the position of the classification training sub-text boxes obtained by dividing the classification training text box in the image is also determined. Specifically, the position information of the classification training sub-text boxes may include the position of at least one pixel of the classification training sub-text box in the image, as well as the height and width of the classification training sub-text box, or the positions of the pixels located at the two vertices on the diagonal of the classification training sub-text box in the image.

[0073] For example, the position information of the sub-text boxes in the classification training can include the pixel coordinates (x0, y0) of the top-left corner, the pixel coordinates (x1, y1) of the bottom-right corner, the width w of the sub-text box, and the height h of the sub-text box. Optionally, for ease of calculation, the values ​​of the above position parameters of the classification training sub-text boxes can be normalized to the range of 0 to 1000. Specifically, the position coordinates of at least two pixels of the classification training sub-text box, the height of the sub-text box, and the width of the sub-text box can all be represented in the form of an embedding vector.

[0074] In step S23, based on the text information of the classification training text box of each classification training sample, the text identifier vector corresponding to each classification training sub-text box can be determined, wherein the text identifier vector represents the semantics represented by the text in the classification training sub-text box.

[0075] In this step, semantic analysis can be performed on the text in each classification training sub-text box according to preset semantic rules, and the semantics of the text can be converted into vector form for representation. For example, the vector value corresponding to each classification training sub-text box can be taken from the learnable weight matrix to form an embedding vector as the text identifier vector, and the embedding vectors of all classification training sub-text boxes have the same dimension.

[0076] In step S24, a text box classification model for classifying text boxes can be trained based on the position information of the sub-text boxes, the text identifier vector, and the classification annotation information.

[0077] Figure 3 This is a flowchart illustrating the steps of training a text box classification model according to an exemplary embodiment of the present disclosure. Figure 4 This is a schematic diagram illustrating the structure of a text box classification model according to an exemplary embodiment of the present disclosure.

[0078] As an example, a text box classification model can include a first neural network and a second neural network. For example... Figure 3 As shown, step S24 may include the following steps:

[0079] In step S241, the position information and text identifier vector of the classification training sub-text box can be input into the first neural network to obtain the sample identifier vector output from the first neural network that corresponds one-to-one with the classification training text box.

[0080] As an example, such as Figure 4 As shown, the position coordinate vectors of at least two pixels of each category training sub-text box, the height vector, the width vector of the sub-text box, and the text label vector can be added together and input into the first neural network. Here, the pixel position coordinate vectors, the height vector, the width vector of the sub-text box, and the text label vector have the same vector dimension. Based on the position information and text label vectors of the category training sub-text boxes divided by each category training text box, the first neural network can learn the semantic information expressed by each category training text box as a whole, and output this semantic information in the form of a label vector according to predetermined semantic rules. For example, the first neural network can output the embedding_segment vector corresponding to each category training text box. Here, the first neural network can be a BERT model, but it is not limited to this; it can also be other machine learning models used for semantic learning.

[0081] Here, the information extraction method according to the exemplary embodiment of this disclosure considers the position of each text box during the training of the text box classification model. This enables the model to learn the varied text box layout information, making the method applicable to image text extraction with different layout methods. It has strong versatility and wide application scenarios.

[0082] In step S242, the sample identifier vectors corresponding to the classification training text boxes can be arranged according to the position order of the training text boxes in the image to obtain a sample identifier vector sequence.

[0083] As an example, the sample label vectors of all the classification training text boxes can be concatenated into a sample label vector sequence according to the arrangement of the text boxes in the image from left to right and from top to bottom. Here, the arrangement order can also be changed according to actual needs, such as from right to left and from bottom to top.

[0084] In step S243, the sample identifier vector sequence can be input into the second neural network to obtain the category prediction results output from the second neural network that correspond one-to-one with the classification training samples.

[0085] In this step, the second neural network can learn the semantic association information between the training text boxes based on the sample identifier vector sequence, and classify the training text boxes according to this semantic association information. For example, the second neural network can output the category corresponding to each training text box. Here, since each training text box may belong to multiple categories, the second neural network can output the probability of each training text box belonging to each category as the category prediction result.

[0086] As an example, the second neural network may include a Transformer Encoder model, and a sigmoid layer may be added after the Transformer Encoder model to obtain the class probabilities of the training text boxes for classification, but this disclosure is not limited thereto, and the second neural network may also be other machine learning models for semantic learning.

[0087] In step S244, the parameters of the first neural network and the second neural network can be adjusted based on the category prediction results and classification labeling information to obtain a trained text box classification model.

[0088] In this step, the category prediction results of the classification training text boxes can be compared with the labeled categories of the classification training text boxes in the classification annotation information, so as to adjust the parameters of the first neural network and the second neural network to obtain the final text box classification model.

[0089] According to exemplary embodiments of this disclosure, such as Figure 4As shown, a text box classification model can include a first neural network (e.g., the BERT model) that performs semantic learning on each training text box for each category, and a second neural network (e.g., the Transformer Encoder) that learns the semantic relationships between training text boxes of different categories. Based on this combined neural network structure, the input of the entire model does not need to be concatenated into a long string of text as in existing methods, which greatly reduces the length of the input text and lowers the computational requirements of a single neural network.

[0090] In summary, for the text box classification model constructed using the above method, the position and text information of each text box can be first input into a first neural network to extract information and output a sample identifier vector representing each text box. Then, the sample identifier vectors of all text boxes are input into a second neural network in an order such as left to right or top to bottom. The second neural network can learn the correlation between text boxes using a self-attention mechanism. Furthermore, a sigmoid layer can be added to the second neural network to obtain the class probability of each text box. The text box classification model trained in this way avoids the problem of excessively long text sequences resulting from concatenating the entire document's text, while also learning the relationships between text boxes and inferring the class of each text box using global information.

[0091] In step S30, the training sample set and annotation information can be obtained.

[0092] Here, each extracted training sample in the training sample set can be an extracted training text sequence, and the extracted annotation information can indicate the position of each text in the extracted training text sequence within the preset reference position of the information entity.

[0093] Specifically, a training sample set can be formed by obtaining multiple training text sequences, where the text in each training text sequence is defined. As an example, the text in the training text sequence can be obtained using existing text recognition models, and this disclosure does not impose any particular limitations on the process of obtaining the text in the training text sequence.

[0094] An information entity can refer to a collection of specific things, which can be things that exist in the objective world. In natural language, they usually appear in the form of nouns. The preset reference positions of information entities can be pre-set according to the application scenario of information extraction. For example, the preset reference positions can include the position of the first character of an information entity and the positions of other characters. The characters at each preset reference position can be determined from the text sequence to extract the information entity.

[0095] As an example, the steps to obtain the extracted annotation information may include the following operations:

[0096] In step S31, a pre-annotated text sequence and text annotation information corresponding to each extracted training text sequence can be obtained. Here, the text annotation information indicates the position of each text in the pre-annotated text sequence in the information entity.

[0097] In this step, one or more characters in the extracted training text sequence can be pre-annotated to obtain a pre-annotated text sequence, and the position of each character in the pre-annotated text sequence within the information entity can be recorded. Here, the pre-annotated text can be a portion of the extracted training text sequence, which can be obtained by selecting content of interest in information extraction. The position of the pre-annotated text within the information entity can be marked according to the preset reference position of the information entity pre-set based on the application scenario of information extraction, as described above. In this way, the characters in the extracted training text sequence can be annotated by comparing the characters in the extracted training text sequence and the characters in the pre-annotated text sequence.

[0098] As an example, the training text sequence can be obtained by concatenating multiple training text boxes. The aforementioned preset reference positions can include the position of the first character of the information entity and the positions of the entity characters other than the position of the first character.

[0099] In this example, the steps of obtaining the pre-labeled text sequence and text annotation information corresponding to each extracted training text sequence include: combining the text contained in each preset information entity in the extracted training text sequence into a pre-labeled text sequence; determining the extracted training text boxes corresponding to the pre-labeled text sequence among multiple extracted training text boxes, wherein, when there is more than one extracted training text box corresponding to the pre-labeled text sequence, the first character in the pre-labeled text sequence is labeled as the first character position, and all characters in the pre-labeled text sequence except the first character are labeled as entity character positions.

[0100] Specifically, taking the traditional method of annotating the position of words encoded as "BIO" as an example, words that do not belong to entities (e.g., tokens) can be annotated as "O", and words belonging to entities can be annotated as "B" or "I" according to whether they are the first letters of the entities. However, in the method of the present disclosure, different from the traditional named entity recognition (NER) problem, since the text box classification model can include a group structure of a first neural network and a second neural network, the input of the entire model does not require all texts to be concatenated into a long text string, but is divided into multiple text box inputs. This makes it possible that an entity may be discontinuously distributed in multiple text boxes. If annotated according to the traditional method of annotating the position of words, an entity may be split into multiple entities. Therefore, in the method of annotating the position of words in the present disclosure, the encoding is changed from the text box granularity to the entity granularity. Specifically, when an entity is discontinuously distributed in multiple text boxes, except for the first text box that retains the first character position of the entity, e.g., annotated as "B", the labels of the characters belonging to the entity in the subsequent text boxes are changed to the entity character positions, e.g., annotated as "I", to avoid splitting an entity into multiple entities.

[0101] For example, the extracted training text sequence can be "Ehan Medical Equipment No. 007", which can be obtained by concatenating two extracted training text boxes [Ehan Medical Equipment] and [No. 007]. Among them, the text to be annotated included in the preset information entity can be "Ehan Medical Equipment No. 007", that is, the pre-annotated text sequence can be "Ehan Medical Equipment No. 007". According to the traditional method of annotating the position of words encoded as "BIO", which does not distinguish the source of the text to be annotated, therefore, the pre-annotated text sequence can be annotated as "BIIIBIII".

[0102] According to the present disclosure, since there is more than one extracted training text box corresponding to the text to be annotated, the first character "E" of the text to be annotated in the extracted training text sequence can be annotated as the first character position (e.g., "B"), and all characters except the first character in the text to be annotated, i.e., "Han Medical Equipment No. 007", can be annotated as the entity character positions (e.g., "I"). Therefore, its final annotation encoding is: "BIIIIIII".

[0103] In step S32, the minimum edit distance for transforming the extracted training text sequence into the corresponding pre-annotated text sequence can be determined.

[0104] In this step, the minimum edit distance (also known as the "shortest edit path") algorithm can be used to determine the minimum edit distance for each character in the extracted training character sequence to be transformed into the corresponding character in the pre-annotated character sequence. The operations for achieving the minimum edit distance include delete, add, and replace. For example, the extracted training character sequence can be "Tuhai City Pudong New Area" (where "Shang" is misrecognized as "Tu"), and its corresponding pre-annotated character sequence can be "Shanghai Pudong". Among them, the characters "Tu", "Hai", "Shi", "Pu", "Dong", "Xin", "Qu" are transformed into "Shanghai Pudong" through the edit processes of replace, no operation, delete, no operation, no operation, delete, delete respectively.

[0105] In step S33, based on the minimum edit distance and character annotation information, each character in the extracted training character sequence can be annotated according to the preset annotation conditions.

[0106] In this step, the preset annotation conditions may include: when the character in the extracted training character sequence is the same as the corresponding character in the pre-annotated character sequence under the minimum edit distance, the character in the extracted training character sequence is annotated as the same position as the corresponding character in the pre-annotated character sequence.

[0107] For example, in the above example, the characters "Hai", "Pu", "Dong" in the extracted training character sequence do not require any editing operations and directly correspond to the characters "Hai Pu Dong" in the pre-annotated character sequence. Therefore, the characters "Hai", "Pu", "Dong" in the extracted training character sequence can be annotated with the positions of each character in the pre-annotated character sequence "Hai Pu Dong".

[0108] When the character in the extracted training character sequence is different from the corresponding character in the pre-annotated character sequence under the minimum edit distance, the annotation can be performed according to the operations for achieving the minimum edit distance.

[0109] Specifically, when the operation for achieving the minimum edit distance is a replace operation, the character in the extracted training character sequence is annotated as the same position as the corresponding character in the pre-annotated character sequence; when the operation for achieving the minimum edit distance is a delete operation or an add operation, the character in the extracted training character sequence is annotated as a position different from each character in the pre-annotated character sequence.

[0110] For example, in the above example, the character "Tu" in the extracted training character sequence is transformed into the character "Shang" in the pre-annotated character sequence through a replace operation. Therefore, the character "Tu" in the extracted training character sequence can be annotated as the same position as the corresponding character "Shang" in the pre-annotated character sequence.

[0111] The characters "city", "new", and "district" in the extracted training text sequence are transformed through deletion operations respectively. This indicates that the characters "city", "new", and "district" have nothing to do with the pre-annotated sequence of interest. Therefore, the characters "city", "new", and "district" in the extracted training text sequence can be labeled at positions that are different from each position of the corresponding characters in the pre-annotated text sequence, for example, labeled as irrelevant positions.

[0112] For text annotation, the optical character recognition (OCR) network may have text recognition errors, resulting in the inability to match the extracted training text sequence and the pre-annotated text sequence one by one, making it difficult to label the text positions. Specifically, the traditional text fuzzy matching method is the sliding window method. Assume the pre-annotated text sequence is label_text and the extracted training text sequence is ocr_text. A window with the same length as label_text is defined and slides character by character on ocr_text. The edit distance between the text in the window and label_text is calculated, and the window with the minimum edit distance is selected as the target window. The text in the target window is labeled as an entity. Although this method can alleviate the difficulties caused by text recognition errors to a certain extent, problems will occur when the pre-annotated text sequence is composed of multiple scattered characters / character segments in the extracted training text sequence.

[0113] Taking the above example, the pre-annotated text sequence is "Shanghai Pudong", and the extracted training text sequence is "Tushanghai Pudong New Area" (the character "上" is recognized as "土"). The size of the window is 4, and the edit distances are the smallest when the target windows are "Shanghai Pudong" and "Tushanghai Pudong". However, the text in such target windows does not contain all of the pre-annotated text sequence, and the annotation results are "BIIIOOO" and "OBIIIOO". In contrast, according to the above annotation method of the present disclosure, the annotation algorithm based on the shortest edit path can be used for annotation. Even when the pre-annotated text sequence is composed of multiple scattered characters / character segments in the extracted training text sequence, the extracted training text sequence can be correctly annotated according to the pre-annotated text sequence, and the annotation result is "BIOIIOO".

[0114] In step S40, a text extraction model for extracting text from text boxes can be trained based on the extracted training sample set and the extracted annotation information.

[0115] Figure 5 It is a flowchart showing the steps of training a text extraction model according to an exemplary embodiment of the present disclosure. Figure 6 It is a schematic structural diagram of a text extraction model according to an exemplary embodiment of the present disclosure.

[0116] As an example, a text extraction model may include a third neural network, a fourth neural network, and a fifth neural network.

[0117] like Figure 5 As shown, step S40 may include the following steps:

[0118] In step S41, the extracted training sample set can be input into the third neural network to obtain the sample identifier vector output from the third neural network that corresponds one-to-one with the extracted training sample.

[0119] As an example, such as Figure 6 As shown, the third neural network can learn the semantic information of the overall expression of each extracted training text sequence based on the text of each extracted training text sequence, and output this semantic information in the form of a label vector according to a predetermined semantic rule. For example, the third neural network can output the embedding_segment vector corresponding to each extracted training text sequence. Here, the third neural network can be a BERT model, but it is not limited to this; it can also be other machine learning models used for semantic learning.

[0120] In step S42, the sample identifier vector can be input into the fourth neural network to obtain the first position prediction probability of each character in the extracted training text sequence appearing at each preset reference position.

[0121] As an example, such as Figure 6 As shown, the fourth neural network can predict the probability of each character in the training text sequence appearing at the first position in each preset reference position based on the sample identifier vector. Here, the fourth neural network can be a BiLSTM model for extracting bidirectional text information, but it is not limited to this; it can also be other machine learning models used for semantic learning.

[0122] In step S43, the first position prediction probability of each character in the extracted training text sequence can be input into the fifth neural network to obtain the second position prediction probability of each character in the extracted training text sequence appearing at each preset reference position, which is output from the fifth neural network. The second position prediction probability is determined based on the first position prediction probability between adjacent characters in the extracted training text sequence.

[0123] As an example, such as Figure 6 As shown, the fifth neural network can predict the second position prediction probability of each character in the training text sequence appearing at each preset reference position based on the first position prediction probability between adjacent characters in the extracted training text sequence. Here, the fifth neural network can be a CRF model that constrains the prediction results of the model by learning the state transition matrix, but it is not limited to this; it can also be other machine learning models used for semantic learning.

[0124] In step S44, the third, fourth, and fifth neural networks can be trained based on the second position prediction probability and extracted annotation information to obtain a trained text extraction model.

[0125] In this step, the predicted probability of the second position can be compared with the labeled position in the extracted annotation information, thereby training the third, fourth and fifth neural networks to obtain the final text extraction model.

[0126] According to an exemplary embodiment of this disclosure, the text extraction model may include a combination structure of a third neural network, a fourth neural network, and a fifth neural network, such as a combination of a BERT model, a BiLSTM model, and a CRF model. The third neural network is placed at the bottom layer to extract the text information of the context of the input extraction training text sequence. The fourth neural network can predict the probability of each character in the extraction training text sequence appearing at a first position at each preset reference position. The fifth neural network can correct the prediction results output by the fourth neural network based on the correlation between the prediction results output by the fourth neural network.

[0127] In step S50, an information extraction model for extracting information entities from an image can be constructed based on the trained text box classification model and the trained text extraction model.

[0128] Figure 7 This is a flowchart illustrating the steps of constructing an information extraction model based on a text box classification model and a text extraction model according to exemplary embodiments of the present disclosure.

[0129] Specifically, step S50 may include the following steps:

[0130] In step 51, the text in the text boxes classified into the same category can be grouped into a text sequence according to the text box classification output from the trained text box classification model.

[0131] For example, it is possible to extract all text boxes whose category is not related to the preset text box category, and to combine the text corresponding to the text boxes of the same category into a long string of text based on the position of the text boxes in the image in order (e.g., from top to bottom, from left to right). The text of different text boxes of the same category can be connected with delimiters, such as the symbol "[SEP]" in the code logic.

[0132] In step 52, the text sequence corresponding to each category can be input into the trained text extraction model to obtain the text position of each character in the information entity in the text sequence output by the trained text extraction model.

[0133] Here, text sequences of each category can be input into the trained text extraction model separately or together. Text sequences of different categories are independent of each other during the calculation process of the trained text extraction model.

[0134] Specifically, the text boxes under each category output by the trained text box classification model are concatenated into a single text sequence, and the corresponding category name is appended to each text sequence. This sequence can then be input into the trained text extraction model. For example, the text sequence can be segmented into multiple characters (e.g., token) by a tokenizer, and the vector value corresponding to each character can be extracted from the learnable weight matrix to form an embedding vector as a text identifier vector. This text identifier vector for each token is then input into the first neural network of the text extraction model.

[0135] In step 53, information entities under each category can be extracted from the image data according to the predicted text positions corresponding to each category, so as to construct an information extraction model.

[0136] In general, the information extraction model constructed according to the method of this disclosure can include a text box classification model and a text extraction model to process the input data in two stages: In the text box classification stage, the input of the text box classification model is the position of the text boxes and their corresponding text in the whole image, and the output of the text box classification model is the text box category to which each text box belongs; In the text extraction stage, the input of the text extraction model is the text sequence of all text boxes corresponding to the same text box category, and the output of the text extraction model is the information entity corresponding to each text box category. Thus, it is possible to understand images containing text based on the information extraction model, extract meaningful key information from the image, and display it as information entities of structured data.

[0137] It should be noted that although the steps are labeled in the accompanying drawings for ease of description, such as S10, S20, S30, S40, S50, etc., these labels and the order of description of each step do not mean that the steps must be executed in the order of the labels or descriptions. Those skilled in the art will understand that adjustments can be made according to the actual situation. For example, steps S30 and S40 can be executed after, before, or simultaneously with steps S10 and S20. That is, either the training process of the text box classification model or the training process of the text extraction model can be executed first, and then the other can be executed. Alternatively, the training processes of the text box classification model and the text extraction model can be executed simultaneously.

[0138] In a second aspect, an exemplary embodiment of the present disclosure proposes an information extraction method, which includes: acquiring image data of an image about which information to be extracted; and extracting information from the image data using an information extraction model based on the image data, wherein the information extraction model is constructed by a method for constructing an information extraction model according to an exemplary embodiment of the present disclosure.

[0139] Specifically, the image data may include the positions of text boxes identified in the image and the text within each text box. Here, the text boxes can be identified using existing text box detection models, and the text within the text boxes can be identified using existing text recognition models; this disclosure does not impose any particular limitations on this.

[0140] The image data described above can be input into the information extraction model to output the information entities of the image, thereby obtaining the information in the image. For details on the information extraction model and its construction process, please refer to the relevant description above; it will not be repeated here.

[0141] like Figure 6 As shown, the information extraction model outputs the predicted position of each character in the text sequence within each category of the information entity. During the text sequence decoding process, all characters in the text sequence can be traversed. A character predicted to be at the first character position is considered the start character of an information entity. Subsequent characters predicted to be at the entity character position are considered other characters belonging to the same information entity after the previous start character. When a character predicted to be at a position different from both the first and entity character positions is encountered, that character can be skipped. When a character predicted to be at the first character position is encountered again and considered the start character of an information entity, if the length of the previous information entity string is greater than 1, it indicates that the previous information entity has been fully decoded. The previous information entity can be saved to the entity sequence, and the category name of the text sequence can be used as the category of this information entity. Then, the entity string is reset to empty, and the next start character is encountered as the start of a new information entity. This decoding process is repeated to decode all information entities.

[0142] In a third aspect, exemplary embodiments of this disclosure propose an information extraction method. Figure 8 A flowchart illustrating this information extraction method is provided. The information extraction method includes the following steps:

[0143] In step S1, image data of the image from which the information to be extracted can be obtained.

[0144] Here, image data can include the position of text boxes in the image and the text in the text boxes.

[0145] In step S2, the text boxes can be categorized according to a preset text box category based on their position and the text within them. Here, the preset text box category represents the category of the information entity represented by the text in the text box.

[0146] Specifically, it can determine which category of the preset text box each text box belongs to based on the position and text of each text box.

[0147] In step S3, the text in text boxes that are classified into the same category can be grouped into a text sequence.

[0148] In step S4, for each category, based on the text sequence, the text position of each character in the text sequence in the corresponding information entity can be predicted.

[0149] Since texts in text boxes of the same category are grouped into text sequences in step S3, there is a certain correlation between each text sequence, making it easier to determine the position of each text in the information entity.

[0150] In step S5, information entities under each category can be extracted from the image data according to the predicted text position, so as to determine the information in the image based on the information entities.

[0151] According to the exemplary embodiment of the present disclosure, the information extraction method can first classify the text boxes, and then perform splicing and text position prediction for each category of text boxes. This avoids the problem of difficulty in processing long texts caused by existing text processing methods that do not classify text boxes but instead splice the text contained in the entire document of the identified image into a text sequence for text position prediction.

[0152] In the fourth aspect, such as Figure 9 As shown, an exemplary embodiment of this disclosure proposes an apparatus for constructing an information extraction model, the apparatus comprising a first acquisition unit 100, a first training unit 200, a second acquisition unit 300, a second training unit 400, and a construction unit 500.

[0153] The first acquisition unit 100 can be configured to acquire a classification training sample set and classification labeling information. Each classification training sample in the classification training sample set includes the position of the classification training text box in the image and the text in the classification training text box. The classification labeling information indicates which category the classification training text box of the classification training sample belongs to in the preset text box category.

[0154] The first training unit 200 can be configured to train a text box classification model for classifying text boxes based on a classification training sample set and classification labeling information.

[0155] The second acquisition unit 300 can be configured to acquire an extraction training sample set and extraction annotation information, wherein each extraction training sample in the extraction training sample set is an extraction training text sequence including one or more characters, and the extraction annotation information indicates the position of each character in the extraction training text sequence in the preset reference position of the information entity.

[0156] The second training unit 400 can be configured to train a text extraction model for extracting text from text boxes based on the extracted training sample set and extracted annotation information.

[0157] The building unit 500 can be configured to build an information extraction model for extracting information entities from an image based on a trained text box classification model and a trained text extraction model.

[0158] As an example, the first acquisition unit 100 can also be configured to: acquire the positions and categories of multiple pre-labeled text boxes in the image; determine the area overlap between the classification training text box and each of the multiple pre-labeled text boxes based on the positions of the classification training text box and the multiple pre-labeled text boxes, and label the classification training text box based on the area overlap to obtain classification labeling information.

[0159] Here, the area overlap refers to the ratio of the overlapping area between the classification training text box and the pre-labeled text box to the area of ​​the pre-labeled text box. When the area overlap between the classification training text box and the pre-labeled text box is greater than a preset threshold, the classification training text box is labeled as the same category as the corresponding pre-labeled text box.

[0160] As an example, the first acquisition unit 100 can also be configured to: when the area overlap between the classification training text box and each of the multiple pre-labeled text boxes is less than or equal to a preset threshold, label the classification training text box as a category different from the categories of the multiple pre-labeled text boxes.

[0161] As an example, the first training unit 200 can also be configured to: divide the classification training text box of each classification training sample into at least one classification training sub-text box; determine the position information of each classification training sub-text box based on the position of the classification training text box of each classification training sample, wherein the position information includes the position of at least one pixel of the classification training sub-text box in the image and the height and width of the classification training sub-text box or the positions of the pixels at the two vertices of the classification training sub-text box on the diagonal in the image; determine the text identifier vector corresponding to each classification training sub-text box based on the text of the classification training text box of each classification training sample, wherein the text identifier vector represents the semantics represented by the text in the classification training sub-text box; and train a text box classification model for classifying text boxes based on the position information of the classification training sub-text boxes, the text identifier vector, and the classification annotation information.

[0162] As an example, a text box classification model may include a first neural network and a second neural network.

[0163] In this example, the first training unit 200 can also be configured to: input the position information and text label vector of the classification training sub-text boxes into the first neural network to obtain sample label vectors output from the first neural network that correspond one-to-one with the classification training text boxes; arrange the sample label vectors corresponding to the classification training text boxes according to their position order in the image in the classification training samples to obtain a sample label vector sequence; input the sample label vector sequence into the second neural network to obtain the category prediction results output from the second neural network that correspond one-to-one with the classification training text boxes; and adjust the parameters of the first neural network and the second neural network based on the category prediction results and classification label information to obtain a trained text box classification model.

[0164] As an example, the second acquisition unit 300 can also be configured to: acquire a pre-annotated text sequence and text annotation information corresponding to each extracted training text sequence, wherein the text annotation information indicates the position of each text in the pre-annotated text sequence in the information entity; determine the minimum edit distance of the extracted training text sequence to the corresponding pre-annotated text sequence; and annotate each text in the extracted training text sequence according to preset annotation conditions based on the minimum edit distance and the text annotation information.

[0165] Here, the preset annotation conditions include: when the text extracted from the training text sequence at the minimum edit distance is the same as the text in the corresponding pre-annotated text sequence, the text extracted from the training text sequence is annotated as being at the same position as the corresponding text in the corresponding pre-annotated text sequence.

[0166] As an example, the preset annotation conditions may also include: when the text extracted from the training text sequence at the minimum edit distance is different from the text in the corresponding pre-annotated text sequence, annotation is performed according to the operation that achieves the minimum edit distance. Specifically, when the operation that achieves the minimum edit distance is a replacement operation, the text extracted from the training text sequence is annotated at the same position as the corresponding text in the corresponding pre-annotated text sequence; when the operation that achieves the minimum edit distance is a deletion operation or an addition operation, the text extracted from the training text sequence is annotated at a position that is different from the position of each text in the corresponding pre-annotated text sequence.

[0167] As an example, a text extraction model may include a third neural network, a fourth neural network, and a fifth neural network.

[0168] In this example, the second training unit 400 can also be configured to: input the extracted training sample set into the third neural network to obtain a sample identifier vector output from the third neural network that corresponds one-to-one with the extracted training sample; input the sample identifier vector into the fourth neural network to obtain a first position prediction probability of each character in the extracted training text sequence appearing at each preset reference position, output from the fourth neural network; input the first position prediction probability of each character in the extracted training text sequence into the fifth neural network to obtain a second position prediction probability of each character in the extracted training text sequence appearing at each preset reference position, wherein the second position prediction probability is determined based on the first position prediction probability between adjacent characters in the extracted training text sequence; and train the third neural network, the fourth neural network, and the fifth neural network based on the second position prediction probability and the extracted annotation information to obtain a trained text extraction model.

[0169] As an example, the preset reference position may include the position of the first character of the information entity and the positions of the entity characters other than the first character position. The extracted training text sequence is obtained by concatenating multiple extraction training text boxes.

[0170] In this example, the second acquisition unit 300 can also be configured to: combine the text contained in each preset information entity in the extracted training text sequence into a pre-labeled text sequence; determine the extraction training text boxes corresponding to the pre-labeled text sequence among multiple extraction training text boxes, wherein when there is more than one extraction training text box corresponding to the pre-labeled text sequence, the first character in the pre-labeled text sequence is labeled as the first character position, and all characters in the pre-labeled text sequence except the first character are labeled as entity character positions.

[0171] As an example, the building unit 500 can also be configured to: combine the text in the text boxes classified into the same category into a text sequence according to the text box classification output from the trained text box classification model; input the text sequence corresponding to each category into the trained text extraction model to obtain the text position of each text in the information entity in the text sequence output from the trained text extraction model; and extract the information entity under each category from the image data according to the predicted text position corresponding to each category to construct the information extraction model.

[0172] The first acquisition unit 100, the first training unit 200, the second acquisition unit 300, the second training unit 400, and the construction unit 500 can be configured as described above. Figures 1 to 7 The method embodiment shown executes the corresponding steps in the method for constructing the information extraction model. The specific implementation of the first acquisition unit 100, the first training unit 200, the second acquisition unit 300, the second training unit 400, and the construction unit 500 can be found in the method embodiment described above, and will not be repeated here.

[0173] In the fifth aspect, such as Figure 10 As shown, an exemplary embodiment of this disclosure proposes an information extraction device, which includes an image acquisition unit 1 and an information extraction unit 2.

[0174] The image acquisition unit 1 can be configured to acquire image data about the image from which information is to be extracted, wherein the image data includes the positions of text boxes in the image and the text in each text box.

[0175] Information extraction unit 2 can be configured to extract information from an image based on image data using an information extraction model. Here, the information extraction model can be constructed according to the information extraction model construction method described in this disclosure.

[0176] Image acquisition unit 1 and information extraction unit 2 can execute the corresponding steps in the method according to the construction method of the information extraction model in the above method embodiment. The specific implementation of image acquisition unit 1 and information extraction unit 2 can be found in the method embodiment described above, and will not be repeated here.

[0177] In the sixth aspect, such as Figure 11 As shown, an exemplary embodiment of this disclosure proposes an information extraction device, which includes an acquisition unit 10, a classification unit 20, a combination unit 30, a prediction unit 40, and an extraction unit 50.

[0178] The acquisition unit 10 can be configured to acquire image data of the image from which information is to be extracted, wherein the image data includes the position of the text boxes in the image and the text in each text box;

[0179] The classification unit 20 can be configured to classify text boxes according to a preset text box category 30 based on the position of the text box and the text in the text box. The preset text box category represents the category of the information entity represented by the text in the text box.

[0180] The combination unit 30 can be configured to combine text from text boxes that are classified into the same category into a text sequence;

[0181] The prediction unit 40 can be configured to predict the position of each character in the text sequence in the corresponding information entity for each category, based on the text sequence.

[0182] The extraction unit 50 can be configured to extract information entities under each category from the image data according to the predicted text positions, so as to determine the information in the image based on the information entities.

[0183] The acquisition unit 10, classification unit 20, combination unit 30, prediction unit 40, and extraction unit 50 can be based on the above reference. Figure 8 The construction method of the information extraction model in the method embodiment executes the corresponding steps in the method. The specific implementation of the acquisition unit 10, classification unit 20, combination unit 30, prediction unit 40 and extraction unit 50 can be found in the method embodiment described above, and will not be repeated here.

[0184] In a seventh aspect, an exemplary embodiment of the present disclosure provides an electronic device comprising: a processor; and a memory for storing processor-executable instructions, wherein the processor-executable instructions, when executed by the processor, cause the processor to perform a method for constructing an information extraction model according to the present disclosure or an information extraction method according to the present disclosure.

[0185] In an eighth aspect, exemplary embodiments of the present disclosure provide a computer-readable storage medium for storing instructions that, when executed by at least one computing device, cause the at least one computing device to perform a method for constructing an information extraction model according to the present disclosure or an information extraction method according to the present disclosure.

[0186] In a ninth aspect, exemplary embodiments of the present disclosure provide a system comprising at least one computing device and at least one storage device for storing instructions, wherein the instructions, when executed by the at least one computing device, cause the at least one computing device to perform a method for constructing an information extraction model according to the present disclosure or an information extraction method according to the present disclosure.

[0187] Figures 9 to 11The illustrated information extraction model construction apparatus and the individual units within the information extraction apparatus can be configured to perform specific functions as software, hardware, firmware, or any combination thereof. For example, each unit may correspond to a dedicated integrated circuit, pure software code, or a module combining software and hardware. Furthermore, one or more functions implemented by each unit can also be uniformly executed by components in a physical entity device (e.g., a processor, client, or server).

[0188] In addition, refer to Figures 1 to 8 The described method for constructing the information extraction model and / or the information extraction method can be implemented by a program (or instructions) recorded on a computer-readable storage medium. For example, according to an exemplary embodiment of the present disclosure, a computer-readable storage medium may be provided that stores instructions, wherein when the instructions are executed by at least one computing device, the at least one computing device causes the at least one computing device to perform the method for constructing the information extraction model and / or the information extraction method according to the present disclosure.

[0189] The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, and servers. It should be noted that the computer program can also be used to perform additional steps beyond those described above, or to perform more specific processing while performing the above steps. The details of these additional steps and further processing are already described in the reference... Figures 1 to 8 The relevant methods were mentioned in the description of the process, so they will not be repeated here to avoid repetition.

[0190] It should be noted that the various units in the information extraction model construction apparatus and the information extraction apparatus according to the exemplary embodiments of this disclosure can rely entirely on the operation of the computer program to realize their respective functions. That is, each unit corresponds to each step in the functional architecture of the computer program, so that the entire system is called through a special software package (e.g., a lib library) to realize the corresponding functions.

[0191] on the other hand, Figures 9 to 11 The units shown can also be implemented using hardware, software, firmware, middleware, microcode, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segment used to perform the corresponding operation can be stored in a computer-readable medium such as a storage medium, so that the processor can perform the corresponding operation by reading and running the corresponding program code or code segment.

[0192] For example, exemplary embodiments of the present disclosure can also be implemented as a computing device, which includes a storage component and a processor, wherein the storage component stores a set of computer-executable instructions, and when the set of computer-executable instructions is executed by the processor, a method for constructing an information extraction model and / or an information extraction method according to exemplary embodiments of the present disclosure is executed.

[0193] Specifically, the computing device can be deployed on a server or client, or on node devices in a distributed network environment. Furthermore, the computing device can be a PC, tablet, personal digital assistant, smartphone, web application, or other device capable of executing the aforementioned set of instructions.

[0194] Here, the computing device is not necessarily a single computing device, but can be any collection of devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. The computing device can also be part of an integrated control system or system manager, or can be configured to interconnect with a portable electronic device locally or remotely (e.g., via wireless transmission) through an interface.

[0195] In a computing device, a processor may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, a processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0196] Some operations described in the method for constructing the information extraction model and / or the information extraction method according to the exemplary embodiments of this disclosure can be implemented in software, some operations can be implemented in hardware, and these operations can also be implemented in a combination of software and hardware.

[0197] The processor can execute instructions or code stored in one of the storage components, which can also store data. Instructions and data can also be sent and received over a network via a network interface device, which can employ any known transport protocol.

[0198] Storage components can be integrated with the processor, for example, by placing RAM or flash memory within an integrated circuit microprocessor. Alternatively, storage components can include separate devices, such as external disk drives, storage arrays, or other storage devices that can be used by any database system. Storage components and the processor can be operatively coupled, or can communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor to read files stored in the storage component.

[0199] In addition, the computing device may include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of the computing device may be interconnected via a bus and / or network.

[0200] The method for constructing an information extraction model and / or an information extraction method according to exemplary embodiments of this disclosure can be described as various interconnected or coupled functional blocks or functional diagrams. However, these functional blocks or functional diagrams can be equally integrated into a single logic device or operate according to non-precise boundaries.

[0201] Therefore, refer to Figures 1 to 8 The described method for constructing the information extraction model and / or the information extraction method can be implemented by a system comprising a storage device including at least one computing device and at least one storage instruction.

[0202] According to an exemplary embodiment of the present disclosure, at least one computing device is a computing device for executing a method for constructing an information extraction model and / or an information extraction method according to an exemplary embodiment of the present disclosure. A storage device stores a set of computer-executable instructions. When the set of computer-executable instructions is executed by at least one computing device, a reference is executed. Figures 1 to 8 The method for constructing the described information extraction model and / or the information extraction method.

[0203] The foregoing has described various exemplary embodiments of this disclosure. It should be understood that the foregoing description is exemplary only and not exhaustive, and this disclosure is not limited to the disclosed exemplary embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A method for constructing an information extraction model, characterized in that, include: Obtain a classification training sample set and classification annotation information, wherein each classification training sample in the classification training sample set includes the position of the classification training text box in the image and the text in the classification training text box, and the classification annotation information indicates the preset text box category of the classification training text box of the classification training sample, and the preset text box category indicates the category of the information entity represented by the text in the text box; Based on the classification training sample set and the classification labeling information, a text box classification model is trained to classify text boxes; Obtain an extraction training sample set and extraction annotation information, wherein each extraction training sample in the extraction training sample set is an extraction training text sequence including one or more characters, and the extraction annotation information indicates the preset reference position of each character in the extraction training text sequence in the information entity; Based on the extracted training sample set and the extracted annotation information, a text extraction model for extracting text from text boxes is trained. Based on a pre-trained text box classification model and a pre-trained text extraction model, an information extraction model is constructed to extract information entities from images. The steps for constructing an information extraction model to extract information entities from an image, based on a trained text box classification model and a trained text extraction model, include: According to the text box classification output from the trained text box classification model, the text in the text boxes classified into the same category are combined into a text sequence; The text sequence corresponding to each category is input into the trained text extraction model to obtain the text position of each character in the information entity in the text sequence output by the trained text extraction model. Based on the predicted text position corresponding to each category, information entities under each category are extracted from the image to construct the information extraction model.

2. The construction method according to claim 1, characterized in that, The steps for obtaining the classification labeling information include: Obtain the positions of multiple pre-labeled text boxes in the image and the categories of the pre-labeled text boxes; Based on the positions of the classification training text boxes and the plurality of pre-labeled text boxes, the area overlap between the classification training text box and each of the plurality of pre-labeled text boxes is determined, and the classification training text box is labeled based on the area overlap to obtain the classification labeling information. The area overlap refers to the ratio of the overlapping area of ​​the classification training text box and the pre-labeled text box to the area of ​​the pre-labeled text box. When the area overlap of the classification training text box and the pre-labeled text box is greater than a preset threshold, the classification training text box is labeled as the same category as the corresponding pre-labeled text box.

3. The construction method according to claim 2, characterized in that, The steps for obtaining the classification labeling information also include: When the area overlap between the classification training text box and each of the plurality of pre-labeled text boxes is less than or equal to the preset threshold, the classification training text box is labeled as a category different from the categories of the plurality of pre-labeled text boxes.

4. The construction method according to claim 1, characterized in that, The steps for training a text box classification model for classifying text boxes based on the classification training sample set and the classification annotation information include: Divide the classification training text box of each classification training sample into at least one classification training sub-text box; Based on the position of the classification training text box of each classification training sample, the position information of each classification training sub-text box is determined, wherein the position information includes the position of at least one pixel of the classification training sub-text box in the image and the height and width of the classification training sub-text box or the position of the pixels of the classification training sub-text box located at the two vertices on the diagonal in the image. Based on the text in the classification training text box of each classification training sample, determine the text identifier vector corresponding to each classification training sub-text box, wherein the text identifier vector represents the semantics represented by the text in the classification training sub-text box; Based on the position information of the sub-text boxes, the text identifier vector, and the classification annotation information, a text box classification model is trained to classify the text boxes.

5. The construction method according to claim 4, characterized in that, The text box classification model includes a first neural network and a second neural network. The steps for training a text box classification model to classify text boxes, based on the position information of the sub-text boxes, the text identifier vector, and the classification annotation information, include: The position information and text identifier vector of the classification training sub-text box are input into the first neural network to obtain the sample identifier vector output from the first neural network that corresponds one-to-one with the classification training text box. According to the position order of the classification training text boxes in the image in the classification training samples, the sample identifier vectors corresponding to the classification training text boxes are arranged to obtain the sample identifier vector sequence; The sample identifier vector sequence is input into the second neural network to obtain the category prediction results output from the second neural network that correspond one-to-one with the classification training text boxes; Based on the category prediction results and the classification labeling information, the parameters of the first neural network and the second neural network are adjusted to obtain a trained text box classification model.

6. The construction method according to claim 1, characterized in that, The steps for obtaining the extracted annotation information include: Obtain a pre-annotated text sequence and text annotation information corresponding to each extracted training text sequence, wherein the text annotation information indicates the position of each text in the pre-annotated text sequence in the information entity; Determine the minimum edit distance from the extracted training text sequence to the corresponding pre-annotated text sequence; Based on the minimum edit distance and the text annotation information, each character in the extracted training text sequence is annotated according to preset annotation conditions. The preset labeling conditions include: When the text in the extracted training text sequence is the same as the text in the corresponding pre-labeled text sequence at the minimum edit distance, the text in the extracted training text sequence is labeled as being at the same position as the corresponding text in the corresponding pre-labeled text sequence.

7. The construction method according to claim 6, characterized in that, The preset annotation conditions also include: when the text in the extracted training text sequence at the minimum edit distance is different from the text in the corresponding pre-annotated text sequence, annotation is performed according to the operation that achieves the minimum edit distance. Specifically, when the operation to achieve the minimum edit distance is a replacement operation, the text in the extracted training text sequence is labeled with the same position as the corresponding text in the corresponding pre-labeled text sequence; when the operation to achieve the minimum edit distance is a deletion operation or an addition operation, the text in the extracted training text sequence is labeled with a position that is different from the position of each text in the corresponding pre-labeled text sequence.

8. The construction method according to claim 1, characterized in that, The text extraction model includes a third neural network, a fourth neural network, and a fifth neural network. The steps for training a text extraction model to extract text from a text box, based on the extracted training sample set and the extracted annotation information, include: The extracted training sample set is input into the third neural network to obtain a sample identifier vector output from the third neural network that corresponds one-to-one with the extracted training sample; The sample identifier vector is input into the fourth neural network to obtain the first position prediction probability of each character in the extracted training text sequence appearing at each preset reference position, which is output from the fourth neural network. The first position prediction probability of each character in the extracted training text sequence is input into the fifth neural network to obtain the second position prediction probability of each character in the extracted training text sequence appearing at each preset reference position, which is output from the fifth neural network. The second position prediction probability is determined based on the first position prediction probability between adjacent characters in the extracted training text sequence. Based on the second position prediction probability and the extracted annotation information, the third neural network, the fourth neural network, and the fifth neural network are trained to obtain a trained text extraction model.

9. The construction method according to any one of claims 6 to 8, characterized in that, The preset reference position includes the position of the first character of the information entity and the positions of other characters in the information entity besides the first character position. The extracted training text sequence is obtained by concatenating multiple extraction training text boxes. The steps for obtaining the pre-annotated text sequence and text annotation information corresponding to each extracted training text sequence include: The text contained in each preset information entity in the extracted training text sequence is combined into a pre-annotated text sequence; Identify the extraction training text box that corresponds to the pre-labeled text sequence among the plurality of extraction training text boxes. Specifically, when there is more than one extraction training text box corresponding to the pre-labeled text sequence, the first character in the pre-labeled text sequence is labeled as the first character position, and all characters in the pre-labeled text sequence except the first character are labeled as the entity character positions.

10. An information extraction method, characterized in that, include: Obtain image data of an image containing information to be extracted, wherein the image data includes the positions of text boxes in the image and the text in each text box; Based on the image data, information is extracted from the image using an information extraction model, wherein the information extraction model is constructed using the method for constructing an information extraction model according to any one of claims 1 to 9.

11. An information extraction method, characterized in that, include: Obtain image data of the image from which information is to be extracted, wherein the image data includes the positions of text boxes in the image and the text in each text box; Based on the position of the text box and the text in the text box, the text box is classified according to a preset text box category, wherein the preset text box category represents the category of the information entity represented by the text in the text box; The text in text boxes that are classified into the same category will be grouped into a text sequence; For each category, based on the text sequence, predict the text position of each character in the text sequence in the information entity corresponding to the corresponding category; Based on the predicted text location, information entities under each category are extracted from the image data to determine the information in the image based on the information entities.

12. An apparatus for constructing an information extraction model, characterized in that, include: The first acquisition unit is configured to acquire a classification training sample set and classification labeling information. Each classification training sample in the classification training sample set includes the position of a classification training text box in the image and the text in the classification training text box. The classification labeling information indicates the preset text box category of the classification training text box of the classification training sample. The preset text box category indicates the category of the information entity represented by the text in the text box. The first training unit is configured to train a text box classification model for classifying text boxes based on the classification training sample set and the classification labeling information. The second acquisition unit is configured to acquire an extraction training sample set and extraction annotation information, wherein each extraction training sample in the extraction training sample set is an extraction training text sequence including one or more characters, and the extraction annotation information indicates the preset reference position of each character in the extraction training text sequence in the information entity. The second training unit is configured to train a text extraction model for extracting text from text boxes based on the extracted training sample set and the extracted annotation information. The building unit is configured to construct an information extraction model for extracting information entities from an image, based on a trained text box classification model and a trained text extraction model. The construction unit is further configured to: combine the text in text boxes classified into the same category into a text sequence according to the text box classification output from the trained text box classification model; input the text sequence corresponding to each category into the trained text extraction model to obtain the text position of each text in the information entity in the text sequence output from the trained text extraction model; and extract the information entity under each category from the image according to the predicted text position corresponding to each category to construct the information extraction model.

13. The construction apparatus according to claim 12, characterized in that, The first acquisition unit is further configured to: Obtain the positions of multiple pre-labeled text boxes in the image and the categories of the pre-labeled text boxes; Based on the positions of the classification training text boxes and the plurality of pre-labeled text boxes, the area overlap between the classification training text box and each of the plurality of pre-labeled text boxes is determined, and the classification training text box is labeled based on the area overlap to obtain the classification labeling information. The area overlap refers to the ratio of the overlapping area of ​​the classification training text box and the pre-labeled text box to the area of ​​the pre-labeled text box. When the area overlap of the classification training text box and the pre-labeled text box is greater than a preset threshold, the classification training text box is labeled as the same category as the corresponding pre-labeled text box.

14. The construction apparatus according to claim 13, characterized in that, The first acquisition unit is further configured to: When the area overlap between the classification training text box and each of the plurality of pre-labeled text boxes is less than or equal to the preset threshold, the classification training text box is labeled as a category different from the categories of the plurality of pre-labeled text boxes.

15. The construction apparatus according to claim 12, characterized in that, The first training unit is also configured as follows: Divide the classification training text box of each classification training sample into at least one classification training sub-text box; Based on the position of the classification training text box of each classification training sample, the position information of each classification training sub-text box is determined, wherein the position information includes the position of at least one pixel of the classification training sub-text box in the image and the height and width of the classification training sub-text box or the position of the pixels of the classification training sub-text box located at the two vertices on the diagonal in the image. Based on the text in the classification training text box of each classification training sample, determine the text identifier vector corresponding to each classification training sub-text box, wherein the text identifier vector represents the semantics represented by the text in the classification training sub-text box; Based on the position information of the sub-text boxes, the text identifier vector, and the classification annotation information, a text box classification model is trained to classify the text boxes.

16. The construction apparatus according to claim 15, characterized in that, The text box classification model includes a first neural network and a second neural network. The first training unit is further configured as follows: The position information and text identifier vector of the classification training sub-text box are input into the first neural network to obtain the sample identifier vector output from the first neural network that corresponds one-to-one with the classification training text box. According to the position order of the classification training text boxes in the image in the classification training samples, the sample identifier vectors corresponding to the classification training text boxes are arranged to obtain the sample identifier vector sequence; The sample identifier vector sequence is input into the second neural network to obtain the category prediction results output from the second neural network that correspond one-to-one with the classification training text boxes; Based on the category prediction results and the classification labeling information, the parameters of the first neural network and the second neural network are adjusted to obtain a trained text box classification model.

17. The construction apparatus according to claim 12, characterized in that, The second acquisition unit is further configured to: Obtain a pre-annotated text sequence and text annotation information corresponding to each extracted training text sequence, wherein the text annotation information indicates the position of each text in the pre-annotated text sequence in the information entity; Determine the minimum edit distance from the extracted training text sequence to the corresponding pre-annotated text sequence; Based on the minimum edit distance and the text annotation information, each character in the extracted training text sequence is annotated according to preset annotation conditions. The preset labeling conditions include: When the text in the extracted training text sequence is the same as the text in the corresponding pre-labeled text sequence at the minimum edit distance, the text in the extracted training text sequence is labeled as being at the same position as the corresponding text in the corresponding pre-labeled text sequence.

18. The construction apparatus according to claim 17, characterized in that, The preset annotation conditions also include: when the text in the extracted training text sequence at the minimum edit distance is different from the text in the corresponding pre-annotated text sequence, annotation is performed according to the operation that achieves the minimum edit distance. Specifically, when the operation to achieve the minimum edit distance is a replacement operation, the text in the extracted training text sequence is labeled with the same position as the corresponding text in the corresponding pre-labeled text sequence; when the operation to achieve the minimum edit distance is a deletion operation or an addition operation, the text in the extracted training text sequence is labeled with a position that is different from the position of each text in the corresponding pre-labeled text sequence.

19. The construction apparatus according to claim 12, characterized in that, The text extraction model includes a third neural network, a fourth neural network, and a fifth neural network. The second training unit is further configured as follows: The extracted training sample set is input into the third neural network to obtain a sample identifier vector output from the third neural network that corresponds one-to-one with the extracted training sample; The sample identifier vector is input into the fourth neural network to obtain the first position prediction probability of each character in the extracted training text sequence appearing at each preset reference position, which is output from the fourth neural network. The first position prediction probability of each character in the extracted training text sequence is input into the fifth neural network to obtain the second position prediction probability of each character in the extracted training text sequence appearing at each preset reference position, which is output from the fifth neural network. The second position prediction probability is determined based on the first position prediction probability between adjacent characters in the extracted training text sequence. Based on the second position prediction probability and the extracted annotation information, the third neural network, the fourth neural network, and the fifth neural network are trained to obtain a trained text extraction model.

20. The construction apparatus according to any one of claims 17 to 19, characterized in that, The preset reference position includes the position of the first character of the information entity and the positions of other characters in the information entity besides the first character position. The extracted training text sequence is obtained by concatenating multiple extraction training text boxes. The second acquisition unit is further configured as follows: The text contained in each preset information entity in the extracted training text sequence is combined into a pre-annotated text sequence; Identify the extraction training text box that corresponds to the pre-labeled text sequence among the plurality of extraction training text boxes. Specifically, when there is more than one extraction training text box corresponding to the pre-labeled text sequence, the first character in the pre-labeled text sequence is labeled as the first character position, and all characters in the pre-labeled text sequence except the first character are labeled as the entity character positions.

21. An information extraction device, characterized in that, include: The image acquisition unit is configured to acquire image data about an image containing information to be extracted, wherein the image data includes the positions of text boxes in the image and the text in each text box; An information extraction unit is configured to extract information from the image based on the image data using an information extraction model, wherein the information extraction model is constructed using the method for constructing an information extraction model according to any one of claims 1 to 9.

22. An information extraction device, characterized in that, include: The acquisition unit is configured to acquire image data of an image from which information is to be extracted, wherein the image data includes the positions of text boxes in the image and the text in each text box; A classification unit is configured to classify the text box according to a preset text box category based on the position of the text box and the text in the text box, wherein the preset text box category represents the category of the information entity represented by the text in the text box; Combination units are configured to combine text from text boxes that will be categorized into the same category into a text sequence; The prediction unit is configured to predict, for each category, the text position of each character in the text sequence within the information entity corresponding to the corresponding category, based on the text sequence; The extraction unit is configured to extract information entities under each category from the image data according to the predicted text positions, so as to determine the information in the image based on the information entities.

23. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions. When the processor executes the processor, it causes the processor to perform the method for constructing the information extraction model according to any one of claims 1 to 9, or the information extraction method according to claim 10, or the information extraction method according to claim 11.

24. A computer-readable storage medium for storing instructions, characterized in that, When the instruction is executed by at least one computing device, it causes the at least one computing device to perform the method for constructing the information extraction model according to any one of claims 1 to 9, or the information extraction method according to claim 10, or the information extraction method according to claim 11.

25. A system comprising at least one computing device and at least one storage device for storing instructions, characterized in that, When the instruction is executed by the at least one computing device, it causes the at least one computing device to perform the method for constructing the information extraction model according to any one of claims 1 to 9, or the information extraction method according to claim 10, or the information extraction method according to claim 11.

Images