Structured information extraction method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a structured information extraction method and device, electronic equipment and storage medium, relates to the technical field of artificial intelligence, in particular to the technical field of deep learning, computer vision and optical character recognition (OCR). The specific implementation scheme is: obtaining image features, text features and text information corresponding to at least one field in a to-be-processed image; fusing the image features and the text features corresponding to the at least one field to obtain fusion features corresponding to the at least one field; determining target categories corresponding to the at least one field based on the fusion features corresponding to the at least one field; and organizing the text information corresponding to the at least one field according to a corresponding relationship between the target categories corresponding to the at least one field to obtain structured information. Thus, the structured information extraction problem is simplified to a classification problem, the structured information extraction process is simplified, and the accuracy of the structured information extraction is improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the technical field of deep learning, computer vision, and optical character recognition (OCR), and more particularly to a structured information extraction method and device, an electronic device, and a storage medium. BACKGROUND

[0002] Bill identification is in demand in various industries. For example, a medical prescription is a type of medical reimbursement bill. Due to the complicated data format and large variation among the same format of medical prescription data, the formats of the medical prescriptions are different among different hospitals. Therefore, the structured information extraction of the prescription data is a difficulty in image text recognition. For scenarios with a large amount of structured text information such as the prescription data, how to improve the accuracy of the structured information extraction is a problem to be solved in image text recognition. SUMMARY

[0003] The present disclosure provides a structured information extraction method, device, electronic device, and storage medium.

[0004] According to an aspect of the present disclosure, a structured information extraction method is provided. The method includes: obtaining image features, text features, and text information corresponding to at least one field in a to-be-processed image; fusing the image features and the text features corresponding to the at least one field to obtain fusion features corresponding to the at least one field; determining target categories corresponding to the at least one field based on the fusion features corresponding to the at least one field; and organizing the text information corresponding to the at least one field according to a correspondence relationship between the target categories corresponding to the at least one field to obtain structured information.

[0005] According to another aspect of the present disclosure, a model training method for structured information extraction is provided. The method includes: obtaining training data and an initial structured information extraction model; wherein the training data includes a plurality of sample images, sample text information corresponding to at least one sample field in the plurality of sample images, and sample position information of a text box to which the at least one sample field belongs, and the at least one sample field is labeled with a corresponding sample category; inputting the at least one sample image, the sample text information corresponding to the at least one sample field in the sample image, and the sample position information of the text box to which the at least one sample field belongs into the initial structured information extraction model to determine a predicted category corresponding to the at least one sample field by using the initial structured information extraction model; obtaining a difference between the predicted category corresponding to the at least one sample field and the sample category to adjust model parameters of the initial structured information extraction model according to the difference to obtain a target structured information extraction model for structured information extraction.

[0006] According to another aspect of the present disclosure, there is provided a structured information extraction apparatus, comprising: a first obtaining module configured to obtain image features, text features and text information corresponding to at least one field in an image to be processed; a fusion module configured to fuse the image features and the text features corresponding to the at least one field to obtain fusion features corresponding to the at least one field; a first determining module configured to determine target categories corresponding to the at least one field based on the fusion features corresponding to the at least one field; and an organizing module configured to organize the text information corresponding to the at least one field according to a correspondence relationship between the target categories corresponding to the at least one field to obtain structured information.

[0007] According to another aspect of the present disclosure, there is provided a model training apparatus for structured information extraction, comprising: a second obtaining module configured to obtain training data and an initial structured information extraction model; wherein the training data comprises a plurality of sample images, sample text information corresponding to at least one sample field in the sample images, and sample position information of a text box to which the at least one sample field belongs, and the at least one sample field is labeled with a corresponding sample category; a second determining module configured to input the at least one sample image, the sample text information corresponding to the at least one sample field in the sample images, and the sample position information of the text box to which the at least one sample field belongs into the initial structured information extraction model to determine predicted categories corresponding to the at least one sample field by using the initial structured information extraction model; and a third obtaining module configured to obtain differences between the predicted categories corresponding to the at least one sample field and the sample categories, and to adjust model parameters of the initial structured information extraction model according to the differences to obtain a target structured information extraction model for structured information extraction.

[0008] According to another aspect of the present disclosure, there is provided an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the structured information extraction method of the present disclosure, or to perform the model training method for structured information extraction of the present disclosure.

[0009] According to another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the structured information extraction method disclosed by the embodiments of the present disclosure, or to perform the model training method for structured information extraction disclosed by the embodiments of the present disclosure.

[0010] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of the structured information extraction method of the present disclosure, or implements the steps of the model training method for structured information extraction of the present disclosure.

[0011] One embodiment in the above application has the following advantages or beneficial effects:

[0012] Only the target categories corresponding to at least one field in the to-be-processed image need to be determined, and then the extraction of the structured information can be realized according to the corresponding relationship between the target categories corresponding to the at least one field, so that the structured information extraction problem is simplified to a classification problem, the process of structured information extraction is simplified, and the accuracy of structured information extraction is improved due to the combination of the two features of the text features and the image features of the at least one field in the to-be-processed image.

[0013] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0014] The accompanying drawings are used to better understand the present scheme and do not constitute a limitation on the present disclosure. Among them:

[0015] Figure 1 is a flowchart of a structured information extraction method according to a first embodiment of the present disclosure;

[0016] Figure 2 is a flowchart of a structured information extraction method according to a second embodiment of the present disclosure;

[0017] Figure 3 is an architecture diagram of a structured information extraction model according to the second embodiment of the present disclosure;

[0018] Figure 4 is a flowchart of a structured information extraction method according to a third embodiment of the present disclosure;

[0019] Figure 5 is a flowchart of a model training method for structured information extraction according to a fourth embodiment of the present disclosure;

[0020] Figure 6 is a flowchart of a model training method for structured information extraction according to a fifth embodiment of the present disclosure;

[0021] Figure 7 is a structural diagram of a structured information extraction device according to a sixth embodiment of the present disclosure

[0022] Figure 8 is a structural schematic diagram of a structured information extraction device according to a seventh embodiment of the present disclosure;

[0023] Figure 9 is a structural schematic diagram of a model training device for structured information extraction according to an eighth embodiment of the present disclosure;

[0024] Figure 10 is a structural schematic diagram of a model training device for structured information extraction according to a ninth embodiment of the present disclosure;

[0025] Figure 11 is a block diagram of an electronic device for implementing a structured information extraction method or a model training method for structured information extraction according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to help in understanding, which should be considered in their context. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0027] The structured information extraction method, the model training method for structured information extraction, the device, the electronic device, the non-transitory computer-readable storage medium, and the computer program product provided by the present disclosure relate to the field of artificial intelligence, specifically to the fields of deep learning, computer vision technology, and optical character recognition (OCR) technology.

[0028] Artificial intelligence is a discipline that studies enabling computers to simulate some thinking processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.) of humans, which includes both hardware-level technologies and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include computer vision, speech recognition technology, natural language processing technology, and machine learning / deep learning, big data processing technology, knowledge graph technology, and several other major directions.

[0029] The structured information extraction method, the model training method for structured information extraction, the device, the electronic device, the non-transitory computer-readable storage medium, and the computer program product of the embodiments of the present disclosure are described below with reference to the accompanying drawings.

[0030] Figure 1is a flowchart of a structured information extraction method according to a first embodiment of the present disclosure. It should be noted that the structured information extraction method of the present embodiment has a subject of execution of a structured information extraction device, which can be implemented by software and / or hardware and can be configured in an electronic device, which can include but is not limited to a terminal device, a server, etc., and the present embodiment does not make a specific limitation on the electronic device.

[0031] As shown in Figure 1 , the structured information extraction method can include:

[0032] In step 101, image features, text features, and text information corresponding to at least one field in a to-be-processed image are obtained.

[0033] The to-be-processed image can be an image with structured text information obtained by photographing, scanning, or the like of entity text, such as a medical prescription type bill image, a form image, a certificate image, etc., and the present disclosure does not make a limitation thereon.

[0034] In the embodiments of the present disclosure, the structured information extraction device can obtain image features, text features, and text information corresponding to each field in the to-be-processed image, respectively.

[0035] The field is composed of consecutive words, such as "name" in a bill image, a field, "gender" is a field, "drug name" is a field, and a specific name "Zhang San" is a field.

[0036] The image features are used to represent the image visual information of the field. The text features are obtained based on the text content of the field and are used to represent the text meaning of the field, such as the semantic features of the field. The text information is the text content information corresponding to the field.

[0037] In step 102, the image features and the text features corresponding to at least one field are fused to obtain fusion features corresponding to at least one field.

[0038] The fusion features corresponding to the field contain the image features and the text features corresponding to the field, such as by splicing the image features and the text features corresponding to the field to obtain the fusion features corresponding to the field.

[0039] In step 103, a target category corresponding to at least one field is determined based on the fusion features corresponding to the at least one field.

[0040] In the embodiments of the present disclosure, a plurality of categories can be preset, and the plurality of categories have a one-to-one correspondence relationship.

[0041] For example, in the related art, fields with key-value correspondence are generally divided into two categories: attribute item keys and attribute values. In the embodiments of the present disclosure, for attribute items, they can be further divided into multiple attribute item categories. For example, the attribute items of "age", "gender", and "address" can be further divided into three attribute item categories: an age item category, a gender item category, and an address item category. Similarly, for attribute values, they can also be further divided into multiple attribute value categories. For example, the attribute values corresponding to the attribute items of "age", "gender", and "address" can be further divided into three attribute value categories: an age value category, a gender value category, and an address value category. The divided age item category and age value category have a corresponding relationship, the gender item category and gender value category have a corresponding relationship, and the address item category and address value category have a corresponding relationship.

[0042] Further, the target categories corresponding to the at least one field can be determined from the preset multiple categories based on the fusion features corresponding to the at least one field. For example, the target category corresponding to each field can be one of the multiple attribute item categories or one of the multiple attribute value categories.

[0043] In step 104, the text information corresponding to the at least one field is organized according to the corresponding relationship between the target categories corresponding to the at least one field to obtain structured information.

[0044] In the embodiments of the present disclosure, since the preset multiple categories have a one-to-one correspondence relationship, after determining the target categories corresponding to the fields, the text information of the fields having a corresponding relationship between the corresponding target categories can be combined according to the corresponding relationship between the target categories corresponding to the fields, thereby obtaining one or more pieces of structured information.

[0045] For example, assume that for a certain image to be processed, the image features, text features, and text information corresponding to four fields included in the image are obtained, where the text information corresponding to the four fields is "name", "age", "18", and "Zhang San". By fusing the image features and text features corresponding to the four fields, the fusion features corresponding to the four fields are obtained, and then the target categories corresponding to the four fields are determined based on the fusion features corresponding to the four fields. Among them, the field 1 with the text information "name" corresponds to the name item category; the field 2 with the text information "age" corresponds to the age item category; the field 3 with the text information "18" corresponds to the age value category; and the field 4 with the text information "Zhang San" corresponds to the name value category.

[0046] Since the age item category corresponding to field 2 and the age value category corresponding to field 3 have a corresponding relationship, and the name item category corresponding to field 1 and the name value category corresponding to field 4 have a corresponding relationship, the "name" corresponding to field 1 and the "Zhang San" corresponding to field 4 can be combined to obtain a piece of structured information, and the "age" corresponding to field 2 and the "18" corresponding to field 3 can be combined to obtain a piece of structured information.

[0047] By presetting a plurality of categories and a one-to-one corresponding relationship between the plurality of categories, after determining the target categories corresponding to the fields respectively from the plurality of preset categories, the text information corresponding to the fields respectively can be organized based on the corresponding relationship between the target categories corresponding to the fields respectively to obtain the structured information. Moreover, since the structured information extraction is performed in combination with the text features and the image features of at least one field in the image to be processed, the accuracy of the structured information extraction can be improved, so that the structured information can be accurately extracted for medical bill data such as medical bill tickets in a natural scene with problems such as misaligned lines and complex formats.

[0048] The structured information extraction method provided in the embodiments of the present disclosure obtains the image features, the text features, and the text information corresponding to at least one field in an image to be processed, fuses the image features and the text features corresponding to the at least one field to obtain fusion features corresponding to the at least one field, determines target categories corresponding to the at least one field based on the fusion features corresponding to the at least one field, and organizes the text information corresponding to the at least one field according to the corresponding relationship between the target categories corresponding to the at least one field to obtain structured information. Therefore, only the target categories corresponding to the at least one field in the image to be processed need to be determined, and the extraction of the structured information can be implemented according to the corresponding relationship between the target categories corresponding to the at least one field, so that the structured information extraction problem is simplified to a classification problem, the process of the structured information extraction is simplified, and the accuracy of the structured information extraction is improved since the structured information extraction is performed in combination with the text features and the image features of at least one field in the image to be processed.

[0049] The following Figure 2 The processes of obtaining the image features, the text features, and the text information corresponding to at least one field in an image to be processed, fusing the image features and the text features corresponding to the at least one field to obtain fusion features, and determining target categories corresponding to the at least one field based on the fusion features corresponding to the at least one field in the structured information extraction method provided in the present disclosure are further described.

[0050] Figure 2 is a flowchart of a structured information extraction method according to a second embodiment of the present disclosure. As Figure 2As shown, the structured information extraction method can include the following steps:

[0051] In step 201, the textual information corresponding to at least one field in the to-be-processed image obtained by performing optical character recognition on the to-be-processed image is acquired, and position information of a text box to which the at least one field belongs is acquired.

[0052] The position information of the text box to which the field belongs can specifically be coordinates of four corner points of the text box to which the field belongs.

[0053] In the embodiments of the present disclosure, the to-be-processed image can be input to an optical character recognition interface, and optical character recognition is performed on the to-be-processed image to acquire the textual information corresponding to each field in the to-be-processed image and the position information of the text box to which each field belongs. The structured information extraction apparatus can be connected to the optical character recognition interface, so that the textual information corresponding to each field in the to-be-processed image and the position information of the text box to which each field belongs can be acquired through the optical character recognition interface.

[0054] In step 202, the to-be-processed image, the textual information corresponding to at least one field in the to-be-processed image, and the position information of the text box to which the at least one field belongs are input to a feature extraction network to perform feature extraction by using the feature extraction network, and image features and textual features corresponding to the at least one field in the to-be-processed image are acquired.

[0055] In the embodiments of the present disclosure, an end-to-end structured information extraction model can be pre-trained, the input of the structured information extraction model is an image, textual information corresponding to at least one field in the image, and position information of a text box to which the at least one field in the image belongs, and the output of the structured information extraction model is a category corresponding to the at least one field in the image. The specific training process can refer to the description of the embodiments of the model training method for structured information extraction in the following embodiments, and will not be described here.

[0056] The feature extraction network is included in the structured information extraction model and is used to extract image features and textual features corresponding to at least one field in an image, so that the to-be-processed image, the textual information corresponding to at least one field in the to-be-processed image, and the position information of the text box to which the at least one field belongs acquired in step 201 can be input to the feature extraction network to perform feature extraction by using the feature extraction network, and the image features and the textual features corresponding to the at least one field in the to-be-processed image are acquired.

[0057] By adopting the feature extraction network, the image features and the text features corresponding to the at least one field in the to-be-processed image are extracted based on the to-be-processed image and the text information corresponding to the at least one field and the position information of the text box to which the at least one field belongs, so that accurate image features and text features can be extracted, and a foundation is laid for subsequent accurate structured information extraction based on the image features and the text features.

[0058] In an embodiment of the present disclosure, referring to Figure 3 , the feature extraction network can include a convolutional neural network 301 and a recurrent neural network 302. The convolutional neural network 301 can be any type of convolutional neural network capable of image feature extraction, and the present disclosure does not limit this. The recurrent neural network 302 can be any type of recurrent neural network capable of text feature extraction, and the present disclosure does not limit this.

[0059] In an embodiment of the present disclosure, referring to Figure 3 The image features and the text features corresponding to the at least one field in the to-be-processed image can be obtained in the following manner:

[0060] The to-be-processed image is input into the convolutional neural network 301 to perform image feature extraction on the to-be-processed image by using the convolutional neural network 301 to obtain a feature map; the corresponding region in the feature map is intercepted according to the position information of the text box to which the at least one field belongs to obtain the image features corresponding to the at least one field; and the text information corresponding to the at least one field is input into the recurrent neural network 302 to perform text feature extraction on the text information by using the recurrent neural network 302 to obtain the text features corresponding to the at least one field.

[0061] The corresponding region in the feature map can be intercepted according to the position information of the text box to which the at least one field belongs by one processing layer of the structured information extraction model to obtain the image features corresponding to the at least one field.

[0062] Through the above process, the image features and the text features corresponding to the at least one field in the to-be-processed image are accurately extracted based on the convolutional neural network and the recurrent neural network, and a foundation is laid for subsequent accurate structured information extraction based on the image features and the text features.

[0063] It is understandable that when using a convolutional neural network to extract image features from an image to be processed, the resulting feature map is the same size as the image to be processed. However, since the number of characters in each field of an image may be different, the size of the text boxes belonging to each field may also be different. Consequently, based on the position information of the text boxes belonging to each field, the size of the corresponding region of the text box extracted in the feature map may also be different, and thus the dimension of the image features corresponding to each field may be different. This dimension includes width, height, and number of channels. For a model, it is required that the dimensions of each input feature be the same. Therefore, in this embodiment of the disclosure, after extracting the corresponding region in the feature map based on the position information of the text boxes belonging to at least one field to obtain the image features corresponding to at least one field, the dimension of the image features corresponding to at least one field can be processed to make the dimensions of the image features corresponding to each field the same.

[0064] Correspondingly, structured information extraction models can also include pooling layers ( Figure 3 (Not shown in the image), so that after extracting the corresponding region in the feature map based on the position information of the text box to which at least one field belongs, and obtaining the image features corresponding to at least one field, the image features can be input into the pooling layer so that the pooling layer can perform pooling processing on the image features corresponding to at least one field so that the dimensions of the image features corresponding to at least one field are the same, that is, the width, height and number of channels of the image features corresponding to at least one field are the same.

[0065] Step 203: Input the image features and text features corresponding to at least one field into the feature fusion layer, so that the feature fusion layer can be used to concatenate the image features and text features corresponding to at least one field, or to perform Kronecker product operation on the image features and text features corresponding to at least one field to obtain the fused features corresponding to at least one field.

[0066] In embodiments of this disclosure, reference is made to Figure 3 The structured information extraction model may also include a feature fusion layer 303, which is used to fuse image features and text features corresponding to at least one field. This allows the image features and text features corresponding to at least one field to be input into the feature fusion layer, so that the feature fusion layer can fuse the image features and text features corresponding to at least one field to obtain the fused features corresponding to at least one field.

[0067] In the embodiments of the present disclosure, for each field, the fusion of the corresponding image features and text features can be implemented in multiple ways. As one possible implementation, for each field, the image features and text features corresponding to the field can be spliced, and the spliced features are taken as the fusion features corresponding to the field. Alternatively, as another possible implementation, for each field, the Kronecker product of the image features and text features corresponding to the field is taken as the fusion features of the field. By using the feature fusion layer to fuse the image features and text features corresponding to at least one field, the fusion features corresponding to each field in the to-be-processed image are obtained, and by using the Kronecker product operation to fuse the image features and text features corresponding to the field, a better feature fusion effect can be achieved.

[0068] It should be noted that the above feature fusion manner is only illustrative and cannot be understood as a limitation on the technical solutions of the present disclosure. In actual applications, those skilled in the art can use any other manner to implement the fusion of the image features and text features corresponding to the field, and the present disclosure does not limit this.

[0069] In step 204, the fusion features corresponding to at least one field are taken as the node features of at least one node in the graph network, and input into the graph network to obtain the probabilities of the multiple preset categories corresponding to the at least one field output by the graph network; wherein the at least one field corresponds to a node in the graph network.

[0070] In the embodiments of the present disclosure, referring to Figure 3 , the structured information extraction model can further include a graph network 304, which is configured to determine the category corresponding to at least one field based on the fusion features corresponding to the at least one field in the image, so that the fusion features corresponding to the at least one field in the to-be-processed image can be input into the graph network as the node features of at least one node in the graph network, and the node features of the at least one node are encoded by using the graph network to obtain the target category corresponding to the at least one field. Each field corresponds to a node in the graph network.

[0071] In the embodiments of the present disclosure, a plurality of categories (i.e., the multiple preset categories in step 204) can be set in advance. After obtaining the fusion features corresponding to the at least one field in the to-be-processed image, the fusion features corresponding to the at least one field can be input into the graph network as the node features of at least one node in the graph network to obtain the probabilities of the multiple preset categories corresponding to the at least one field output by the graph network. That is, the graph network can be used to predict the probabilities of the multiple preset categories corresponding to each field.

[0072] In step 205, the target category corresponding to the at least one field is determined from the multiple preset categories according to the probabilities.

[0073] In the embodiments of the present disclosure, for each field, a preset category with the maximum corresponding probability in the plurality of preset categories can be determined as the target category corresponding to the field.

[0074] Through the above process, the target category corresponding to each field is accurately determined based on the fusion features of each field in the to-be-processed image by using the graph network, which lays a foundation for subsequent accurate structured information extraction according to the corresponding relationship between the target categories corresponding to the at least one field.

[0075] Step 206: Organize the text information corresponding to the at least one field according to the corresponding relationship between the target categories corresponding to the at least one field, to obtain the structured information.

[0076] The specific implementation process and principles of step 206 can be referred to the description of the above embodiments, which will not be described here.

[0077] The structured information extraction method of the embodiments of the present disclosure realizes the fusion of the image features and the text features corresponding to the at least one field based on the image features, the text features and the text information of the at least one field in the to-be-processed image by using the end-to-end model, to obtain the fusion features corresponding to the at least one field, and determines the target categories corresponding to the at least one field based on the fusion features corresponding to the at least one field, and then enables the organization of the text information corresponding to the at least one field according to the corresponding relationship between the target categories corresponding to the at least one field, to obtain the structured information, thereby simplifying the structured information extraction problem into a classification problem, simplifying the process of structured information extraction, and improving the accuracy of structured information extraction by combining the text features and the image features of the at least one field in the to-be-processed image for structured information extraction.

[0078] The following will be described in combination with Figure 4 The process of organizing the text information corresponding to the at least one field according to the corresponding relationship between the target categories corresponding to the at least one field to obtain the structured information in the structured information extraction method provided by the present disclosure will be further described.

[0079] Figure 4 is a flowchart of the structured information extraction method according to the third embodiment of the present disclosure. As shown in Figure 4 The structured information extraction method can include the following steps:

[0080] Step 401: Obtain the image features, the text features and the text information corresponding to the at least one field in the to-be-processed image.

[0081] At step 402, the image features and text features corresponding to the at least one field are fused to obtain fused features corresponding to the at least one field.

[0082] The specific implementation process and principles of steps 401-402 can refer to the descriptions of the above embodiments, which will not be repeated here.

[0083] At step 403, the target category corresponding to the at least one field is determined based on the fused features corresponding to the at least one field.

[0084] In the embodiments of the present disclosure, a graph network can be pre-trained, so that the fused features corresponding to the at least one field can be input into the graph network as node features of at least one node in the graph network to obtain probabilities of a plurality of preset categories corresponding to the at least one field output by the graph network, wherein the at least one field corresponds to a node in the graph network, and then for each field, the preset category with the largest probability in the plurality of preset categories is determined as the target category corresponding to the field according to the above probabilities.

[0085] Through the above process, the graph network is used to accurately determine the target categories corresponding to the fields based on the fused features corresponding to the fields in the image to be processed, which lays a foundation for subsequent accurate structured information extraction based on the corresponding relationship between the target categories corresponding to the fields.

[0086] In the embodiments of the present disclosure, a plurality of categories (i.e., the plurality of preset categories in the above embodiments) can be pre-set, and the preset plurality of categories have a one-to-one correspondence relationship, and then the fused features corresponding to the at least one field can be used for reasoning to determine the target categories corresponding to the respective fields from the plurality of preset categories, and then the text information of the fields with the corresponding relationship between the target categories corresponding to the fields is combined according to the corresponding relationship between the target categories corresponding to the fields, so as to obtain one or more pieces of structured information.

[0087] It can be understood that in the related art, fields with key-value correspondence are generally divided into two categories: attribute item key and attribute value value. In the embodiments of the present disclosure, for the attribute item, it can be further divided into multiple attribute item categories, for example, the attribute items of "age", "gender" and "address" can be further divided into three attribute item categories: age item category, gender item category and address item category. Similarly, for the attribute value, it can also be further divided into multiple attribute value categories, for example, the attribute values corresponding to the attribute items of "age", "gender" and "address" are further divided into three attribute value categories: age value category, gender value category and address value category. Among them, the age item category and the age value category have a corresponding relationship, the gender item category and the gender value category have a corresponding relationship, and the address item category and the address value category have a corresponding relationship.

[0088] That is, in the embodiments of the present disclosure, multiple attribute item categories and multiple attribute value categories can be pre-set, and the multiple attribute item categories and the multiple attribute value categories have a one-to-one correspondence relationship. The target category corresponding to at least one field in the to-be-processed image is at least one of the multiple attribute item categories or at least one of the multiple attribute value categories. Correspondingly, the text information corresponding to the at least one field can be organized to obtain structured information according to the correspondence relationship between the target categories corresponding to the at least one field in the to-be-processed image by the following steps 404-406.

[0089] Step 404, for a first field with a first attribute item category as the target category, querying a second field with a first attribute value category having a corresponding relationship as the target category.

[0090] Among them, the first attribute item category is one of the multiple attribute item categories. The first attribute value category is one of the multiple attribute value categories. The first field and the second field are respectively one of the at least one field in the to-be-processed image.

[0091] For example, assuming that the first attribute item category corresponding to a first field in the to-be-processed image is the name item category, and the first attribute value category having a corresponding relationship with the name item category is the name value category, then the second field with the target category of the name value category can be queried from each field in the to-be-processed image.

[0092] Step 405, obtaining first text information corresponding to the first field and second text information corresponding to the second field.

[0093] The step of obtaining the first literal information corresponding to the first field can also be performed before the step of querying the second field whose target category is the first attribute value category having the corresponding relationship. The execution timing of the step of obtaining the first literal information corresponding to the first field is not limited in the embodiment of the present disclosure.

[0094] At step 406, a piece of structured information is generated according to the first literal information and the second literal information.

[0095] Continuing the above example, after the second field whose target category is the name value category is queried, the second literal information corresponding to the second field can be obtained, and then the first literal information and the second literal information are combined to generate a piece of structured information.

[0096] By dividing the attribute item category and the attribute value category into more fine-grained categories, and by defaulting that the attribute item category and the attribute value category have a one-to-one corresponding relationship, then according to the first field whose target category is the first attribute item category, the second field whose target category is the first attribute value category having the corresponding relationship is queried, the first literal information corresponding to the first field and the second literal information corresponding to the second field are obtained, and a piece of structured information is generated according to the first literal information and the second literal information. This realizes the extraction of the structured information of the field whose target category is the attribute item category and the attribute value category in the to-be-processed image, only according to the corresponding relationship between the target categories of at least one field.

[0097] It can be understood that in actual application, for a field whose target category is a certain attribute item category, the field whose target category is the attribute value category having the corresponding relationship with the attribute item category can be multiple, that is, the number of the above-mentioned second field can be multiple. In this case, the second literal information corresponding to multiple second fields can be merged to obtain merged information, and then a piece of structured information is generated according to the first literal information and the merged information.

[0098] For example, it is assumed that the address of a certain place in the to-be-processed image includes 3 lines, each line is a field, and the target categories of the 3 fields are all address value categories corresponding to the address item category. Correspondingly, according to the first field whose target category is the address item category, it can be queried that the number of the second field whose target category is the address value category is 3, so that the literal information corresponding to the 3 fields can be merged, that is, merged into a piece of address information, and then the merged address information can be combined with the first literal information of the first field whose target category is the address item category to generate a piece of structured information.

[0099] By merging the second literal information corresponding to the multiple second fields when the number of the second fields is multiple, the merged information is obtained, and then according to the first literal information and the merged information, a piece of structured information is generated, which realizes that in the case that the number of the second fields of the target category being the attribute item category and the target category being the attribute value category is one-to-many, the literal information corresponding to each field is organized according to the corresponding relationship between the target categories of each field, and accurate structured information is obtained.

[0100] In the embodiments of the present disclosure, similar to the above-mentioned division manner of the multiple attribute item categories and the multiple attribute value categories, the two categories of the table header of the table and the cell of the table can also be further divided into multiple table header categories and multiple table cell categories respectively. Among them, the multiple table header categories and the multiple table cell categories after division have a one-to-one corresponding relationship. The target category corresponding to at least one field is at least one of the multiple table header categories or at least one of the multiple table cell categories.

[0101] Correspondingly, the organization of the literal information corresponding to at least one field according to the corresponding relationship between the target categories of at least one field to obtain structured information can be realized in the following manner: querying the fourth field whose target category is the first table cell category having a corresponding relationship according to the third field whose target category is the first table header category; obtaining the third literal information corresponding to the third field and the fourth literal information corresponding to the fourth field; and generating a piece of structured information according to the third literal information and the fourth literal information.

[0102] Among them, the first table header category is one of the multiple table header categories. The first table cell category is one of the multiple table cell categories. The third field and the fourth field are respectively one of the at least one field in the image to be processed.

[0103] For example, assuming that the first table header category corresponding to a certain third field in the image to be processed is category A, and the third field corresponds to third literal information, since the first table cell category having a corresponding relationship with category A is category B, the fourth field whose target category is category B can be queried from each field in the image to be processed. After the fourth field whose target category is category B is queried, the fourth literal information corresponding to the fourth field can be obtained, and then the third literal information and the fourth literal information are combined to generate a piece of structured information.

[0104] By dividing the table header category and the table cell category into more fine-grained categories, and predefining that the table header category has a one-to-one correspondence with the table cell category, and then according to the target category for the third field of the first table header category, querying the fourth field of the first table cell category having the corresponding relationship with the target category, obtaining the third text information corresponding to the third field and the fourth text information corresponding to the fourth field, and generating a piece of structured information according to the third text information and the fourth text information, the extraction of the structured information of the field whose target category is the table header category and the table cell category in the to-be-processed image is implemented only according to the corresponding relationship between the target categories corresponding to at least one field.

[0105] It should be noted that, in a possible implementation form, the extraction result of the structured information can include not only the first recognition information corresponding to the first field whose target category is the attribute item category and the second recognition information corresponding to the second field whose target category is the attribute value category, or the third recognition information corresponding to the third field whose target category is the table header category and the fourth recognition information corresponding to the fourth field whose target category is the table cell, but also the probability that the determined first field, second field, third field or fourth field is the corresponding target category, the position information of the text box to which the field belongs, and the like, which are not limited by the present disclosure.

[0106] According to the embodiments of the present disclosure, a model training method for structured information extraction is also provided.

[0107] Figure 5 is a flowchart of the model training method for structured information extraction according to the fourth embodiment of the present disclosure.

[0108] It should be noted that the model training method for structured information extraction provided by the embodiments of the present disclosure has a subject of a model training device for structured information extraction, which is referred to as a model training device hereinafter. The model training device can be implemented by software and / or hardware, and the model training device can be configured in an electronic device, which can include but is not limited to a terminal device, a server, and the like, and the embodiments of the present disclosure do not make specific limitations on the electronic device.

[0109] As shown in Figure 5 , the model training method for structured information extraction can include the following steps:

[0110] Step 501, obtaining training data and an initial structured information extraction model.

[0111] The training data includes a plurality of sample images, sample text information corresponding to at least one sample field in the plurality of sample images, and sample position information of a text box to which the at least one sample field belongs, and the at least one sample field is labeled with a corresponding sample category.

[0112] The initial structured information extraction model is an end-to-end model, and the initial structured information extraction model takes an image, text information corresponding to at least one field in the image, and position information of a text box to which the at least one field belongs as input, and outputs a category corresponding to the at least one field in the image.

[0113] In the embodiments of the present disclosure, the sample text information corresponding to the at least one sample field in each sample image included in the training data and the sample position information of the text box to which the at least one sample field belongs can be obtained in advance by using an optical character recognition technology, so that when the model is trained, the optical character recognition result can be directly used for model training as known.

[0114] Step 502, input at least one sample image, sample text information corresponding to at least one sample field in the sample image, and sample position information of a text box to which the at least one sample field belongs into the initial structured information extraction model, to determine a predicted category corresponding to the at least one sample field by using the initial structured information extraction model.

[0115] In the embodiments of the present disclosure, the initial structured information extraction model can predict the category corresponding to the at least one sample field based on the sample image, the sample text information corresponding to the at least one sample field in the sample image, and the sample position information of the text box to which the at least one sample field belongs, to obtain the predicted category corresponding to the at least one sample field.

[0116] Step 503, obtain a difference between the predicted category corresponding to the at least one sample field and the sample category, to adjust model parameters of the initial structured information extraction model according to the difference, to obtain a target structured information extraction model for structured information extraction.

[0117] In the embodiments of the present disclosure, after obtaining the predicted category corresponding to the at least one sample field, the difference between the predicted category corresponding to the at least one sample field and the sample category can be obtained, and then the model parameters of the initial structured information extraction model can be adjusted according to the difference, to obtain the target structured information extraction model for structured information extraction.

[0118] When the initial structured information extraction model is trained, the loss function of the initial structured information extraction model can be a cross-entropy loss of the category of the sample field shown in the following formula (1).

[0119] L = -t j log(y j ) (1)

[0120] Wherein, L is the cross-entropy loss of the category corresponding to each sample field. t jy is a sample category corresponding to the sample field j y is a predicted category corresponding to the sample field. j represents the jth category.

[0121] In the model training of the initial structured information extraction model, a deep learning method can be used. Compared with other training methods, the deep learning method has better performance on big data. In the process of training the initial structured information extraction model according to the difference between the predicted category and the sample category corresponding to at least one sample field, reference can be made to related technologies, which will not be described herein.

[0122] It should be noted that the target structured information extraction model obtained after training in the embodiments of the present disclosure can be used to execute the following steps:

[0123] Obtaining image features and text features corresponding to at least one field in the to-be-processed image;

[0124] Fusing the image features and the text features corresponding to the at least one field to obtain fusion features corresponding to the at least one field;

[0125] Determining target categories corresponding to the at least one field based on the fusion features corresponding to the at least one field, wherein a corresponding relationship between the target categories corresponding to the at least one field is used to organize text information corresponding to the at least one field to obtain structured information.

[0126] The process of executing the above steps by using the target structured information extraction model obtained after training can refer to the description of the embodiments of the above structured information extraction method, which will not be described herein.

[0127] In summary, the model training method for structured information extraction provided by the embodiments of the present disclosure includes the following steps: obtaining training data and an initial structured information extraction model; inputting at least one sample image, sample text information corresponding to at least one sample field in the sample image, and sample position information of a sample text box to which the at least one sample field belongs into the initial structured information extraction model to determine a predicted category corresponding to the at least one sample field by using the initial structured information extraction model; obtaining a difference between the predicted category corresponding to the at least one sample field and a sample category; adjusting model parameters of the initial structured information extraction model according to the difference to obtain a target structured information extraction model for structured information extraction. The model training method for structured information extraction provided by the embodiments of the present disclosure can realize model training of the initial structured information extraction model based on the training data to obtain the target structured information extraction model for structured information extraction. The target structured information extraction model obtained after training can be used for structured information extraction, which can simplify the structured information extraction problem into a classification problem, simplify the process of structured information extraction, and improve the accuracy of structured information extraction by combining the text features and image features of at least one field in the image to be processed.

[0128] The following will be described in detail Figure 6 with reference to the model training device for structured information extraction provided by the present disclosure.

[0129] Figure 6 FIG. 1 is a flowchart of a model training method for structured information extraction according to a fifth embodiment of the present disclosure.

[0130] As shown in Figure 6 , the model training method for structured information extraction can include the following steps:

[0131] Step 601: obtaining training data and an initial structured information extraction model.

[0132] The training data includes a plurality of sample images, sample text information corresponding to at least one sample field in the plurality of sample images, and sample position information of a sample text box to which the at least one sample field belongs. The at least one sample field is labeled with a corresponding sample category.

[0133] The specific implementation process and principles of step 601 can refer to the description of the above embodiments, which will not be described here.

[0134] Step 602: inputting the sample image, sample text information corresponding to the at least one sample field in the sample image, and sample position information of the sample text box to which the at least one sample field belongs into a feature extraction network to perform feature extraction by using the feature extraction network to obtain sample image features and sample text features corresponding to the at least one sample field in the sample image.

[0135] In the embodiment of the present disclosure, the initial structured information extraction model can include a feature extraction network for extracting image features and text features corresponding to the at least one field in the image, so that the sample image, the sample text information corresponding to the at least one sample field in the sample image, and the sample position information of the text box to which the at least one sample field belongs can be input into the feature extraction network to perform feature extraction by using the feature extraction network, to obtain sample image features and sample text features corresponding to the at least one sample field in the sample image.

[0136] In the embodiment of the present disclosure, the feature extraction network can include a convolutional neural network and a recurrent neural network, and correspondingly, the step 602 can be implemented in the following manner:

[0137] The sample image is input into the convolutional neural network to perform image feature extraction on the sample image by using the convolutional neural network, to obtain a sample feature map;

[0138] According to the sample position information of the text box to which the at least one sample field belongs, a corresponding region in the sample feature map is intercepted to obtain sample image features corresponding to the at least one sample field;

[0139] The sample text information corresponding to the at least one sample field in the sample image is input into the recurrent neural network to perform text feature extraction on the sample text information by using the recurrent neural network, to obtain sample text features corresponding to the at least one sample field.

[0140] Therefore, based on the convolutional neural network and the recurrent neural network, the sample image features and the sample text features corresponding to the at least one sample field in the sample image are accurately extracted, which facilitates accurate prediction of the prediction category corresponding to the at least one sample field based on the sample image features and the sample text features, and thus improves the performance of the target structured information extraction model obtained after training.

[0141] The process of extracting the sample image features and the sample text features corresponding to the at least one sample field in the sample image by using the convolutional neural network and the recurrent neural network can refer to the process of extracting the image features and the text features corresponding to the at least one field in the to-be-processed image based on the convolutional neural network and the recurrent neural network, which will not be described here.

[0142] In step 603, the sample image features and the sample text features corresponding to the at least one sample field are input into a feature fusion layer to perform fusion on the sample image features and the sample text features corresponding to the at least one sample field by using the feature fusion layer, to obtain sample fusion features corresponding to the at least one sample field.

[0143] In an embodiment of the present disclosure, the initial structured information extraction model can further include a feature fusion layer, configured to fuse the image features and the text features corresponding to the at least one field, so that the sample image features and the sample text features corresponding to the at least one sample field can be input into the feature fusion layer, and the sample image features and the sample text features corresponding to the at least one sample field are fused by using the feature fusion layer to obtain sample fusion features corresponding to the at least one sample field.

[0144] In an embodiment of the present disclosure, for each sample field, the fusion of the sample image features and the sample text features can be implemented in multiple ways. As one possible implementation, for each sample field, the sample image features and the sample text features corresponding to the sample field can be spliced, and the spliced features are taken as the sample fusion features corresponding to the sample field. Alternatively, as another possible implementation, for each sample field, the Kronecker product of the sample image features and the sample text features corresponding to the sample field is taken as the sample fusion features of the sample field. By using the feature fusion layer to fuse the sample image features and the sample text features corresponding to the at least one sample field, the sample fusion features corresponding to each sample field in the sample image are obtained, and by using the Kronecker product operation to fuse the sample image features and the sample text features corresponding to the sample field, a better feature fusion effect can be achieved.

[0145] It should be noted that the above feature fusion method is only illustrative and cannot be understood as a limitation on the technical solutions of the present disclosure. In actual applications, those skilled in the art can use any other method to implement the fusion of the sample image features and the sample text features, which is not limited in the present disclosure.

[0146] In step 604, the sample fusion features corresponding to the at least one sample field are taken as the node features of at least one node in the graph network, and input into the graph network, so that the node features of the at least one node are encoded by using the graph network to obtain the predicted categories of the at least one sample field.

[0147] In an embodiment of the present disclosure, the initial structured information extraction model can further include a graph network, configured to determine the categories of the at least one field in the image based on the fusion features of the at least one field in the image, so that the sample fusion features of the at least one sample field in the image are taken as the node features of at least one node in the graph network, and input into the graph network, so that the node features of the at least one node are encoded by using the graph network to obtain the predicted categories of the at least one sample field.

[0148] In an embodiment of the present disclosure, a plurality of categories can be pre-set, and after the sample fusion features corresponding to the at least one sample field in the sample image are obtained, the sample fusion features corresponding to the at least one sample field can be taken as the node features of at least one node in the graph network, and input into the graph network to obtain the probabilities of a plurality of pre-set categories corresponding to the at least one sample field output by the graph network. That is, the graph network can be used to predict the probabilities of a plurality of pre-set categories corresponding to each sample field respectively. Then, for each sample field, the pre-set category with the maximum corresponding probability in the plurality of pre-set categories can be determined as the predicted category corresponding to the sample field.

[0149] In step 605, the difference between the predicted category corresponding to the at least one sample field and the sample category is obtained, and the model parameters of the initial structured information extraction model are adjusted according to the difference to obtain a target structured information extraction model for structured information extraction.

[0150] In an embodiment of the present disclosure, the target structured information extraction model obtained after training is used to perform the following steps: obtaining image features and text features corresponding to at least one field in a to-be-processed image; fusing the image features and the text features corresponding to the at least one field to obtain fusion features corresponding to the at least one field; determining a target category corresponding to the at least one field based on the fusion features corresponding to the at least one field, wherein the target category corresponding to the at least one field is used to organize the text information corresponding to the at least one field to obtain structured information.

[0151] The predicted category corresponding to the at least one sample field, the sample category, or the target category is at least one of a plurality of attribute item categories, at least one of a plurality of attribute value categories, at least one of a plurality of table header categories, or at least one of a plurality of table cell categories. The attribute item category and the attribute value category have a one-to-one correspondence relationship. The table header category and the table cell category have a one-to-one correspondence relationship. By dividing the attribute item category, the attribute value category, the table header category, and the table cell category into more fine-grained categories respectively, the target structured information extraction model obtained by training can perform more fine-grained category prediction, and then realize the extraction of structured information only according to the corresponding relationship between the at least one field in the to-be-processed image, thereby simplifying the extraction process of structured information.

[0152] The initial structured information extraction model comprises a feature extraction network, a feature fusion layer and a graph network. The feature extraction network is used to extract sample image features and sample text features. The feature fusion layer is used to fuse the sample image features and the sample text features. The graph network is used to determine the predicted categories corresponding to the sample fields. Then, the model parameters of the initial structured information extraction model are adjusted according to the difference between the predicted categories corresponding to the sample fields and the sample categories, so as to obtain a target structured information extraction model used for structured information extraction. In this way, the feature extraction network, the feature fusion layer and the graph network do not need to be trained separately, and the trained feature extraction network, the feature fusion layer and the graph network can be obtained through one training, thereby reducing the training cost of the feature extraction network, the feature fusion layer and the graph network.

[0153] In the embodiments of the present disclosure, after obtaining the initial structured information extraction model, the weight matrix of the edges between the nodes in the graph network can be initialized according to the spatial layout information of the nodes in the graph network, such as the distance between two nodes. In this way, when the initial structured information extraction model is trained, the model parameters of the initialized initial structured information extraction model can be adjusted according to the difference between the predicted categories corresponding to the sample fields and the sample categories, so as to improve the model performance of the target structured information extraction model obtained through subsequent training and improve the classification accuracy.

[0154] Specifically, the weight matrix of the edges between at least two nodes in the graph network can be initialized in the following manner: the center coordinates, the width and the height of the text boxes to which the sample fields belong are obtained according to the sample position information of the text boxes to which the sample fields belong in the sample image; the initial weight parameters corresponding to the edges between the nodes corresponding to the sample fields in the graph network are determined according to the center coordinates, the width and the height of the text boxes to which the sample fields belong; and the weight matrix of the edges between the nodes in the graph network is initialized according to the initial weight parameters corresponding to the edges between the nodes.

[0155] The sample position information of the text boxes to which the sample fields belong can be the coordinates of the four corner points of the text boxes to which the sample fields belong.

[0156] The center coordinates are the coordinates of the center points of the text boxes to which the sample fields belong.

[0157] The sample image can be any sample image containing multiple sample fields in the multiple sample images included in the training data. That is, in the embodiments of the present disclosure, the weight matrix of the edges between the nodes in the graph network can be initialized according to the sample position information of the text boxes to which the multiple sample fields belong in any sample image containing multiple sample fields.

[0158] It can be understood that for each sample field in a sample image, the center coordinates, the width and the height of the text box to which the sample field belongs can be determined according to the sample position information of the text box to which the sample field belongs, the initial weight parameters corresponding to the edges between the nodes corresponding to the plurality of sample fields in the graph network can be determined according to the center coordinates, the width and the height of the text boxes to which the plurality of sample fields belong, and then the weight matrix of the edges between the plurality of nodes in the graph network can be initialized according to the initial weight parameters corresponding to the edges between the plurality of nodes.

[0159] In an embodiment of the present disclosure, the initial weight parameters corresponding to the edges between the nodes corresponding to the plurality of sample fields in the graph network can be determined according to the center coordinates, the width and the height of the text boxes to which the plurality of sample fields belong in the following manner:

[0160] In the case where the first node points to the second node among the plurality of nodes, the first coordinate difference value corresponding to the first coordinate axis and the second coordinate difference value corresponding to the second coordinate axis between the second node and the first node can be determined according to the center coordinates of the text boxes to which the sample fields corresponding to the first node and the second node respectively belong, the first product can be obtained by multiplying the first coordinate difference value and the second coordinate difference value, the second product can be obtained by multiplying the width and the height of the text box to which the sample field corresponding to the first node belongs, and the ratio of the first product to the second product can be determined as the initial weight parameter of the edge between the first node and the second node.

[0161] The first coordinate axis and the second coordinate axis can be the x-axis and the y-axis respectively.

[0162] The first coordinate difference value is the difference between the coordinate value of the center coordinates of the text box to which the sample field corresponding to the second node belongs on the first coordinate axis and the coordinate value of the center coordinates of the text box to which the sample field corresponding to the first node belongs on the first coordinate axis.

[0163] The second coordinate difference value is the difference between the coordinate value of the center coordinates of the text box to which the sample field corresponding to the second node belongs on the second coordinate axis and the coordinate value of the center coordinates of the text box to which the sample field corresponding to the first node belongs on the second coordinate axis.

[0164] That is, in the case where the first node points to the second node, the initial weight parameters corresponding to the edges between the nodes corresponding to the plurality of sample fields in the graph network can be determined according to the center coordinates, the width and the height of the text box to which the sample field corresponding to the first node belongs, and the center coordinates of the text box to which the sample field corresponding to the second node belongs by using the following formula (2).

[0165]

[0166] wherein R ijdenotes an initial weight parameter of an edge between the first node and the second node, (X i , Y i ) denotes a center coordinate of a text box to which a sample field corresponding to the first node belongs, W i denotes a width of the text box to which the sample field corresponding to the first node belongs, H i denotes a height of the text box to which the sample field corresponding to the first node belongs, (X j , Y j ) denotes a center coordinate of a text box to which a sample field corresponding to the second node belongs.

[0167] In the foregoing manner, in the case where the first node points to the second node, the initial weight parameter of the edge between the first node and the second node is calculated based on the center coordinate, the width and the height of the text box to which the sample field corresponding to the first node belongs, and the center coordinate of the text box to which the sample field corresponding to the second node belongs. In this way, the initial weight parameters of the edges between the nodes in the graph network can be calculated, and then the weight matrix of the edges between the nodes in the graph network can be initialized according to the initial weight parameters of the edges between the nodes, thereby improving the model performance of the target structured information extraction model obtained through subsequent training.

[0168] The structured information extraction apparatus provided by the present disclosure will be described below. Figure 7

[0169] Figure 7 is a structural schematic diagram of a structured information extraction apparatus according to the sixth embodiment of the present disclosure.

[0170] As shown in Figure 7 , the structured information extraction apparatus 700 provided by the present disclosure includes a first acquisition module 701, a fusion module 702, a first determination module 703 and an organization module 704.

[0171] The first acquisition module 701 is configured to acquire image features, text features and text information corresponding to at least one field in a to-be-processed image.

[0172] The fusion module 702 is configured to fuse the image features and the text features corresponding to the at least one field to obtain fusion features corresponding to the at least one field.

[0173] The first determination module 703 is configured to determine target categories corresponding to the at least one field based on the fusion features corresponding to the at least one field.

[0174] The organization module 704 is configured to organize the text information corresponding to the at least one field according to a corresponding relationship between the target categories corresponding to the at least one field to obtain structured information. ​

[0175] It should be noted that the structured information extraction apparatus 700 provided in this embodiment can execute the structured information extraction method of the foregoing embodiments. The structured information extraction apparatus 700 can be implemented by software and / or hardware, and can be configured in an electronic device, which can include but is not limited to a terminal device, a server, and the like, and the electronic device is not limited in this embodiment.

[0176] It should be noted that the foregoing description of the embodiments of the structured information extraction method also applies to the structured information extraction apparatus provided in this disclosure, which will not be described here.

[0177] The structured information extraction apparatus provided in the embodiments of this disclosure obtains image features, text features, and text information corresponding to at least one field in a to-be-processed image, fuses the image features and the text features corresponding to the at least one field to obtain fusion features corresponding to the at least one field, determines target categories corresponding to the at least one field based on the fusion features corresponding to the at least one field, organizes the text information corresponding to the at least one field according to a corresponding relationship between the target categories corresponding to the at least one field to obtain structured information. Thus, only the target categories corresponding to the at least one field in the to-be-processed image need to be determined, and the extraction of the structured information can be implemented according to the corresponding relationship between the target categories corresponding to the at least one field, so that the structured information extraction problem is simplified to a classification problem, the process of structured information extraction is simplified, and the accuracy of structured information extraction is improved because the text features and the image features of the at least one field in the to-be-processed image are combined for structured information extraction.

[0178] The structured information extraction apparatus provided in this disclosure will be further described below. Figure 8

[0179] Figure 8 FIG. 8 is a structural schematic diagram of a structured information extraction apparatus according to a seventh embodiment of the present disclosure.

[0180] As shown in FIG. 8, the structured information extraction apparatus 800 can specifically include a first obtaining module 801, a fusion module 802, a first determining module 803, and an organizing module 804. Among them, Figure 8 The first obtaining module 801, the fusion module 802, the first determining module 803, and the organizing module 804 in the structured information extraction apparatus 800 have the same functions and structures as the first obtaining module 701, the fusion module 702, the first determining module 703, and the organizing module 704 in the structured information extraction apparatus 700. Figure 8 Figure 7

[0181] ​​​In the embodiment of the present disclosure, the target category corresponding to the at least one field is at least one of the plurality of attribute item categories or at least one of the plurality of attribute value categories; and the attribute item category and the attribute value category have a one-to-one correspondence relationship;

[0182] The organization module 804 includes:

[0183] The first query unit 8041 is configured to query, according to a first field of a first attribute item category as the target category, a second field of a first attribute value category having a corresponding relationship as the target category;

[0184] The first acquisition unit 8042 is configured to acquire first text information corresponding to the first field and second text information corresponding to the second field;

[0185] The first generation unit 8043 is configured to generate a piece of structured information according to the first text information and the second text information.

[0186] In the embodiment of the present disclosure, the target category corresponding to the at least one field is at least one of the plurality of table header categories or at least one of the plurality of table cell categories; and the table header category and the table cell category have a one-to-one correspondence relationship;

[0187] The organization module 804 includes:

[0188] The second query unit is configured to query, according to a third field of a first table header category as the target category, a fourth field of a first table cell category having a corresponding relationship as the target category;

[0189] The second acquisition unit is configured to acquire third text information corresponding to the third field and fourth text information corresponding to the fourth field;

[0190] The second generation unit is configured to generate a piece of structured information according to the third text information and the fourth text information.

[0191] In the embodiment of the present disclosure, the number of the second fields is a plurality, and the first generation unit 8043 includes:

[0192] The merging subunit is configured to merge the second text information corresponding to the plurality of second fields to obtain merged information;

[0193] The generation subunit is configured to generate a piece of structured information according to the first text information and the merged information.

[0194] In the embodiment of the present disclosure, the first determination module 803 includes:

[0195] The third acquisition unit 8031 ​​is used to input the fusion feature corresponding to at least one field into the graph network as the node feature of at least one node in the graph network, so as to obtain the probability of at least one field corresponding to multiple preset categories in the output of the graph network; wherein, at least one field corresponds to a node in the graph network.

[0196] The first determining unit 8032 is used to determine the target category corresponding to at least one field from multiple preset categories based on probability.

[0197] In embodiments of this disclosure, the graph network is included in the structured information extraction model, which further includes a feature extraction network. The first acquisition module 801 includes:

[0198] The fourth acquisition unit 8011 is used to acquire text information corresponding to at least one field in the image to be processed obtained by optical character recognition of the image to be processed, and the position information of the text box to which at least one field belongs;

[0199] The first feature extraction unit 8012 is used to input the image to be processed, the text information corresponding to at least one field in the image to be processed, and the position information of the text box to which at least one field belongs into the feature extraction network, so as to perform feature extraction using the feature extraction network and obtain the image features and text features corresponding to at least one field in the image to be processed.

[0200] In embodiments of this disclosure, the feature extraction network includes a convolutional neural network and a recurrent neural network;

[0201] The first feature extraction unit 8012 includes:

[0202] The first extraction subunit is used to input the image to be processed into the convolutional neural network so that the convolutional neural network can extract image features from the image to be processed to obtain a feature map.

[0203] The first cropping subunit is used to crop the corresponding region in the feature map based on the position information of the text box to which at least one field belongs, so as to obtain the image features corresponding to at least one field.

[0204] The second extraction subunit is used to input the text information corresponding to at least one field into a recurrent neural network, so as to use the recurrent neural network to extract text features from the text information and obtain the text features corresponding to at least one field.

[0205] In embodiments of this disclosure, the structured information extraction model further includes a pooling layer, and the first feature extraction unit 8012 further includes:

[0206] The pooling processing subunit is used to input image features into the pooling layer so that the image features are pooled by the pooling layer so that the dimensions of the image features corresponding to at least one field are the same.

[0207] In embodiments of this disclosure, the structured information extraction model further includes a feature fusion layer, and the fusion module 802 includes:

[0208] The first processing unit 8021 is used to input image features and text features corresponding to at least one field into the feature fusion layer, so as to use the feature fusion layer to concatenate the image features and text features corresponding to at least one field, or to perform Kronecker product operation on the image features and text features corresponding to at least one field to obtain the fused features corresponding to at least one field.

[0209] It should be noted that the foregoing description of the embodiments of the structured information extraction method also applies to the structured information extraction apparatus provided in this disclosure, and will not be repeated here.

[0210] The structured information extraction apparatus provided in this embodiment acquires image features, text features, and text information corresponding to at least one field in an image to be processed. It then fuses the image features and text features corresponding to at least one field to obtain a fused feature corresponding to at least one field. Based on the fused feature corresponding to at least one field, it determines the target category corresponding to at least one field. According to the correspondence between the target categories corresponding to at least one field, it organizes the text information corresponding to at least one field to obtain structured information. Therefore, by simply determining the target category corresponding to at least one field in the image to be processed, structured information can be extracted based on the correspondence between the target categories corresponding to at least one field. This simplifies the structured information extraction problem to a classification problem, simplifying the structured information extraction process. Furthermore, by combining the text features and image features of at least one field in the image to be processed for structured information extraction, the accuracy of structured information extraction is improved.

[0211] According to embodiments of this disclosure, a model training apparatus for structured information extraction is also provided.

[0212] The following is combined Figure 9 The present disclosure describes the model training apparatus for structured information extraction provided in this disclosure.

[0213] Figure 9 This is a schematic diagram of a model training apparatus for structured information extraction according to the eighth embodiment of the present disclosure.

[0214] like Figure 9As shown, the model training device 900 for structured information extraction provided in this disclosure includes: a second acquisition module 901, a second determination module 902, and a third acquisition module 903.

[0215] The second acquisition module 901 is used to acquire training data and an initial structured information extraction model. The training data includes multiple sample images, sample text information corresponding to at least one sample field in the multiple sample images, and sample position information of the text box to which at least one sample field belongs. At least one sample field is labeled with the corresponding sample category.

[0216] The second determining module 902 is used to input at least one sample image, sample text information corresponding to at least one sample field in the sample image, and sample position information of the text box to which at least one sample field belongs into the initial structured information extraction model, so as to determine the predicted category corresponding to at least one sample field using the initial structured information extraction model.

[0217] The third acquisition module 903 is used to acquire the difference between the predicted category and the sample category corresponding to at least one sample field, so as to adjust the model parameters of the initial structured information extraction model according to the difference, and obtain the target structured information extraction model for structured information extraction.

[0218] The target structured information extraction model is used to perform the following steps:

[0219] Obtain image features and text features corresponding to at least one field in the image to be processed;

[0220] The image features and text features corresponding to at least one field are fused to obtain the fused features corresponding to at least one field.

[0221] Based on the fusion features corresponding to at least one field, the target category corresponding to at least one field is determined. The correspondence between the target categories corresponding to at least one field is used to organize the text information corresponding to at least one field to obtain structured information.

[0222] It should be noted that the model training device 900 for structured information extraction provided in this embodiment, hereinafter referred to as the model training device, can execute the model training method for structured information extraction in the aforementioned embodiments. The model training device can be implemented by software and / or hardware, and can be configured on an electronic device, which may include, but is not limited to, terminal devices, servers, etc. This embodiment does not specifically limit the electronic device.

[0223] It should be noted that the foregoing description of the model training method for structured information extraction also applies to the model training apparatus for structured information extraction provided in this disclosure, and will not be repeated here.

[0224] The model training apparatus for structured information extraction provided in this embodiment acquires training data and an initial structured information extraction model. It inputs at least one sample image, sample text information corresponding to at least one sample field in the sample image, and sample position information of the text box to which at least one sample field belongs into the initial structured information extraction model. The initial structured information extraction model is used to determine the predicted category corresponding to at least one sample field. The difference between the predicted category corresponding to at least one sample field and the sample category is obtained. The model parameters of the initial structured information extraction model are adjusted based on the difference to obtain a target structured information extraction model for structured information extraction. This realizes model training based on training data to obtain a target structured information extraction model for structured information extraction. Using the trained target structured information extraction model for structured information extraction simplifies the structured information extraction problem into a classification problem, simplifying the structured information extraction process. Furthermore, by combining text features and image features of at least one field in the image to be processed for structured information extraction, the accuracy of structured information extraction is improved.

[0225] The following is combined Figure 10 The present disclosure provides a further explanation of the model training apparatus for structured information extraction.

[0226] Figure 10 This is a schematic diagram of a model training apparatus for structured information extraction according to the ninth embodiment of this disclosure.

[0227] like Figure 10 As shown, the model training device 1000 for structured information extraction may specifically include: a second acquisition module 1001, a second determination module 1002, and a third acquisition module 1003. Among them, Figure 10 The second acquisition module 1001, the second determination module 1002, and the third acquisition module 1003 are in conjunction with Figure 9 The second acquisition module 901, the second determination module 902, and the third acquisition module 903 have the same function and structure.

[0228] In the embodiments of this disclosure, the prediction category, sample category, or target category corresponding to at least one sample field is at least one of at least one attribute value category among multiple attribute item categories, or at least one of at least one table cell category among multiple header categories; there is a one-to-one correspondence between attribute item categories and attribute value categories; there is a one-to-one correspondence between header categories and table cell categories.

[0229] In embodiments of this disclosure, the initial structured information extraction model includes a feature extraction network, a feature fusion layer, and a graph network;

[0230] The second determining module 1002 includes:

[0231] The second feature extraction unit is used to input the sample image, the sample text information corresponding to at least one sample field in the sample image, and the sample position information of the text box to which at least one sample field belongs into the feature extraction network, so as to use the feature extraction network to perform feature extraction and obtain the sample image features and sample text features corresponding to at least one sample field in the sample image.

[0232] The second fusion unit is used to input the sample image features and sample text features corresponding to at least one sample field into the feature fusion layer, so as to use the feature fusion layer to fuse the sample image features and sample text features corresponding to at least one sample field to obtain the sample fusion features corresponding to at least one sample field.

[0233] The feature encoding unit is used to input the sample fusion features corresponding to at least one sample field into the graph network as the node features of at least one node in the graph network, so as to use the graph network to perform feature encoding on the node features of at least one node to obtain the predicted category corresponding to at least one sample field.

[0234] In embodiments of this disclosure, the model training apparatus for structured information extraction further includes:

[0235] The fourth acquisition module 1004 is used to acquire the center coordinates, width, and height of the text boxes to which multiple sample fields belong, based on the sample position information of the text boxes to which multiple sample fields belong in the sample image;

[0236] The third determining module 1005 is used to determine the initial weight parameters of the edges between the nodes corresponding to the multiple sample fields in the graph network based on the center coordinates, width and height of the text boxes to which the multiple sample fields belong.

[0237] The initialization module 1006 is used to initialize the weight matrix of the edges between multiple nodes in the graph network according to the initial weight parameters corresponding to the edges between multiple nodes.

[0238] In embodiments of this disclosure, the third determining module 1005 includes:

[0239] The second determining unit is used to determine, in the case where the first node points to the second node among multiple nodes, the first coordinate difference between the second node and the first node and the second coordinate difference between the second node and the first node based on the center coordinates of the text boxes to which the sample fields to which the first node and the second node belong, respectively.

[0240] The second processing unit is used to product the first coordinate difference and the second coordinate difference to obtain the first product;

[0241] The third processing unit is used to multiply the width and height of the text box to which the sample field corresponding to the first node belongs, and obtain the second product;

[0242] The third determining unit is used to determine the ratio of the first product to the second product as the initial weight parameter of the edge between the first node and the second node.

[0243] In embodiments of this disclosure, the second fusion unit includes:

[0244] The fusion subunit is used to use the feature fusion layer to concatenate the sample image features and sample text features corresponding to at least one sample field, or to perform Kronecker product operation on the image features and text features corresponding to at least one sample field to obtain the sample fusion feature corresponding to at least one sample field.

[0245] In embodiments of this disclosure, the feature extraction network includes a convolutional neural network and a trained neural network;

[0246] The second feature extraction unit includes:

[0247] The third extraction subunit is used to input the sample image into the convolutional neural network so that the convolutional neural network can extract image features from the sample image to obtain the sample feature map;

[0248] The second extraction subunit is used to extract the corresponding region in the sample feature map based on the sample position information of the text box to which at least one sample field belongs, so as to obtain the sample image features corresponding to at least one sample field.

[0249] The fourth extraction subunit is used to input the sample text information corresponding to at least one sample field in the sample image into the recurrent neural network, so as to use the recurrent neural network to extract the text features of the sample text information and obtain the sample text features corresponding to at least one sample field.

[0250] The model training apparatus for structured information extraction provided in this embodiment acquires training data and an initial structured information extraction model. It inputs at least one sample image, sample text information corresponding to at least one sample field in the sample image, and sample position information of the text box to which at least one sample field belongs into the initial structured information extraction model. The initial structured information extraction model is used to determine the predicted category corresponding to at least one sample field. The difference between the predicted category corresponding to at least one sample field and the sample category is obtained. The model parameters of the initial structured information extraction model are adjusted based on the difference to obtain a target structured information extraction model for structured information extraction. This realizes model training based on training data to obtain a target structured information extraction model for structured information extraction. Using the trained target structured information extraction model for structured information extraction simplifies the structured information extraction problem into a classification problem, simplifying the structured information extraction process. Furthermore, by combining text features and image features of at least one field in the image to be processed for structured information extraction, the accuracy of structured information extraction is improved.

[0251] Based on the above embodiments, this disclosure also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to execute the structured information extraction method of this disclosure, or to execute the model training method for structured information extraction of this disclosure.

[0252] Based on the above embodiments, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, the computer instructions being used to cause the computer to execute the structured information extraction method disclosed in the embodiments of this disclosure, or to execute the model training method for structured information extraction disclosed in the embodiments of this disclosure.

[0253] Based on the above embodiments, this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the structured information extraction method of this disclosure, or implements the steps of the model training method for structured information extraction of this disclosure.

[0254] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0255] Figure 11A schematic block diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0256] like Figure 11 As shown, the electronic device 1100 may include a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. The RAM 1103 may also store various programs and data required for the operation of the device 1100. The computing unit 1101, ROM 1102, and RAM 1103 are interconnected via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0257] Multiple components in device 1100 are connected to I / O interface 1105, including: input unit 1106, such as keyboard, mouse, etc.; output unit 1107, such as various types of monitors, speakers, etc.; storage unit 1108, such as disk, optical disk, etc.; and communication unit 1109, such as network card, modem, wireless transceiver, etc. Communication unit 1109 allows device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0258] The computing unit 1101 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above, such as structured information extraction methods or model training methods for structured information extraction. For example, in some embodiments, the structured information extraction method or the model training method for structured information extraction can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by computing unit 1101, one or more steps of the structured information extraction method or model training method for structured information extraction described above can be performed. Alternatively, in other embodiments, computing unit 1101 can be configured to perform the structured information extraction method or model training method for structured information extraction by any other suitable means (e.g., by means of firmware).

[0259] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0260] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0261] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0262] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0263] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0264] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. A server can be a cloud server, a server in a distributed system, or a server incorporating blockchain technology.

[0265] It's important to note that artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0266] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0267] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for extracting structured information, comprising: Obtain image features, text features, and text information corresponding to at least one field in the image to be processed; The image features and text features corresponding to at least one of the fields are fused to obtain a fused feature corresponding to at least one of the fields; Based on the fusion features corresponding to at least one of the fields, a target category corresponding to at least one of the fields is determined from multiple preset categories; the preset categories include at least two of the following: attribute item category, attribute value category, table header category, and table cell category; There is a one-to-one correspondence between the attribute item category and the attribute value category, and a one-to-one correspondence between the table header category and the table cell category; Based on the correspondence between the target categories corresponding to at least one of the fields, the text information corresponding to at least one of the fields is organized to obtain structured information; Where, in the case that the target category corresponding to at least one of the multiple attribute item categories, the step of organizing the text information corresponding to at least one of the fields according to the correspondence between the target categories corresponding to at least one of the fields to obtain structured information includes: Based on the first field of the target category being the first attribute item category, query the second field of the first attribute value category that corresponds to the target category from each field in the image to be processed; Obtain the first text information corresponding to the first field, and the second text information corresponding to the second field; A structured message is generated based on the first text information and the second text information.

2. The method according to claim 1, wherein the target category corresponding to at least one of the fields is at least one of a plurality of header categories; in, The step of organizing the text information corresponding to at least one of the fields according to the correspondence between the target categories corresponding to at least one of the fields to obtain structured information includes: Based on the third field of the target category being the first header category, query the fourth field of the first table cell category that corresponds to the target category; Obtain the third text information corresponding to the third field, and the fourth text information corresponding to the fourth field; A structured message is generated based on the third and fourth text information.

3. The method according to claim 1, wherein, The second field has multiple elements. The step of generating a structured piece of information based on the first text information and the second text information includes: Merge the second text information corresponding to multiple second fields to obtain merged information; Based on the first text information and the merged information, a structured information is generated.

4. The method according to any one of claims 1-3, wherein, The step of determining the target category corresponding to at least one of the fields based on the fusion features corresponding to at least one of the fields includes: The fusion feature corresponding to at least one of the fields is used as the node feature of at least one node in the graph network and input into the graph network to obtain the probability that at least one of the fields output by the graph network corresponds to multiple preset categories; wherein, at least one of the fields corresponds to a node in the graph network; Based on the probability, at least one target category corresponding to the field is determined from the plurality of preset categories.

5. The method according to claim 4, wherein the graph network is included in a structured information extraction model, the structured information extraction model further includes a feature extraction network, and the step of obtaining image features, text features, and text information corresponding to at least one field in the image to be processed includes: Obtain text information corresponding to at least one field in the image to be processed, obtained by optical character recognition of the image to be processed, and position information of the text box to which at least one field belongs; The image to be processed, the text information corresponding to at least one of the fields in the image to be processed, and the position information of the text box to which at least one of the fields belongs are input into the feature extraction network to perform feature extraction using the feature extraction network, thereby obtaining image features and text features corresponding to at least one of the fields in the image to be processed.

6. The method according to claim 5, wherein the feature extraction network comprises a convolutional neural network and a recurrent neural network; in, The step of inputting the image to be processed, the text information corresponding to at least one of the fields in the image to be processed, and the position information of the text box to which at least one of the fields belongs into the feature extraction network, so as to perform feature extraction using the feature extraction network to obtain image features and text features corresponding to at least one of the fields in the image to be processed, includes: The image to be processed is input into the convolutional neural network to extract image features from the image and obtain a feature map. Based on the position information of the text box to which at least one of the fields belongs, the corresponding region in the feature map is extracted to obtain the image features corresponding to at least one of the fields; The text information corresponding to at least one of the fields is input into the recurrent neural network to extract text features from the text information using the recurrent neural network, thereby obtaining text features corresponding to at least one of the fields.

7. The method according to claim 6, wherein, The structured information extraction model further includes a pooling layer. After obtaining the image features corresponding to at least one field by extracting the corresponding region from the feature map based on the position information of the text box to which the at least one field belongs, the model further includes: The image features are input into the pooling layer to perform pooling processing on the image features, so that the image features corresponding to at least one of the fields have the same dimension.

8. The method according to claim 5, wherein, The structured information extraction model further includes a feature fusion layer, wherein fusing the image features and text features corresponding to at least one of the fields to obtain fused features corresponding to at least one of the fields includes: The image features and text features corresponding to at least one of the fields are input into the feature fusion layer, so that the feature fusion layer can be used to concatenate the image features and text features corresponding to at least one of the fields, or to perform Kronecker product operation on the image features and text features corresponding to at least one of the fields to obtain the fused features corresponding to at least one of the fields.

9. A model training method for structured information extraction, comprising: Acquire training data and an initial structured information extraction model; wherein, the training data includes multiple sample images, sample text information corresponding to at least one sample field in the multiple sample images, and sample position information of the text box to which at least one sample field belongs, and at least one sample field is labeled with a corresponding sample category, and the initial structured information extraction model includes a feature extraction network, a feature fusion layer and a graph network; Input at least one sample image, sample text information corresponding to at least one sample field in the sample image, and sample position information of the text box to which at least one sample field belongs into the initial structured information extraction model, so as to determine the predicted category corresponding to at least one sample field using the initial structured information extraction model; Obtain the difference between the predicted category and the sample category corresponding to at least one of the sample fields, and adjust the model parameters of the initial structured information extraction model according to the difference to obtain a target structured information extraction model for structured information extraction. Wherein, at least one of the sample fields corresponds to a prediction category or sample category, which is at least one of at least one attribute value category among multiple attribute item categories, or at least one of at least one table cell category among multiple header categories; there is a one-to-one correspondence between the attribute item categories and the attribute value categories; there is a one-to-one correspondence between the header categories and the table cell categories; The method further includes: Based on the sample position information of the text boxes to which the sample fields belong in the sample image, obtain the center coordinates, width, and height of the text boxes to which the sample fields belong; Based on the center coordinates, width, and height of the text boxes to which the sample fields belong, determine the initial weight parameters of the edges between the nodes corresponding to the sample fields in the graph network; Specifically, based on the center coordinates, width, and height of the text boxes to which the multiple sample fields belong, the initial weight parameters corresponding to the edges between the nodes corresponding to the multiple sample fields in the graph network are determined, including: In the case where the first node points to the second node among multiple nodes, the first coordinate difference between the second node and the first node on the first coordinate axis and the second coordinate difference on the second coordinate axis are determined based on the center coordinates of the text boxes to which the sample fields corresponding to the first node and the second node belong, respectively. The first coordinate difference and the second coordinate difference are multiplied to obtain the first product; The second product is obtained by multiplying the width and height of the text box to which the sample field corresponding to the first node belongs; The ratio of the first product to the second product is determined as the initial weight parameter of the edge between the first node and the second node.

10. The method according to claim 9, wherein inputting at least one sample image, sample text information corresponding to at least one sample field in the sample image, and sample position information of the text box to which at least one sample field belongs, into the initial structured information extraction model to determine the predicted category corresponding to at least one sample field using the initial structured information extraction model, comprises: The sample image, sample text information corresponding to at least one of the sample fields in the sample image, and sample position information of the text box to which at least one of the sample fields belongs are input into the feature extraction network to perform feature extraction using the feature extraction network, thereby obtaining sample image features and sample text features corresponding to at least one of the sample fields in the sample image. The sample image features and sample text features corresponding to at least one of the sample fields are input into the feature fusion layer, so that the sample image features and sample text features corresponding to at least one of the sample fields are fused using the feature fusion layer to obtain sample fusion features corresponding to at least one of the sample fields; The sample fusion features corresponding to at least one of the sample fields are used as node features of at least one node in the graph network and input into the graph network to perform feature encoding on the node features of at least one of the nodes to obtain the predicted category corresponding to at least one of the sample fields.

11. The method according to claim 10, wherein, The step of inputting the sample image features and sample text features corresponding to at least one of the sample fields into the feature fusion layer, and using the feature fusion layer to fuse the sample image features and sample text features corresponding to at least one of the sample fields to obtain sample fusion features corresponding to at least one of the sample fields, includes: The feature fusion layer is used to concatenate the sample image features and the sample text features corresponding to at least one of the sample fields, or to perform a Kronecker product operation on the image features and the text features corresponding to at least one of the sample fields to obtain the sample fusion features corresponding to at least one of the sample fields.

12. The method according to claim 10, wherein the feature extraction network comprises a convolutional neural network and a recurrent neural network; in, The step of inputting the sample image, sample text information corresponding to at least one sample field in the sample image, and sample position information of the text box to which at least one sample field belongs into the feature extraction network, so as to perform feature extraction using the feature extraction network to obtain sample image features and sample text features corresponding to at least one sample field in the sample image, includes: The sample image is input into the convolutional neural network to extract image features from the sample image using the convolutional neural network, thereby obtaining a sample feature map; Based on the sample position information of the text box to which at least one of the sample fields belongs, the corresponding region in the sample feature map is extracted to obtain the sample image features corresponding to at least one of the sample fields; The sample text information corresponding to at least one of the sample fields in the sample image is input into the recurrent neural network to extract text features from the sample text information using the recurrent neural network, thereby obtaining sample text features corresponding to at least one of the sample fields.

13. A structured information extraction device, comprising: The first acquisition module is used to acquire image features, text features, and text information corresponding to at least one field in the image to be processed; A fusion module is used to fuse the image features and text features corresponding to at least one of the fields to obtain a fused feature corresponding to at least one of the fields; The first determining module is used to determine a target category corresponding to at least one of the fields from multiple preset categories based on the fusion features corresponding to at least one of the fields. The preset categories include at least two of the following: attribute item category, attribute value category, table header category, and table cell category. There is a one-to-one correspondence between the attribute item category and the attribute value category, and a one-to-one correspondence between the table header category and the table cell category; An organization module is used to organize the text information corresponding to at least one of the fields according to the correspondence between the target categories corresponding to at least one of the fields, so as to obtain structured information; Wherein, if the target category corresponding to at least one of the said fields is at least one of multiple attribute item categories, the organization module is specifically used for: Based on the first field of the target category being the first attribute item category, query the second field of the first attribute value category that corresponds to the target category from each field in the image to be processed; Obtain the first text information corresponding to the first field, and the second text information corresponding to the second field; A structured message is generated based on the first text information and the second text information.

14. A model training device for structured information extraction, comprising: The second acquisition module is used to acquire training data and an initial structured information extraction model; wherein, the training data includes multiple sample images, sample text information corresponding to at least one sample field in the multiple sample images, and sample position information of the text box to which at least one sample field belongs, and at least one sample field is labeled with a corresponding sample category, and the initial structured information extraction model includes a feature extraction network, a feature fusion layer and a graph network. The second determining module is used to input at least one of the sample images, sample text information corresponding to at least one of the sample fields in the sample images, and sample position information of the text box to which at least one of the sample fields belongs into the initial structured information extraction model, so as to determine the predicted category corresponding to at least one of the sample fields using the initial structured information extraction model. The third acquisition module is used to acquire the difference between the predicted category and the sample category corresponding to at least one of the sample fields, so as to adjust the model parameters of the initial structured information extraction model according to the difference, and obtain the target structured information extraction model for structured information extraction. Wherein, at least one of the sample fields corresponds to a prediction category or sample category, which is at least one of at least one attribute value category among multiple attribute item categories, or at least one of at least one table cell category among multiple header categories; there is a one-to-one correspondence between the attribute item categories and the attribute value categories; there is a one-to-one correspondence between the header categories and the table cell categories; The device is also used for: Based on the sample position information of the text boxes to which the sample fields belong in the sample image, obtain the center coordinates, width, and height of the text boxes to which the sample fields belong; Based on the center coordinates, width, and height of the text boxes to which the sample fields belong, determine the initial weight parameters of the edges between the nodes corresponding to the sample fields in the graph network; Specifically, based on the center coordinates, width, and height of the text boxes to which the multiple sample fields belong, the initial weight parameters corresponding to the edges between the nodes corresponding to the multiple sample fields in the graph network are determined, including: In the case where the first node points to the second node among multiple nodes, the first coordinate difference between the second node and the first node on the first coordinate axis and the second coordinate difference on the second coordinate axis are determined based on the center coordinates of the text boxes to which the sample fields corresponding to the first node and the second node belong, respectively. The first coordinate difference and the second coordinate difference are multiplied to obtain the first product; The second product is obtained by multiplying the width and height of the text box to which the sample field corresponding to the first node belongs; The ratio of the first product to the second product is determined as the initial weight parameter of the edge between the first node and the second node.

15. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1-8, or the method of any one of claims 9-12.

16. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-8, or the method of any one of claims 9-12.

17. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-8, or the steps of the method according to any one of claims 9-12.

Images