Method and apparatus for extracting information

Abstract

The present application discloses a method and device for extracting information, which relates to the field of image recognition and structured information processing technology. The method comprises: obtaining a picture to be identified; obtaining elements in the picture to be identified, wherein the elements include multiple text entities in the picture to be identified; determining the entity label of each text entity in the multiple text entities; and extracting text information in the picture to be identified according to the entity label of each text entity. The method can improve the efficiency and accuracy of extracting text information in pictures.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, specifically to the field of image recognition and structured information processing technology, and especially to a method and device for extracting information. Background Art

[0002] Structural extraction of text information in images helps to improve the efficient use and effective maintenance of information. Existing methods for structural extraction of text information in images include: using OCR (optical character recognition) technology to identify the content of preset fields in the image and splicing them according to certain rules; or using deep learning technology to identify text entities in the image and then splice the entities.

[0003] However, existing methods for extracting structured text information from images have the problems of low efficiency and inaccuracy in extracting information. Summary of the invention

[0004] The present disclosure provides a method, an apparatus, an electronic device, and a computer-readable storage medium for extracting information.

[0005] According to a first aspect, a method for extracting information is provided, the method comprising: obtaining a picture to be identified; obtaining elements in the picture to be identified, wherein the elements include multiple text entities in the picture to be identified; determining an entity label of each text entity in the multiple text entities; and extracting text information in the picture to be identified based on the entity label of each text entity.

[0006] According to a second aspect, a device for extracting information is provided, the device comprising: an acquisition unit, configured to acquire a picture to be identified; a recognition unit, configured to acquire elements in the picture to be identified, wherein the elements include multiple text entities in the picture to be identified; a marking unit, configured to determine an entity label of each text entity in the multiple text entities; and an extraction unit, configured to extract text information in the picture to be identified based on the entity label of each text entity.

[0007] According to a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method for extracting information as provided in the first aspect.

[0008] According to a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method for extracting information provided in the first aspect is implemented.

[0009] The method and device for extracting information provided by the present disclosure obtain a picture to be identified; obtain elements in the picture to be identified, wherein the elements include multiple text entities in the picture to be identified; determine the entity label of each text entity in the multiple text entities; extract text information in the picture to be identified based on the entity label of each text entity, which can improve the efficiency and accuracy of extracting text information in the picture.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.

[0012] Figure 1 is an exemplary system architecture diagram to which embodiments of the present application may be applied;

[0013] Figure 2 is a flow chart of an embodiment of a method for extracting information according to the present application;

[0014] Figure 3 is a flow chart of another embodiment of a method for extracting information according to the present application;

[0015] Figure 4 is a schematic diagram of a region of a preset position in one embodiment of a method for extracting information according to the present application;

[0016] Figure 5 is a flow chart of another embodiment of a method for extracting information according to the present application;

[0017] Figure 6 is a schematic diagram of calculating the coverage between detection frames in one embodiment of the method for extracting information according to the present application;

[0018] Figure 7 It is a schematic diagram of BIO sequence annotation in an application scenario of the method for extracting information according to the present application;

[0019] Figure 8 is a schematic diagram of regions in the same row or column in an application scenario of the method for extracting information according to the present application;

[0020] Fig. 9 It is a schematic diagram of the positional relationship between entities of a key-value pair relationship in an application scenario of the method for extracting information according to the present application;

[0021] Fig.10 is a schematic structural diagram of an embodiment of a device for extracting information according to the present application;

[0022] Fig.11 It is a block diagram of an electronic device used to implement the method for extracting information in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0024] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the method for extracting information or the apparatus for extracting information of the present application can be applied.

[0025] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0026] The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. The terminal devices 101, 102, 103 can be user terminal devices, on which various client applications can be installed, such as data reading applications, image applications, video applications, search applications, financial applications, etc.

[0027] The terminal devices 101, 102, 103 may be various electronic devices having a display screen and supporting receiving server messages, including but not limited to smart phones, tablet computers, e-book readers, electronic players, laptop computers, desktop computers, and the like.

[0028] Terminal devices 101, 102, 103 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices. When terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules (for example, multiple software modules for providing distributed services), or they can be implemented as a single software or software module. No specific limitation is made here.

[0029] The server 105 can obtain a picture to be identified; obtain elements in the picture to be identified, wherein the elements include multiple text entities in the picture to be identified; determine an entity tag of each text entity in the multiple text entities; and extract text information in the picture to be identified based on the entity tag of each text entity.

[0030] It should be noted that the method for determining training data provided in the embodiments of the present disclosure may be executed by the server 105 , and accordingly, the device for determining training data may be disposed in the server 105 .

[0031] It should be understood that Figure 1 The number of devices, networks and servers in the embodiment is only for illustration. Any number of devices, networks and servers may be provided as required.

[0032] Continue to refer Figure 2 , shows a process 200 of an embodiment of a method for extracting information according to the present disclosure. The method for extracting information comprises the following steps:

[0033] Step 201, obtaining a picture to be identified.

[0034] In this embodiment, the execution subject of the method for extracting information (eg Figure 1 The server shown in the figure) obtains the image to be identified by wired or wireless means. The image to be identified can be any image containing text information, such as medical documents, advertisements, warehouse documents, etc.

[0035] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0036] Step 202, obtaining elements in the image to be identified, wherein the elements include multiple text entities in the image to be identified.

[0037] In this embodiment, text recognition technology (for example, OCR technology, a pre-trained natural language recognition model, or a pre-trained vocabulary recognition model, etc.) can be used to identify the image to be identified, and multiple text entities in the identified image to be identified can be obtained. The text entity can be a vocabulary or a text. For example, "gender" in a medical document can be a text entity, and "female" in a medical document can also be a text entity.

[0038] Step 203: determine an entity label for each of the multiple text entities.

[0039] In this embodiment, the entity label of each text entity among multiple text entities can be determined, wherein the entity label can be any predefined label such as "heading", "title", "question", "reply", "table header", "table content", etc.

[0040] Specifically, OCR technology and a predefined label library can be used to assign a predefined entity label to the recognized text; a pre-trained label prediction model can be used to predict the label of the text entity based on the characteristics of the text entity; a predetermined correspondence table between entities and labels can be used to traverse and query the entity label of each text entity, and so on.

[0041] Step 204: extract text information from the image to be identified based on the entity label of each text entity.

[0042] In this embodiment, the text information in the image to be identified can be extracted according to the entity label of each text entity. For example, the text entity with the label "question" and the text entity with the label "answer" can be extracted as the text information in the image to be identified. For another example, the text entity with the label "table header" and the text entity with the label "table content" can be extracted as the text information in the table in the image to be identified, etc.

[0043] The method for extracting information provided in the present embodiment comprises the steps of obtaining a picture to be identified; obtaining elements in the picture to be identified, wherein the elements include a plurality of text entities in the picture to be identified; determining an entity label of each text entity in the plurality of text entities; and extracting text information in the picture to be identified based on the entity label of each text entity. The text information in the picture to be identified can be extracted based on the label of the text entity, thereby improving the efficiency of identifying text information in the picture. Labels can be assigned to each text entity in the picture, and information can be extracted based on the labels, thereby avoiding information omissions and the problem of erroneously outputting adjacent text entities as a whole due to the close proximity of adjacent text entities, thereby improving the accuracy of extracted information.

[0044] Optionally, obtaining elements in the picture to be identified includes: inputting the picture to be identified into a multimodal information extraction pre-training model, and obtaining the elements in the picture to be identified output by the multimodal information extraction pre-training model.

[0045] In this embodiment, the multimodal information extraction pre-training model is used to extract the features of elements in the image to be identified, wherein the elements include text in the image, the position of the text, etc. The multimodal information extraction pre-training model can be a model such as LayoutXLM. The multimodal information extraction pre-training model can use fields to organize document visual clues, build matching relationships between fields, align image and text features, build a "masked visual language model", "field length prediction" and "field orientation prediction" pre-training tasks, promote cross-modal feature interaction, help the model learn information associations between modalities, and increase the ability to comprehensively understand documents.

[0046] Continue to refer Figure 3 , shows a process 300 of another embodiment of a method for extracting information according to the present disclosure. The method for extracting information comprises the following steps:

[0047] Step 301, obtaining a picture to be identified.

[0048] The description of step 301 in this embodiment is consistent with the description of step 201 and will not be repeated here.

[0049] Step 302, obtaining elements in the image to be identified, wherein the elements include multiple text entities in the image to be identified and the position of each text entity in the image to be identified.

[0050] In this embodiment, text recognition technology (for example, OCR technology, a pre-trained natural language recognition model, or a pre-trained vocabulary recognition model, etc.) can be used to identify the image to be identified, and obtain multiple text entities in the image to be identified, as well as the position of each text entity in the image.

[0051] Step 303, determining an entity tag for each text entity in the plurality of text entities; wherein the entity tags include: question, answer.

[0052] In this embodiment, the entity tag of each of the multiple text entities can be determined, where the entity tag can be "question" or "reply". For example, the entity tag of the text entity "gender" can be "question", and the entity tag of the text entity "female" can be "reply". For another example, the entity tag of the text entity "date" can be "question", and the entity tag of the text entity "January 1, 2022" can be "reply".

[0053] Step 304, for a text entity with an entity tag of a question, the text entity and a text entity with an entity tag of an answer located at a preset position of the text entity are determined as candidate key-value pairs.

[0054] In this embodiment, for a text entity with an entity tag of "question", a candidate answer from the text entity is determined based on the position relationship, that is, the text entity and a text entity with an entity tag of "answer" located at a preset position of the text entity are determined as candidate key-value pairs. Figure 4 As shown, the preset position can be set to the fan-shaped area on the right side of the text entity (positive direction of the x-axis), and the text entity in the figure and the three text entities in the fan-shaped area can be combined to form three candidate key-value pairs.

[0055] Step 305: Determine whether the text entity included in the candidate key-value pair is a key-value relationship based on the vector representation of the text entity included in the candidate key-value pair.

[0056] In this embodiment, after determining multiple candidate key-value pairs, for each of the multiple candidate key-value pairs, it is determined whether the text entity included in the candidate key-value pair is a key-value relationship based on the vector representation of the text entity included in the candidate key-value pair, that is, the target key-value pair is determined from the candidate key-value pairs.

[0057] Specifically, the head vector representation of each text entity in the candidate key-value pair can be taken, and the head vector representation can be input into a pre-trained classification model to obtain a classification result, wherein the pre-trained classification model is trained based on the sample as the concatenation result of the head vector representation of the entity and the training data in which the sample label is the relationship between the entities. For example, if "gender" and "name" constitute a candidate key-value pair, the vector representation of "gender" and the vector representation of "surname" are taken, and the concatenation is input into the classification model, and the classification result is "no relationship".

[0058] Specifically, the average vector representation of each text entity in the candidate key-value pair can be taken, and the average vector can be input into a pre-trained classification model to obtain a classification result, wherein the pre-trained classification model is trained based on training data in which samples are the average vector representation of entities and sample labels are the relationships between entities.

[0059] Step 305: determine the text entity with the key-value relationship as the text information in the image to be identified.

[0060] In this embodiment, the text entity having a key-value relationship is determined as the text information in the image to be identified.

[0061] The method for extracting information provided in this embodiment is compared with Figure 2The method in the described embodiment specifically defines that when the entity label is "question", a text entity with an entity label of "reply" that constitutes a candidate key-value pair with the text entity with the entity label of "question" is determined based on the positional relationship, and a key-value pair with a key-value relationship is determined from the candidate key-value pairs based on the classification model, and the key-value pair with a key-value relationship is used as extracted information, which can improve the accuracy of the extracted information.

[0062] Continue to refer Figure 5 , shows a process 500 of another embodiment of a method for extracting information according to the present disclosure. The method for extracting information comprises the following steps:

[0063] Step 501, obtaining a picture to be identified.

[0064] The description of step 501 in this embodiment is consistent with the description of step 201 and will not be repeated here.

[0065] Step 502, obtaining elements in the image to be recognized, wherein the elements include multiple text entities in the image to be recognized and the position of each text entity in the image to be recognized.

[0066] In this embodiment, text recognition technology (for example, OCR technology, a pre-trained natural language recognition model, or a pre-trained vocabulary recognition model, etc.) can be used to identify the image to be identified, and obtain multiple text entities in the image to be identified, as well as the position of each text entity in the image.

[0067] Step 503, determining an entity tag of each text entity in the plurality of text entities; wherein the entity tag further includes: a table header.

[0068] In this embodiment, an entity tag of each text entity among multiple text entities may be determined, wherein the entity tag may be “table header” or “table content”.

[0069] Step 5041, for each text entity whose entity tag is the table header, extract the text entities in the table that have the same column relationship with the text entity; for each text entity in the text entities that have the same column relationship, extract the text entities that have the same row relationship with the text entity.

[0070] In this embodiment, for the text entity with the entity tag "table header", we can first find the elements (text entities) in the same column under the table header and sort them according to the y coordinate (upper and lower position). After that, we can traverse each column in turn according to the number of elements in each column, starting from the column with the most elements, to obtain the elements (text entities) in the same row, thereby realizing structured extraction of table information.

[0071] Step 5042, for each text entity whose entity tag is the table header, extract the text entities in the table that have the same row relationship with the text entity; for each text entity in the text entities that have the same row relationship, extract the text entities that have the same column relationship with the text entity.

[0072] In this embodiment, for the text entity with the entity tag "table header", first find the same row elements (text entities) under the table header according to the table header, and sort them according to the x-coordinate (left and right position). After that, start from the row with the most elements according to the number of elements in each row, traverse each row in turn, and obtain the elements (text entities) in the same column, thereby realizing structured extraction of table information.

[0073] The method for extracting information provided in this embodiment is compared with Figure 2 The method in the described embodiment specifically defines that when the entity tag of the text entity is distinguished as "table header" or "table content", the text entity is extracted based on the row / column relationship, which can improve the accuracy of the extracted information.

[0074] Optionally, the method for determining whether at least two elements have the same row relationship includes: determining whether the first element and the second element have the same row relationship based on the coverage ratio between the horizontal projection of the area where the first element is located and the horizontal projection of the area where the second element is located; or, the method for determining whether at least two elements have the same column relationship includes: determining whether the third element and the fourth element have the same column relationship based on the coverage ratio between the vertical projection of the area where the third element is located and the vertical projection of the area where the fourth element is located.

[0075] In this embodiment, since the image to be identified is shot at an angle rather than vertically, the image to be identified has perspective. When detecting whether two elements in a table have the same row relationship (i.e., whether they are located in the same row) or the same column relationship (i.e., whether they are located in the same column), the coverage of the positions of adjacent detection boxes (detection boxes are used to detect text entities) can be used to determine whether the elements are on the same horizontal line or the same vertical line, so as to improve the accuracy of determining the same row / column. For example, the calculation method of the vertical coverage is as follows: Figure 6 As shown in the figure, the coverage ratio is area C / area A, where the numerator is the overlapped part of A and B after alignment, and the denominator is the smaller area of ​​A and B. At this time, we can judge whether A and B are on the same straight line (row / column) according to the threshold. For example, when the coverage ratio is greater than 0.5, it is determined that the two are on the same straight line. A and B are both text entity detection boxes / recognition boxes.

[0076] In some application scenarios, the image to be identified is a medical document. When extracting structured information from the medical document, first, the medical document is obtained. After obtaining the medical document, OCR technology can be used to identify the medical document to obtain the text entity and the position of each text entity in the medical document, and the entity label of each text entity is determined based on the model.

[0077] Secondly, entity classification is performed, and the elements recognized by OCR technology (including text and position, etc.) are input into the LayoutXLM pre-trained model to obtain the characteristics of each text entity. After that, the LayoutXLM pre-trained model performs BIO sequence annotation on these texts (such as Figure 7 ), where BIO annotation is used to label each element as "B", "I", "O", "B" indicates that the segment where this element is located belongs to the start position, "I" indicates that the segment where this element is located belongs to the middle position, and "O" indicates that the segment where this element is located does not belong to any type. Then the entity label of each text entity is obtained. For example, when the OCR technology detects the text box "Gender: Male", the expected predicted label is "QQQA" (Q stands for "QUESTION" / "question", A stands for "ANSWER" / "reply"). In order to solve the problem that the OCR technology recognizes the key values ​​with small intervals in the document into the same text box, the decoding algorithm is adapted so that the decoding results are ("Gender:", QQQ), ("Male", A). According to the above decoding rules, it is possible to avoid the situation where the OCR technology recognizes the key values ​​in the document in the same detection box / recognition box.

[0078] Since it is necessary to process the key value information and table information in the medical record documents at the same time, in order to facilitate subsequent decoding and structured output, six entity tags (QUESTION, ANSWER, HEADER, TABLE, ELEMENT, OTHER) are predefined. The specific meanings are as follows:

[0079]

[0080] Next, perform entity connection: Based on the text entities and entity labels, determine whether two text entities can form a key-value pair, or whether the two text entities are in the same row or column in the table. That is, the classification model can classify the relationship between two text entities into: key-value relationship, relationship in the same row in the table, relationship in the same column in the table, and no relationship.

[0081] In the text entity preset area of ​​the QUESTION tag (such as Figure 4 and Figure 8 ), traverse the text entities of the ANSWER tag. Among them, the preset area can be based on the position of the recognition box recognized by the OCR technology, such as Figure 4The candidate text entities are traversed in the fan-shaped area in the schematic diagram, and the text entity of the QUESTION label and the candidate text entities are combined into candidate key-value pairs. The text entities included in each pair of candidate key-value pairs can be input into the LayoutXLM pre-trained model, and the head vector representation of each pair of entities (for example: "gender", the vector without the word "sex") is taken out, and after splicing, they are input into the classification model / prediction model. Based on the classification model / prediction model, it is determined whether the text entities in each pair of candidate key-value pairs have a key-value relationship, so as to determine the text entities with a key-value relationship.

[0082] When matching cross-row key-value pairs, such as Fig. 9 The text entities E1, E2, and E3 in the example are actually a value entity. However, due to the small amount of such data in the training data, it is difficult for the model to learn the positional relationship feature, so the model cannot connect the text entities E1 and E3 with the key entity. At this time, for the text entities E1 and E3 that do not have a key-value connection relationship with the key (type ANSWER), according to the pre-defined positional relationship (such as key on the left, value on the right, or key on the top, value on the bottom), the nearest entity of type QUESTION that meets the pre-defined positional relationship can be determined around them, and the connection relationship can be supplemented.

[0083] The LayoutXLM pre-trained model is used to coarsely classify the layout of medical documents into table areas and non-table areas to improve the accuracy of entity classification. The entity labels of text entities in the table area are fixed to only one of TABLE and ELEMENT. For elements located in the same row or column of the table, the table is "restored" according to the relative position information of the text, that is, the table is structured. Specifically, according to the table header, find the elements in the same column under the header and sort them according to the y coordinate (upper and lower position); then find the column with the most elements, traverse it in turn, and get the elements in the same row to complete the structuring of the table.

[0084] Further references Fig.10 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a device for extracting information. Figure 2 , Figure 3 or Figure 5 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0085] like Fig.10As shown, the apparatus 1000 for extracting information in this embodiment includes: an acquisition unit 1001, an identification unit 1002, a marking unit 1003, and an extraction unit 1004. The acquisition unit 1001 is configured to acquire a picture to be identified; the identification unit is configured to acquire elements in the picture to be identified, wherein the elements include multiple text entities in the picture to be identified; the marking unit is configured to determine an entity label of each text entity in the multiple text entities; and the extraction unit is configured to extract text information in the picture to be identified according to the entity label of each text entity.

[0086] In some embodiments, the recognition unit includes: a recognition module configured to input the image to be recognized into a multimodal information extraction pre-training model to obtain elements in the image to be recognized output by the multimodal information extraction pre-training model.

[0087] In some embodiments, the element also includes position information of each text entity in the picture to be identified, and the entity labels include: question, answer; the extraction unit includes: a determination module, configured to, for a text entity with an entity label of question, determine the text entity and a text entity with an entity label of answer located at a preset position of the text entity as a candidate key-value pair; a judgment module, configured to determine whether the text entities included in the candidate key-value pair are in a key-value relationship based on the vector representation of the text entities included in the candidate key-value pair; a first extraction module, configured to determine the text entities with a key-value relationship as text information in the picture to be identified.

[0088] In some embodiments, the element also includes position information of each text entity in the image to be identified, and the entity label also includes: a table header; an extraction module, including: a second extraction module, configured to extract, for each text entity whose entity label is a table header, text entities in the table that have the same column relationship with the text entity; for each text entity in the text entities that have the same column relationship, extract text entities that have the same row relationship with the text entity; or, a third extraction module, configured to extract, for each text entity whose entity label is a table header, text entities in the table that have the same row relationship with the text entity; for each text entity in the text entities that have the same row relationship, extract text entities that have the same column relationship with the text entity.

[0089] In some embodiments, a method for determining whether at least two elements have the same row relationship includes: determining whether the first element and the second element have the same row relationship based on the coverage ratio between the horizontal projection of the area where the first element is located and the horizontal projection of the area where the second element is located; or, a device for determining whether at least two elements have the same column relationship includes: determining whether the third element and the fourth element have the same column relationship based on the coverage ratio between the vertical projection of the area where the third element is located and the vertical projection of the area where the fourth element is located.

[0090] Each unit in the above device 1000 is similar to the reference Figure 2 , Figure 3 or Figure 5 Therefore, the operations, features and technical effects described above for the method for extracting information are also applicable to the device 1000 and the units contained therein, and will not be described in detail here.

[0091] According to an embodiment of the present application, the present application also provides an electronic device, a readable storage medium and a computer program product.

[0092] Fig.11 A schematic block diagram of an example electronic device 1100 that can be used to implement an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0093] like Fig.11 As shown, the device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the device 1100 can also be stored. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0094] Multiple components in the device 1100 are connected to the I / O interface 1105, including: an input unit 1106, such as a keyboard, a mouse, etc.; an output unit 1107, such as various types of displays, speakers, etc.; a storage unit 1108, such as a disk, an optical disk, etc.; and a communication unit 1105, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1105 allows the device 1100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0095] The computing unit 1101 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1101 performs the various methods and processes described above, such as methods for extracting information. For example, in some embodiments, the method for extracting information may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1100 via the ROM 1102 and / or the communication unit 1105. When the computer program is loaded into the RAM 1103 and executed by the computing unit 1101, one or more steps of the method for extracting information described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to execute the method for extracting information in any other appropriate manner (eg, by means of firmware).

[0096] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0097] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0098] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0099] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0100] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0101] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0102] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the data expected by the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0103] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A method for extracting information, comprising: Get the image to be identified; Input the image to be identified into a multimodal information extraction pre-training model, and obtain elements in the image to be identified output by the multimodal information extraction pre-training model, wherein the elements include multiple text entities in the image to be identified, and the multimodal information extraction pre-training model organizes document visual clues by fields, constructs matching relationships between fields, aligns image and text features, and constructs "masked visual language model", "field length prediction" and "field orientation prediction" pre-training tasks; Determining an entity label for each of the plurality of text entities; Extracting text information from the image to be identified according to the entity label of each text entity; The element also includes the location information of each text entity in the image to be identified, and the entity label includes: question, answer; The extracting text information in the to-be-recognized image according to the entity label of each text entity includes: For a text entity whose entity tag is a question, the text entity and a text entity whose entity tag is an answer and located at a preset position of the text entity are determined as candidate key-value pairs; Determining whether the text entity included in the candidate key-value pair is a key-value relationship according to the vector representation of the text entity included in the candidate key-value pair; The text entity having the key-value relationship is determined as the text information in the image to be identified.

2. The method according to claim 1, wherein: The element also includes the position information of each text entity in the image to be identified, and the entity tag also includes: a table header; The extracting text information in the to-be-recognized image according to the entity label of each text entity includes: For each text entity whose entity tag is a table header, extract the text entity in the table that has the same column relationship with the text entity; for each text entity in the text entity that has the same column relationship, extract the text entity that has the same row relationship with the text entity; or, For each text entity whose entity tag is a table header, extract the text entities in the table that have the same row relationship with the text entity; for each text entity in the text entities that have the same row relationship, extract the text entities that have the same column relationship with the text entity.

3. The method according to claim 2, wherein: The method for determining whether at least two elements have a same row relationship includes: determining whether the first element and the second element are in the same row based on a coverage ratio between a lateral projection of a region where the first element is located and a lateral projection of a region where the second element is located; or The method for determining whether at least two elements have a same column relationship includes: Based on the coverage ratio between the longitudinal projection of the region where the third element is located and the longitudinal projection of the region where the fourth element is located, it is determined whether the third element and the fourth element have a same column relationship.

4. A device for extracting information, comprising: An acquisition unit, configured to acquire a picture to be identified; A recognition unit is configured to input the image to be recognized into a multimodal information extraction pre-training model, obtain elements in the image to be recognized output by the multimodal information extraction pre-training model, wherein the elements include multiple text entities in the image to be recognized, and the multimodal information extraction pre-training model organizes document visual clues by fields, constructs matching relationships between fields, aligns image and text features, and constructs "masked visual language model", "field length prediction" and "field orientation prediction" pre-training tasks; a labeling unit configured to determine an entity label for each of the plurality of text entities; An extraction unit, configured to extract text information from the to-be-recognized image according to the entity label of each text entity; The element also includes the location information of each text entity in the image to be identified, and the entity label includes: question, answer; The extraction unit comprises: A determination module is configured to determine, for a text entity with an entity tag of a question, the text entity and a text entity with an entity tag of an answer located at a preset position of the text entity as a candidate key-value pair; A judgment module is configured to determine whether the text entity included in the candidate key-value pair is a key-value relationship according to the vector representation of the text entity included in the candidate key-value pair; The first extraction module is configured to determine the text entity having the key-value relationship as the text information in the image to be identified.

5. The device according to claim 4, wherein: The element also includes the position information of each text entity in the image to be identified, and the entity tag also includes: a table header; The extraction module comprises: The second extraction module is configured to extract, for each text entity whose entity tag is a table header, text entities in the table that have a same column relationship with the text entity; for each text entity in the text entity that has a same column relationship, extract text entities that have a same row relationship with the text entity; or, The third extraction module is configured to extract, for each text entity whose entity tag is a table header, text entities in the table that have the same row relationship with the text entity; and for each text entity in the text entities that have the same row relationship, extract text entities that have the same column relationship with the text entity.

6. The device according to claim 5, wherein: The method for determining whether at least two elements have a same row relationship includes: determining whether the first element and the second element are in the same row based on a coverage ratio between a lateral projection of a region where the first element is located and a lateral projection of a region where the second element is located; or The device for determining whether at least two elements have a same column relationship comprises: Based on the coverage ratio between the longitudinal projection of the region where the third element is located and the longitudinal projection of the region where the fourth element is located, it is determined whether the third element and the fourth element have a same column relationship.

7. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 3.

8. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 3.

9. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 3.

Images