Information extraction method and apparatus

Abstract

An information extraction method and apparatus. The information extraction method comprises: receiving an image to be processed, wherein said image comprises unstructured information (102); and inputting said image into an information extraction model to obtain structured information outputted by the information extraction model, wherein the information extraction model determines word information and word position information in said image, determines entity information on the basis of the word information and the word position information, and generates the structured information on the basis of the entity information (104). In the present method, the process of determining the entity information on the basis of the word information and the word position information and generating the structured information on the basis of the entity information, and the process of determining the word information and the word position information from said image are two branches that are processed in parallel, and the two branches share the same feature extraction module, such that the entire link is relatively short.

Description

Information extraction method and device TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of computer, in particular to an information extraction method. BACKGROUND

[0002] Visual information extraction is one of the important capabilities in image processing, involving capabilities such as text detection, text recognition, and information extraction. Through a visual information extraction algorithm, information can be extracted from scattered, redundant, and unstructured images, thereby obtaining focused and simplified structured information.

[0003] Current visual information extraction methods include OCR-based, OCR-dependent, and OCR-free. OCR-based integrates text detection, recognition, and extraction into the same framework to achieve information extraction. OCR-dependent extracts OCR information through a third-party OCR engine and then inputs the information into a language model for information extraction. OCR-free inputs an image into a model, and the model directly extracts information without text detection and recognition. These three methods have problems such as relying on the capabilities of a third-party OCR engine, being unable to determine the content and location of text, or having a long processing flow and difficulty in model training. Therefore, there is an urgent need for technical personnel to adjust the information extraction scheme and enhance the flexibility and convenience of the model. SUMMARY

[0004] Therefore, the embodiments of the present disclosure provide an information extraction method. One or more embodiments of the present disclosure also provide an information extraction device, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects in the prior art.

[0005] According to a first aspect of the embodiments of the present disclosure, an information extraction method is provided, including:

[0006] receiving a to-be-processed image, wherein the to-be-processed image includes unstructured information;

[0007] inputting the to-be-processed image into an information extraction model to obtain structured information output by the information extraction model, wherein the information extraction model determines word information and word position information in the to-be-processed image, determines entity information according to the word information and the word position information, and generates the structured information according to the entity information.

[0008] According to a second aspect of the embodiments of the present disclosure, an information extraction method applied to a cloud-side device is provided, including:

[0009] receiving a to-be-processed image sent by an end-side device, wherein the to-be-processed image includes unstructured information;

[0010] input the image to be processed into the information extraction model to obtain structured information output by the information extraction model, wherein the information extraction model determines word information and word position information in the image to be processed, determines entity information according to the word information and the word position information, and generates the structured information according to the entity information;

[0011] send the structured information to the terminal device.

[0012] According to a third aspect of the embodiments of the present disclosure, an information extraction apparatus is provided, comprising:

[0013] a receiving module configured to receive an image to be processed, wherein the image to be processed comprises unstructured information;

[0014] an extraction module configured to input the image to be processed into the information extraction model to obtain structured information output by the information extraction model, wherein the information extraction model determines word information and word position information in the image to be processed, determines entity information according to the word information and the word position information, and generates the structured information according to the entity information.

[0015] According to a fourth aspect of the embodiments of the present disclosure, a computing device is provided, comprising:

[0016] a memory and a processor;

[0017] the memory is configured to store computer programs / instructions, and the processor is configured to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the information extraction method.

[0018] According to a fifth aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, which stores computer programs / instructions, which, when executed by a processor, implement the steps of the information extraction method.

[0019] According to a sixth aspect of the embodiments of the present disclosure, a computer program product is provided, comprising computer programs / instructions, which, when executed by a processor, implement the steps of the information extraction method.

[0020] Through the method provided by the present disclosure, the text detection and the text recognition of the image to be processed are combined and processed to obtain the word information and the word position information. Meanwhile, the process of determining the entity information according to the word information and the word position information and generating the structured information according to the entity information is a parallel branch with the process of determining the word information and the word position information from the image to be processed. The two branches share the same feature extraction module, so that the entire link is relatively short. The information extraction framework is further simplified. The processing flow of the information extraction framework is short. BRIEF DESCRIPTION OF DRAWINGS

[0021] FIG. 1 is a flowchart of a method for information extraction according to an embodiment of the present disclosure;

[0022] FIG. 2 is a schematic diagram of a model structure of an information extraction model according to an embodiment of the present disclosure;

[0023] FIG. 3 is a flowchart of a processing procedure of a method for information extraction according to an embodiment of the present disclosure;

[0024] FIG. 4 is a schematic diagram of a structure of an information extraction apparatus according to an embodiment of the present disclosure;

[0025] FIG. 5 is an architecture diagram of an information extraction system according to an embodiment of the present disclosure;

[0026] FIG. 6 is a flowchart of a method for information extraction applied to a cloud-side device according to an embodiment of the present disclosure;

[0027] FIG. 7 is a structural block diagram of a computing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, the present disclosure can be practiced without the specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to obscure the present disclosure. Some portions of the detailed description are presented in terms of algorithms, procedures, logic blocks, processing and other symbolic representations of operations on data bits that can be stored within a computer memory. These algorithmic descriptions and representations can be the techniques used by those of ordinary skill in the data processing arts to convey the substance of their work to others.

[0029] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of one or more embodiments of the present disclosure. As used in one or more embodiments of the present disclosure and the accompanying claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in one or more embodiments of the present disclosure and the accompanying claims, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0030] It will be understood that, although the terms first, second, etc. can be used herein to describe various information, these terms are not intended to denote a temporal or chronological order. Rather, these terms are used solely to distinguish one from another. For example, a first entity discussed below could be termed a second entity, and, similarly, a second entity could be termed a first entity without departing from the scope of one or more embodiments of the present disclosure. As used herein, the term "if' can be construed to mean "when" or "in response to determining" or "in response to a determination" or "in response to the fact that...."

[0031] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present disclosure are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards in the relevant region, and provide corresponding operation portal for user to choose authorization or refusal.

[0032] Firstly, the nomenclature involved in one or more embodiments of the present disclosure is explained.

[0033] OCR: Optical Character Recognition, optical character recognition, refers to the process of electronic equipment checking the characters printed on the picture, determining the shape by detecting the dark and light patterns, and then using character recognition method to translate the shape into computer text.

[0034] OCR-based: unify text detection, recognition and extraction into the same framework to realize information extraction.

[0035] OCR-dependent: extract OCR information including text position and content through a third-party OCR engine and input a language model for extraction.

[0036] OCR-free: only input pictures, and the model directly extracts the content without text detection and recognition.

[0037] Visual information extraction is one of the important capabilities in image processing, involving text detection, text recognition, information extraction and other capabilities. Through the visual information extraction algorithm, information can be extracted from scattered, redundant and unstructured images, so as to obtain focused and simplified structured information.

[0038] The current visual information extraction algorithm has the above-mentioned OCR-based, OCR-dependent, OCR-free, etc. Among them, the OCR-dependent scheme only considers the last field classification link, greatly simplifies information extraction, which is not conducive to information extraction and will be affected by the effect of the third-party OCR engine; in the industrial scene of visual information extraction, the text content and position need to be output, so the OCR-free scheme is not suitable for industrial scenes; the OCR-based scheme is less, mostly training machine learning models for text detection, recognition and extraction respectively, and then connecting each module, resulting in a long process and high training difficulty.

[0039] Based on this, in the present disclosure, an information extraction method is provided, and the present disclosure also relates to an information extraction device, a computing device, a computer readable storage medium and a computer program product, which are described in detail in the following embodiments.

[0040] Referring to FIG. 1, FIG. 1 shows a flowchart of an information extraction method according to an embodiment of the present disclosure, which specifically includes the following steps.

[0041] Step 102: receiving an image to be processed.

[0042] The image to be processed includes unstructured information.

[0043] The image to be processed can be understood as an image that needs to be subjected to information extraction, and the image to be processed includes unstructured information. The unstructured information can be understood as information contained in the image to be processed being unstructured. The unstructured information can be understood as information contained in the image to be processed not being information in a preset structured format. The method provided by the present disclosure needs to extract information from the image to be processed and store the information in a structured manner. The structured storage refers to storing information according to a preset format.

[0044] In actual application, the image to be processed can be various types of bill images, photos, etc. The image to be processed includes text information, and the text information in the image to be processed is extracted by an OCR recognition manner and stored in a structured manner.

[0045] Step 104: inputting the image to be processed into an information extraction model to obtain structured information output by the information extraction model.

[0046] The information extraction model determines word information and word position information in the image to be processed, determines entity information according to the word information and the word position information, and generates structured information according to the entity information.

[0047] In the method provided by the present disclosure, the information extraction model specifically refers to a machine learning model for processing images, recognizing and extracting information in the images. The information extraction model is used to perform a visual information extraction task for images. In the method provided by the embodiment of the present disclosure, the information extraction model divides the visual information extraction task into three subtasks, namely, a word positioning task, an entity grouping task and an entity classification task.

[0048] Specifically, the information extraction model first processes the to-be-processed image to determine word information and word position information in the to-be-processed image. Then, entity grouping is performed according to the word information and the word position information to determine entity information. The entity information can be understood as obtained entity content information. After the entity information is identified and classified, the category of the entity information is obtained. Finally, structured information is output according to the entity information and the category of the entity information. The structured information can be understood as storing the extracted entity information and the category of the entity information in a preset format.

[0049] In a specific embodiment provided in the present disclosure, the information extraction model includes a word positioning unit, an entity grouping unit, and an entity classification unit.

[0050] The to-be-processed image is input into the information extraction model to obtain structured information output by the information extraction model, including:

[0051] In S1042, the to-be-processed image is input into the word positioning unit to obtain at least one word information and word position information corresponding to each word information.

[0052] Based on the above three subtasks, the information extraction model includes three units, namely, a word positioning unit, an entity grouping unit, and an entity classification unit. The word positioning unit is used to perform a word spotting task. The word positioning unit can recognize and locate the text in the picture at the word granularity to obtain word information of the text in the to-be-processed image at the word granularity and word position information corresponding to each word information.

[0053] The word granularity is the word granularity. A word is a language unit in natural language. For example, for the sentence "today the weather is very good", the words can be divided according to the word granularity as "today", "weather", "very", and "good".

[0054] The word information set is information obtained after division according to the word granularity. The word position information specifically refers to the position information of each word information. In the method provided in the embodiment of the present disclosure, the word position information can be understood as the position information of each word information in the to-be-processed image.

[0055] In the word positioning unit, the word spotting task is performed, specifically, the text in the to-be-processed image is detected and recognized. In the current task, only the text content at the word granularity can be recognized, but the meaning of the word cannot be known.

[0056] In a specific embodiment provided in the present disclosure, the word positioning unit includes a feature extraction module and a positioning module.

[0057] The to-be-processed image is input into the word positioning unit to obtain at least one word information and word position information corresponding to each word information, including:

[0058] input the image to be processed into a feature extraction module to obtain multi-scale feature information corresponding to the image to be processed;

[0059] input the multi-scale feature information into a positioning module to obtain at least one word information and word position information corresponding to each word information output by the positioning module.

[0060] In the embodiment, the word positioning unit includes a feature extraction module and a positioning module, wherein the feature extraction module is configured to extract image features of the image to be processed, and the positioning module is configured to recognize and position the text in the image to be processed according to the image features.

[0061] Specifically, the image to be processed is input into the feature extraction module, and the multi-scale feature information corresponding to the image to be processed is obtained through processing of the image to be processed by the feature extraction module. The feature extraction module can be understood as at least one convolution layer, and the multi-scale feature information corresponding to the image to be processed is obtained by convolution processing of the image to be processed by each convolution layer.

[0062] The multi-scale feature information is input into the positioning module, and the word information in the image to be processed and the word position information corresponding to each word information are recognized through decoding processing of the multi-scale feature information in the positioning module.

[0063] In the method provided in the present disclosure, the tasks of text detection and text recognition in the image to be processed are completed in the word positioning unit, that is, whether there is text in the image to be processed can be detected, and the detected text can also be recognized, realizing the unification of text detection and text recognition. In the word positioning unit, the same feature extraction module can be used to process the two tasks of text detection and text recognition, avoiding the training of a text detection module and a text recognition module respectively, shortening the model volume of the information extraction model, and shortening the processing link of the model.

[0064] S1044, input the image to be processed, at least one word information and word position information corresponding to each word information into an entity grouping unit to obtain at least one entity detection frame and entity information corresponding to each entity detection frame.

[0065] After obtaining at least one word information and word position information corresponding to each word information, the entity grouping task (word grouping) can be further performed according to at least one word information, word position information corresponding to each word information and the image to be processed. The entity grouping task is used to group the detected word information. It should be noted that in the embodiment, the focus is on grouping each word information according to the visual information of the image to be processed. The grouping according to the visual information can be understood as visually grouping the content in the same row, the content in adjacent columns, etc.

[0066] The entity grouping unit performs an entity grouping task to obtain at least one entity bounding box and entity information corresponding to each entity bounding box. The entity bounding box can be understood as a bounding box marking the corresponding entity.

[0067] In a specific embodiment provided by the present disclosure, the entity grouping unit includes a character encoding module, a visual feature fusion module, and an entity grouping module.

[0068] The processed image, at least one word information, and word position information corresponding to each word information are input into the entity grouping unit to obtain at least one entity bounding box and entity information corresponding to each entity bounding box, including:

[0069] A processed grid positioning image is generated according to the processed image, at least one word information, and word position information corresponding to each word information.

[0070] The processed grid positioning image is input into the character encoding module to obtain character encoding features.

[0071] The character encoding features and multi-scale feature information are input into the visual feature fusion module to obtain image fusion feature information.

[0072] The image fusion feature information is input into the entity grouping module to obtain at least one entity bounding box and entity information corresponding to each entity bounding box output by the entity grouping module.

[0073] The entity grouping unit specifically includes a character encoding module, a visual feature fusion module, and an entity grouping module. The character encoding module is configured to encode word information at a character granularity (char granularity). The visual feature fusion module extracts image fusion feature information according to word information positions. Finally, the image fusion feature information is used to obtain entity bounding boxes and entity information corresponding to each entity bounding box.

[0074] Specifically, in the process of performing the entity grouping task, the method provided by the present disclosure uses a target detection algorithm to implement. In actual application, if the entity boundary is to be recognized, in addition to the layout information in the image, the semantic information of the text also needs to be known to more accurately obtain the entity boundary. Therefore, in the method provided by the present disclosure, a processed grid positioning image (Chargrid) is introduced. The processed grid positioning image can be understood as an image marked with character identification information corresponding to each word information.

[0075] Specifically, the processed grid positioning image is generated according to the processed image, at least one word information, and word position information corresponding to each word information. The processed grid positioning image is presented at a character granularity (char). The processed grid positioning image is generated according to the processed image, at least one word information, and word position information corresponding to each word information, including:

[0076] generating a to-be-processed empty image according to the to-be-processed image;

[0077] determining character identification information corresponding to each word information according to the at least one word information;

[0078] adding the character identification information corresponding to each word information to the to-be-processed empty image according to word position information corresponding to each word information, to generate a to-be-processed grid positioning image.

[0079] In actual application, an empty image is created according to the size of the to-be-processed image, the to-be-processed empty image is an image with blank information, and the image size of the to-be-processed empty image is the same as that of the to-be-processed image. The character identification information corresponding to each word information is determined, and the character identification information is specifically ID information corresponding to each character. For example, “today the weather is good” includes three words “today”, “weather” and “good”, wherein “today” includes two characters “today” and “weather”, and the two characters correspond to two character identification information respectively; “weather” includes two characters “weather” and “good”, and the two characters correspond to two character identification information respectively. It should be noted that the “today” of “today” and the “weather” of “weather” correspond to the same character identification information.

[0080] In a specific embodiment provided in the present disclosure,

[0081] determining character identification information corresponding to each word information according to the at least one word information, comprising:

[0082] determining at least one character information corresponding to target word information, wherein the target word information is any one of the at least one word information;

[0083] determining character identification corresponding to each character information according to a preset character correspondence table;

[0084] mapping each character identification to three-channel feature information to generate character identification information corresponding to each character identification.

[0085] In actual application, a character and character ID correspondence table is maintained, and according to each character in the word information, the character ID corresponding thereto can be uniquely determined. In the process of determining the character identification information corresponding to each word information, a target word information can be determined first, and at least one character information corresponding to the target word information is determined simultaneously. The target word information is any one of the word information.

[0086] According to the preset character correspondence table, the character identification corresponding to each character information can be determined. In the method provided in the present disclosure, in order to facilitate subsequent visual grouping, each character identification is mapped to three-channel feature information, and the three-channel feature information corresponding to each character identification is the character identification information.

[0087] Each character identifier is mapped to three-channel feature information, the purpose of which is to convert the input integer sequence into a dense vector representation, which can be achieved by the nn.embedding() function, specifically, the character identifier information corresponding to id1 can be represented by nn.embedding(id1, 3), where di1 is the character ID and 3 represents the dimension of embedding.

[0088] After determining the character identifier information corresponding to each word information, the character identifier information corresponding to each word information is added to the above-mentioned to-be-processed empty image according to the word position information corresponding to each word information, so as to generate a to-be-processed grid positioning image.

[0089] After determining the to-be-processed grid positioning image, the to-be-processed grid positioning image is input into the character encoding module for encoding to obtain the character encoding feature corresponding to the to-be-processed grid positioning image.

[0090] The character encoding feature and the multi-scale feature information are input into the visual feature fusion module for feature fusion processing, which can obtain layout information in the image according to the multi-scale feature information and obtain text semantic information according to the character encoding feature, so as to more accurately obtain the entity boundary and determine at least one entity detection frame and the entity information corresponding to each entity detection frame.

[0091] In a specific embodiment provided in the present disclosure, the visual feature fusion module includes a pyramid pooling layer and a visual feature fusion layer.

[0092] The character encoding feature and the multi-scale feature information are input into the visual feature fusion module to obtain image fusion feature information, including:

[0093] The multi-scale feature information is input into the pyramid pooling layer to obtain pooling feature information.

[0094] The pooling feature information and the character encoding feature are input into the visual feature fusion layer to obtain image fusion feature information.

[0095] In the present embodiment, the visual feature fusion module includes a pyramid pooling layer and a visual feature fusion layer. The multi-scale feature information generated in the previous task is input into the pyramid pooling layer for feature pooling, which can obtain feature information of different scales in the multi-scale feature information, and can obtain the pooling feature information of the to-be-processed image according to different size fields of view.

[0096] The pooling feature information and the character encoding feature information are input into the visual feature fusion layer for fusion to obtain image fusion feature information.

[0097] After obtaining the image fusion feature information, the image fusion feature information is input into an entity grouping module for processing, and at least one entity detection box and entity information corresponding to the entity detection box output by the entity grouping module are obtained.

[0098] In the entity grouping task provided in the embodiment, the visual feature information in the picture is referred to, and the text semantic information in the picture is also fused, so that the boundary of the entity is accurately determined, and the entity detection box is generated. After the entity detection box is determined, the entity information corresponding to the entity detection box can be determined according to the entity detection box.

[0099] In the entity grouping task provided in the embodiment, the visual feature information in the picture is referred to, and the text semantic information in the picture is also fused, so that the boundary of the entity is accurately determined, and the entity detection box is generated. After the entity detection box is determined, the entity information corresponding to the entity detection box can be determined according to the entity detection box.

[0100] The entity classification unit is used to perform an entity classification task (entity labeling), and each entity information is labeled and classified by the entity classification unit, so that the entity attribute of each entity is obtained. In the method provided in the disclosure, each entity information can be classified to determine whether the entity information is a key or a value. When the entity information is a value, the specific category of the entity information, such as a person's name, a place name, etc., can be further determined.

[0101] In a specific embodiment provided in the disclosure, the entity classification unit is used to perform an entity classification task (entity labeling), and each entity information is labeled and classified by the entity classification unit, so that the entity attribute of each entity is obtained. In the method provided in the disclosure, each entity information can be classified to determine whether the entity information is a key or a value. When the entity information is a value, the specific category of the entity information, such as a person's name, a place name, etc., can be further determined.

[0102] The entity classification unit is used to perform an entity classification task (entity labeling), and each entity information is labeled and classified by the entity classification unit, so that the entity attribute of each entity is obtained. In the method provided in the disclosure, each entity information can be classified to determine whether the entity information is a key or a value. When the entity information is a value, the specific category of the entity information, such as a person's name, a place name, etc., can be further determined.

[0103] In the method provided in the embodiment of the disclosure, in addition to inputting the entity detection box and the entity information into the entity classification unit, the grid positioning image to be processed also needs to be input into the entity classification unit. The visual information of the word granularity in the grid positioning image to be processed is used to assist coding.

[0104] Further, the entity classification unit includes a word granularity feature extraction module, a semantic coding module, and a semantic classifier.

[0105] The entity classification unit is used to perform an entity classification task (entity labeling), and each entity information is labeled and classified by the entity classification unit, so that the entity attribute of each entity is obtained. In the method provided in the disclosure, each entity information can be classified to determine whether the entity information is a key or a value. When the entity information is a value, the specific category of the entity information, such as a person's name, a place name, etc., can be further determined.

[0106] The grid positioning image to be processed is input into the word granularity feature extraction module, and the word granularity feature information output by the word granularity feature extraction module is obtained.

[0107] The word granularity feature information is input into a semantic coding module to obtain word granularity semantic feature information.

[0108] The word granularity semantic feature information, the entity detection frame and the entity information are input into a semantic classifier to obtain entity category information corresponding to the entity information.

[0109] In the embodiment, the entity classification unit comprises a word granularity feature extraction module, a semantic coding module and a semantic classifier. The word granularity feature extraction module is configured to perform word granularity feature extraction on the grid positioning image to be processed. The semantic coding module is configured to perform semantic coding on the features extracted by the word granularity feature extraction module to obtain encoded feature information of the word granularity. The encoded features are input into the semantic classifier to obtain entity classification information corresponding to the entity information.

[0110] Specifically, in the word granularity feature extraction module, feature points of each word information are determined in the grid positioning image to be processed, and three-dimensional feature values of the word information are converted into two-dimensional feature values of the corresponding feature points. Specifically, the three-dimensional feature values (length, width and channel) of the word information of the word granularity can be converted into two-dimensional features of the feature points by a words as point semantic model, i.e., word granularity feature information is obtained.

[0111] For example, the three-dimensional feature of the word granularity is H*W*C, where H represents the height of the word granularity, W represents the width of the word granularity, and C represents the number of channels of the word granularity. H and W of the word granularity are converted into 1 by the words as point model, i.e., the feature points corresponding to the word information, to obtain two-dimensional features of the feature points corresponding to the word information. In the case where the grid positioning image to be processed comprises N words, N pieces of 1*C word granularity feature information can be obtained.

[0112] After obtaining the word granularity feature information, the position coding of the word information is spliced and input into the semantic coding module for processing to obtain word granularity semantic feature information output by the semantic coding module.

[0113] The word granularity semantic feature information, the entity detection frame and the entity information are input into a semantic classifier, and the semantic classifier performs semantic classification according to the entity detection frame, the entity information corresponding to the entity detection frame and the word granularity semantic feature information to obtain entity category information corresponding to the entity information.

[0114] Specifically, the entity category information comprises whether the entity information is a key (kye) or a value (value). If the entity information is a value, it can be further determined that the entity information is of which category, for example, a certain entity information is a value, and the corresponding category is a name; for example, a certain entity information is a value, and the corresponding category is an address, and the like.

[0115] S1048, determine the structured information according to the entity information and the entity category information corresponding to the entity information.

[0116] After determining the entity information and the entity category information corresponding to the entity information, the corresponding structured storage can be performed according to the correspondence between the entity information and the entity category information, and the structured information is generated.

[0117] In a specific embodiment provided by the present disclosure, the structured information is determined according to the entity information and the entity category information corresponding to the entity information, including:

[0118] determining target entity information and target entity category information corresponding to the target entity information;

[0119] correspondingly adding the target entity information and the target entity category information to the structured information.

[0120] In actual application, the target entity information and the target entity category information corresponding to the target entity information can be determined, the target entity information and the target entity category information are correspondingly saved, and are added to the structured information.

[0121] In another specific embodiment provided by the present disclosure, the information extraction model is trained and generated through the following steps:

[0122] obtaining a sample image and sample word information, sample word position information and sample structured information corresponding to the sample image;

[0123] inputting the sample image into the information extraction model to obtain predicted structured information, predicted word information and predicted word position information output by the information extraction model;

[0124] calculating a model loss value according to the sample word information, the sample word position information, the sample structured information, the predicted structured information, the predicted word information and the predicted word position information;

[0125] adjusting model parameters of the information extraction model according to the model loss value, and continuing to train the information extraction model until a model training stop condition is reached.

[0126] In a specific embodiment provided by the present disclosure, the information extraction model is trained in a supervised training manner, which includes a sample image and sample word information, sample word position information and sample structured information corresponding to the sample image. In the training method provided by the present disclosure, the training sample pair is used, the sample image is sample data, and the sample word information, the sample word position information and the sample structured information are sample labels.

[0127] After obtaining the sample image, the sample image is input to the information extraction model, at this time the information extraction model is an untrained information extraction model, in the information extraction model, the corresponding predicted structured information, predicted word information and predicted word position information are generated according to each sample image, the data processing process of the sample image in the untrained information extraction model is the same as the data processing process in the above embodiment, for the data processing process of the sample image in the untrained information extraction model, refer to the data processing process of the information extraction model above, which will not be repeated here.

[0128] After obtaining the predicted structured information, predicted word information and predicted word position information, the model loss value can be calculated according to the prediction result and the sample label. That is, the model loss value is calculated according to the sample word information, sample word position information, sample structured information, predicted structured information, predicted word information and predicted word position information. In the method provided in the present disclosure, there are many methods for calculating the model loss value, such as cross-entropy loss function, maximum loss function, average value loss function, etc. In the present disclosure, the specific way of the loss function is not limited, and the actual application is used as the criterion.

[0129] In one or more specific embodiments provided in the present disclosure, the model loss value is calculated according to the sample word information, sample word position information, sample structured information, predicted structured information, predicted word information and predicted word position information, including:

[0130] The first loss value is calculated according to the sample word information, sample word position information and predicted word information, and predicted word position information;

[0131] The second loss value is calculated according to the sample structured information and predicted structured information;

[0132] The model loss value is calculated according to the first loss value and the second loss value.

[0133] Specifically, the first loss value is calculated according to the sample word information and predicted word information, and the sample word position information and predicted word position information, the second loss value is calculated according to the sample structured information and predicted structured information, and finally the final model loss value is determined according to the first loss value and the second loss value. The first loss value and the second loss value are loss values calculated by comparing different prediction information and sample information in the model training process, which facilitates the subsequent targeted parameter adjustment of the model.

[0134] The information extraction method provided by the embodiments of the present disclosure includes receiving an image to be processed, wherein the image to be processed includes unstructured information; inputting the image to be processed into an information extraction model to obtain structured information output by the information extraction model, wherein the information extraction model determines word information and word position information in the image to be processed, determines entity information according to the word information and the word position information, and generates the structured information according to the entity information.

[0135] The method provided by the present disclosure combines word detection and word recognition of the image to be processed, simultaneously processes to obtain word information and word position information, simultaneously determines entity information according to the word information and the word position information, and generates the structured information according to the entity information. The process of determining the word information and the word position information from the image to be processed is a parallel branch of the two branches, and the two branches share the same feature extraction module, so that the entire link is relatively short. The information extraction framework is further simplified. The processing flow of the information extraction framework is short.

[0136] In addition, by multi-level representation of character granularity, word granularity and entity granularity, different granularity tasks are unified into the same information extraction framework, so that the information extraction framework is simpler and lighter.

[0137] Referring to FIG. 2, FIG. 2 shows a model structure diagram of an information extraction model provided by an embodiment of the present disclosure. As shown in FIG. 2, the information extraction model includes a word positioning unit, an entity grouping unit and an entity classification unit.

[0138] The word positioning unit is used to perform a word positioning task, the entity grouping unit is used to perform an entity grouping task, and the entity classification unit is used to perform an entity classification task. In the information extraction model, the word positioning unit is a first branch, the entity grouping unit and the entity classification unit are a second branch, and the first branch and the second branch are two parallel branches. In the model training stage, the two branches can be trained respectively by using labels, thereby shortening the model training link and reducing the model training difficulty.

[0139] The word positioning unit includes a feature extraction module and a positioning module. The feature extraction module is used to extract multi-scale feature information of the image to be processed. The multi-scale feature information is input into the positioning module to determine word information and word position information corresponding to the word information in the image to be processed. In the word positioning unit, word detection and word recognition are combined into a word positioning task. The multi-scale feature information in the word positioning unit is also input into the entity grouping unit for feature fusion, so that the feature information in the first branch can be used in the second branch, thereby reducing the step of extracting feature information in the second branch and shortening the model volume of the information extraction model.

[0140] The character coding module, the visual feature fusion module and the entity grouping module are included in the entity grouping unit. The character coding feature is obtained by inputting the to-be-processed grid positioning image generated by the to-be-processed image and the word information and the word position information into the character coding module for processing. The image fusion feature information is obtained by inputting the character coding feature and the multi-scale feature into the visual feature fusion module for feature fusion. The entity detection frame and the entity information corresponding to the entity detection frame are obtained by inputting the image fusion feature information into the entity grouping module for entity grouping.

[0141] The word granularity feature extraction module, the semantic coding module and the semantic classification module are included in the entity classification unit. The word granularity feature information in the to-be-processed grid positioning image is extracted by the word granularity feature extraction module. The word granularity semantic feature information is obtained by inputting the word granularity feature information after being spliced with the position coding into the semantic coding module. The entity category information corresponding to each entity information is obtained by inputting the word granularity semantic feature information into the semantic classifier for classification.

[0142] Finally, the structured information corresponding to the to-be-processed image is further generated according to the entity information and the entity category information.

[0143] By the method provided in the disclosure, the text detection and the text recognition of the to-be-processed image are combined and processed to obtain the word information and the word position information. Meanwhile, the entity information is determined according to the word information and the word position information, and the process of generating the structured information according to the entity information is a parallel branch with the process of determining the word information and the word position information from the to-be-processed image. The two branches share the same feature extraction module, so that the whole link is relatively short. The information extraction framework is further simplified. The processing flow of the information extraction framework is short.

[0144] In addition, the method provided in the embodiment of the disclosure uses the information extraction model without calling the third-party OCR engine.

[0145] By the multi-level representation of the character granularity, the word granularity and the entity granularity, the tasks of different granularities are unified into the same information extraction framework, so that the information extraction framework is simpler and lighter.

[0146] The information extraction method provided in the disclosure is further described below by taking the application of the information extraction method to the bill image as an example in combination with FIG. 3. FIG. 3 is a process flow diagram of the information extraction method provided in one embodiment of the disclosure, which specifically includes the following steps.

[0147] Step 302: Obtain a to-be-processed bill image.

[0148] The to-be-processed bill image includes unstructured information.

[0149] Step 304: input the to-be-processed bill image into the information extraction model to obtain bill structured information output by the information extraction model.

[0150] The information extraction model determines word information and word position information in the to-be-processed bill image, determines entity information according to the word information and the word position information, and generates the bill structured information according to the entity information.

[0151] Optionally, the information extraction model comprises a word positioning unit, an entity grouping unit, and an entity classification unit.

[0152] The information extraction model comprises a word positioning unit, an entity grouping unit, and an entity classification unit.

[0153] The word positioning unit inputs the to-be-processed bill image to obtain at least one word information and word position information corresponding to each word information.

[0154] The entity grouping unit inputs the to-be-processed bill image, the at least one word information, and the word position information corresponding to each word information to obtain at least one entity bounding box and entity information corresponding to each entity bounding box.

[0155] The entity classification unit inputs each entity bounding box and each entity information to obtain entity category information corresponding to each entity information.

[0156] The information extraction model determines bill structured information according to each entity information and the entity category information corresponding to each entity information.

[0157] Optionally, the word positioning unit comprises a feature extraction module and a positioning module.

[0158] The word positioning unit inputs the to-be-processed bill image to obtain at least one word information and word position information corresponding to each word information.

[0159] The feature extraction module inputs the to-be-processed bill image to obtain multi-scale feature information corresponding to the to-be-processed bill image.

[0160] The positioning module inputs the multi-scale feature information to obtain at least one word information and word position information corresponding to each word information output by the positioning module.

[0161] Optionally, the entity grouping unit comprises a character encoding module, a visual feature fusion module, and an entity grouping module.

[0162] The entity grouping unit inputs the to-be-processed bill image, the at least one word information, and the word position information corresponding to each word information to obtain at least one entity bounding box and entity information corresponding to each entity bounding box.

[0163] generate a to-be-processed grid positioning image according to the to-be-processed bill image, the at least one word information, and word position information corresponding to each word information;

[0164] input the to-be-processed grid positioning image into a character encoding module to obtain character encoding features;

[0165] input the character encoding features and the multi-scale feature information into a visual feature fusion module to obtain image fusion feature information;

[0166] input the image fusion feature information into an entity grouping module to obtain at least one entity detection box and entity information corresponding to each entity detection box output by the entity grouping module.

[0167] Optionally, generating the to-be-processed grid positioning image according to the to-be-processed bill image, the at least one word information, and the word position information corresponding to each word information comprises:

[0168] generate a to-be-processed empty image according to the to-be-processed bill image;

[0169] determine character identification information corresponding to each word information according to the at least one word information;

[0170] add the character identification information corresponding to each word information to the to-be-processed empty image according to word position information corresponding to each word information to generate the to-be-processed grid positioning image.

[0171] Optionally, determining the character identification information corresponding to each word information according to the at least one word information comprises:

[0172] determine at least one character information corresponding to target word information, wherein the target word information is any one of the at least one word information;

[0173] determine character identification corresponding to each character information according to a preset character correspondence table;

[0174] map each character identification to three-channel feature information to generate character identification information corresponding to each character identification.

[0175] Optionally, the visual feature fusion module comprises a pyramid pooling layer and a visual feature fusion layer.

[0176] Inputting the character encoding features and the multi-scale feature information into the visual feature fusion module to obtain the image fusion feature information comprises:

[0177] input the multi-scale feature information into the pyramid pooling layer to obtain pooling feature information;

[0178] input the pooling feature information and the character encoding features into the visual feature fusion layer to obtain the image fusion feature information.

[0179] Optionally, the entity detection frame and the entity information are input into the entity classification unit to obtain entity category information corresponding to the entity information, including:

[0180] The grid positioning image to be processed, the entity detection frame and the entity information are input into the entity classification unit to obtain entity category information corresponding to the entity information.

[0181] Optionally, the entity classification unit includes a word granularity feature extraction module, a semantic encoding module and a semantic classifier.

[0182] The grid positioning image to be processed, the entity detection frame and the entity information are input into the entity classification unit to obtain entity category information corresponding to the entity information, including:

[0183] The grid positioning image to be processed is input into the word granularity feature extraction module to obtain word granularity feature information output by the word granularity feature extraction module.

[0184] The word granularity feature information is input into the semantic encoding module to obtain word granularity semantic feature information.

[0185] The word granularity semantic feature information, the entity detection frame and the entity information are input into the semantic classifier to obtain entity category information corresponding to the entity information.

[0186] Optionally, the bill structured information is determined according to the entity information and the entity category information corresponding to the entity information, including:

[0187] The target entity information and the target entity category information corresponding to the target entity information are determined.

[0188] The target entity information and the target entity category information are added to the bill structured information.

[0189] Through the method provided by the present disclosure, the text detection and text recognition of the to-be-processed bill image are combined, and the word information and the word position information are obtained by processing at the same time. At the same time, the process of determining the entity information according to the word information and the word position information and generating the structured information according to the entity information is a parallel processing of two branches of the process of determining the word information and the word position information from the to-be-processed image. The two branches share the same feature extraction module, so that the entire link is relatively short. The information extraction framework is further simplified. The processing flow of the information extraction framework is short.

[0190] Corresponding to the method embodiments described above, the present disclosure also provides information extraction device embodiments. FIG. 4 shows a structural schematic diagram of an information extraction device according to an embodiment of the present disclosure. As shown in FIG. 4, the device includes:

[0191] The receiving module 402 is configured to receive an image to be processed. The image to be processed includes unstructured information.

[0192] The extracting module 404 is configured to input the image to be processed into an information extraction model to obtain structured information output by the information extraction model. The information extraction model determines word information and word position information in the image to be processed, determines entity information according to the word information and the word position information, and generates the structured information according to the entity information.

[0193] The above is a schematic scheme of the information extraction device of the embodiment. It should be noted that the technical scheme of the information extraction device belongs to the same concept as the technical scheme of the information extraction method described above. The technical scheme of the information extraction device is not described in detail, and the description of the technical scheme of the information extraction method can be referred to.

[0194] Referring to FIG. 5, FIG. 5 shows an architecture diagram of an information extraction system according to an embodiment of the present disclosure. The information extraction system can include a client 100 and a server 200.

[0195] The client 100 is configured to send an image to be processed to the server 200. The image to be processed includes unstructured information.

[0196] The server 200 is configured to input the image to be processed into an information extraction model to obtain structured information output by the information extraction model. The information extraction model determines word information and word position information in the image to be processed, determines entity information according to the word information and the word position information, and generates the structured information according to the entity information. The server 200 sends the structured information to the client 100.

[0197] The client 100 is further configured to receive the structured information sent by the server 200.

[0198] The information extraction system can include a plurality of clients 100 and a server 200. The client 100 can be referred to as an end-side device, and the server 200 can be referred to as a cloud-side device. The plurality of clients 100 can establish a communication connection through the server 200. In the information extraction scenario, the server 200 is used to provide information extraction services between the plurality of clients 100. The plurality of clients 100 can be used as a sending end or a receiving end to realize communication through the server 200.

[0199] A user can interact with the server 200 through the client 100 to receive data sent by other clients 100 or send data to other clients 100, etc. In the information extraction scenario, the user can publish a data stream to the server 200 through the client 100. The server 200 generates structured information according to the data stream and pushes the structured information to other clients that establish a communication connection.

[0200] The client 100 and the server 200 establish a connection through a network. The network provides a medium for a communication link between the client 100 and the server 200. The network can include various connection types, such as wired, wireless communication links, or fiber optic cables, and the like. The data transmitted by the client 100 can need to be encoded, transcoded, compressed, and the like before being published to the server 200.

[0201] The client 100 can be a browser, an APP (Application), or a web application such as an H5 (HyperText Markup Language 5) application, or a light application (also known as a small program, a lightweight application), or a cloud application, and the like. The client 100 can be developed based on a software development kit (SDK) provided by the server 200 for the corresponding service, such as a real-time communication (RTC) SDK, and the like. The client 100 can be deployed in an electronic device and needs to rely on the device or some APP in the device to run, and the like. The electronic device can have a display screen and support information browsing, such as a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, and the like. Various other types of applications can also be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, and the like.

[0202] The server 200 can include servers that provide various services, such as servers that provide communication services for multiple clients, servers that provide support for models used on clients for background training, servers that process data sent by clients, and the like. It should be noted that the server 200 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. The server can also be a server of a distributed system, or a server combined with a blockchain. The server can also be a cloud server of cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, and the like. The server can also be an intelligent cloud computing server or an intelligent cloud host with artificial intelligence technology.

[0203] It should be noted that the information extraction method provided in the embodiments of the present disclosure is generally executed by the server, but in other embodiments of the present disclosure, the client can also have similar functions as the server, so as to execute the information extraction method provided in the embodiments of the present disclosure. In other embodiments, the information extraction method provided in the embodiments of the present disclosure can also be executed by the client and the server together.

[0204] FIG. 6 shows a flowchart of an information extraction method applied to a cloud-side device according to an embodiment of the present disclosure. The method is applied to a cloud-side device, and includes the following steps:

[0205] Step 602: receiving an image to be processed sent by an end-side device.

[0206] The image to be processed includes unstructured information.

[0207] Step 604: inputting the image to be processed into an information extraction model to obtain structured information output by the information extraction model.

[0208] The information extraction model determines word information and word position information in the image to be processed, determines entity information according to the word information and the word position information, and generates the structured information according to the entity information.

[0209] Step 606: sending the structured information to the end-side device.

[0210] FIG. 7 shows a structural block diagram of a computing device 700 according to an embodiment of the present disclosure. The components of the computing device 700 include but are not limited to a memory 710 and a processor 720. The processor 720 is connected with the memory 710 through a bus 730, and a database 750 is used to save data.

[0211] The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or combinations of such networks, such as the Internet. The access device 740 can include one or more of any type of network interface (for example, a network interface card (NIC)) such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC).

[0212] In one embodiment of the present disclosure, the above-mentioned components of the computing device 700 and other components not shown in FIG. 7 can also be connected to each other, for example, through a bus. It should be understood that the computing device structure block diagram shown in FIG. 7 is only for the purpose of example, and is not a limitation on the scope of the present disclosure. Those skilled in the art can add or replace other components as needed.

[0213] The computing device 700 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (for example, a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, and the like), a mobile phone (for example, a smartphone), a wearable computing device (for example, a smartwatch, smart glasses, and the like), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 700 can also be a mobile or stationary server.

[0214] The processor 720 is configured to execute computer program / instructions that implement the steps of the above information extraction method when executed by the processor.

[0215] The various embodiments in the present disclosure are described in a progressive manner, and the same or similar parts among the various embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, the computing device embodiment is basically similar to the information extraction method embodiment, and thus the description is relatively simple, and the relevant parts can be referred to the description of the information extraction method embodiment.

[0216] An embodiment of the present disclosure further provides a computer readable storage medium storing computer programs / instructions, which are executed by a processor to implement the steps of the information extraction method.

[0217] The various embodiments in the present disclosure are described in a progressive manner, and the same or similar parts among the various embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, the computer readable storage medium embodiment is basically similar to the information extraction method embodiment, and thus the description is relatively simple, and the relevant parts can be referred to the description of the information extraction method embodiment.

[0218] An embodiment of the present disclosure further provides a computer program product, which includes computer programs / instructions, which are executed by a processor to implement the steps of the information extraction method.

[0219] The above is a schematic scheme of the computer program product of the embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the information extraction method belong to the same concept, and the details of the technical scheme of the computer program product which are not described in detail can be referred to the description of the technical scheme of the information extraction method.

[0220] The above describes a specific embodiment of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims can be performed in a different order than the order in which they are recited and still accomplish desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0221] The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, software distribution medium, etc. It should be noted that the computer readable medium can include appropriate additions or subtractions according to the requirements of patent practice. For example, according to the patent practice in some regions, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0222] It should be noted that the above describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order than the order in which they are recited and still achieve the desired results. In addition, the processes depicted in the figures do not necessarily require the particular order shown or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments of the present disclosure.

[0223] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0224] The preferred embodiments of the present disclosure disclosed above are only used to help explain the present disclosure. The alternative embodiments do not describe all the details and do not limit the invention to the specific embodiments described. Obviously, according to the content of the embodiments of the present disclosure, many modifications and changes can be made. The present disclosure selects and describes these embodiments in order to better explain the principles and practical applications of the embodiments of the present disclosure, so that those skilled in the art can well understand and utilize the present disclosure. The present disclosure is limited only by the claims and their full scope and equivalents.

Claims

1. An information extraction method, comprising: Receive an image to be processed, wherein the image to be processed includes unstructured information; The image to be processed is input into an information extraction model to obtain structured information output by the information extraction model. The information extraction model determines word information and word position information in the image to be processed, determines entity information based on word information and word position information, and generates structured information based on entity information.

2. The method as described in claim 1, wherein the information extraction model includes a word localization unit, an entity grouping unit, and an entity classification unit; The image to be processed is input into the information extraction model to obtain the structured information output by the information extraction model, including: The image to be processed is input into the word localization unit to obtain at least one word information and word position information corresponding to each word information; The image to be processed, at least one word information, and word position information corresponding to each word information are input into the entity grouping unit to obtain at least one entity detection box and entity information corresponding to each entity detection box. Input each entity detection box and each entity information into the entity classification unit to obtain the entity category information corresponding to each entity information; Structured information is determined based on the information of each entity and the entity category information corresponding to each entity.

3. The method as described in claim 2, wherein the word localization unit comprises a feature extraction module and a localization module; The image to be processed is input into the word localization unit to obtain at least one word and word position information corresponding to each word, including: The image to be processed is input into the feature extraction module to obtain multi-scale feature information corresponding to the image to be processed. The multi-scale feature information is input into the localization module to obtain at least one word information and the word position information corresponding to each word information output by the localization module.

4. The method as described in claim 3, wherein the entity grouping unit comprises a character encoding module, a visual feature fusion module, and an entity grouping module; The image to be processed, at least one word information, and the word position information corresponding to each word information are input into the entity grouping unit to obtain at least one entity detection box and entity information corresponding to each entity detection box, including: Generate a grid positioning image to be processed based on the image to be processed, at least one word information, and word position information corresponding to each word information; The grid positioning image to be processed is input into the character encoding module to obtain character encoding features; The character encoding features and the multi-scale feature information are input into the visual feature fusion module to obtain image fusion feature information; The image fusion feature information is input into the entity grouping module to obtain at least one entity detection box and entity information corresponding to each entity detection box output by the entity grouping module.

5. The method as described in claim 4, comprising generating a grid positioning image to be processed based on the image to be processed, at least one word information, and word position information corresponding to each word information, comprising: Generate an empty image to be processed based on the image to be processed; Determine the character identifier information corresponding to each word based on at least one word information; Based on the word position information corresponding to each word, the character identification information corresponding to each word is added to the empty image to be processed, generating the grid positioning image to be processed.

6. The method of claim 5, wherein determining the character identifier information corresponding to each word information based on at least one word information includes: Determine at least one character information corresponding to the target word information, wherein the target word information is any one of at least one word information; Based on the preset character lookup table, determine the character identifier corresponding to each character information; Each character identifier is mapped to three-channel feature information to generate the character identifier information corresponding to each character identifier.

7. The method as described in claim 4, wherein the visual feature fusion module comprises a pyramid pooling layer and a visual feature fusion layer; The character encoding features and the multi-scale feature information are input into the visual feature fusion module to obtain image fusion feature information, including: The multi-scale feature information is input into the pyramid pooling layer to obtain pooling feature information; The pooling feature information and the character encoding feature are input into the visual feature fusion layer to obtain image fusion feature information.

8. The method as described in claim 4, wherein each entity detection box and each entity information are input into the entity classification unit to obtain entity category information corresponding to each entity information, comprising: The image of the grid to be processed, the detection boxes of each entity, and the information of each entity are input into the entity classification unit to obtain the entity category information corresponding to each entity information.

9. The method as described in claim 8, wherein the entity classification unit comprises a word-granularity feature extraction module, a semantic encoding module, and a semantic classifier; The image of the grid to be processed, the detection boxes of each entity, and the information of each entity are input into the entity classification unit to obtain the entity category information corresponding to each entity information, including: The grid positioning image to be processed is input into the word granularity feature extraction module to obtain the word granularity feature information output by the word granularity feature extraction module; The word granularity feature information is input into the semantic encoding module to obtain word granularity semantic feature information; The word-level semantic feature information, the detection boxes for each entity, and the information for each entity are input into the semantic classifier to obtain the entity category information corresponding to each entity information.

10. The method of claim 2, wherein structured information is determined based on each entity information and the entity category information corresponding to each entity information, comprising: Determine the target entity information and the target entity category information corresponding to the target entity information; The target entity information and the target entity category information are added to the structured information accordingly.

11. The method of claim 1, wherein the information extraction model is trained and generated through the following steps: Obtain sample images and corresponding sample word information, sample word location information, and sample structure information; The sample image is input into the information extraction model to obtain the predicted structured information, predicted word information and predicted word position information output by the information extraction model; The model loss value is calculated based on the sample word information, the sample word position information, the sample structure information, the predicted structure information, the predicted word information, and the predicted word position information. Adjust the model parameters of the information extraction model based on the model loss value, and continue training the information extraction model until the model training stops.

12. The method of claim 11, wherein calculating the model loss value based on the sample word information, the sample word position information, the sample structured information and the prediction structured information, the prediction word information, and the prediction word position information comprises: Calculate a first loss value based on the sample word information, the sample word position information, the predicted word information, and the predicted word position information; The second loss value is calculated based on the sample structured information and the predicted structured information; The model loss value is calculated based on the first loss value and the second loss value.

13. An information extraction method, applied to cloud-side devices, comprising: The image to be processed is sent by the receiving end device, wherein the image to be processed includes unstructured information; The image to be processed is input into an information extraction model to obtain structured information output by the information extraction model. The information extraction model determines word information and word position information in the image to be processed, determines entity information based on word information and word position information, and generates structured information based on entity information. The structured information is sent to the end-side device.

14. An information extraction device, comprising: A receiving module is configured to receive an image to be processed, wherein the image to be processed includes unstructured information; An extraction module is configured to input the image to be processed into an information extraction model to obtain structured information output by the information extraction model, wherein the information extraction model determines word information and word position information in the image to be processed, determines entity information based on the word information and word position information, and generates structured information based on the entity information.

15. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 13.

16. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 13.

17. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 13.

Images