Key information extraction method, device, electronic device and storage medium

Abstract

The present disclosure provides a key information extraction method, apparatus, electronic device, and storage medium, relating to the fields of computer technology, particularly intelligent search, computer vision, deep learning, and other technical fields. The specific implementation scheme comprises: obtaining optical character recognition results for an image and obtaining target field information to be extracted; determining a target pipeline corresponding to the image file type from among multiple pre-configured pipelines corresponding to file types; and processing the optical character recognition results using the operators included in the target pipeline to obtain an extraction result corresponding to the target field information.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to technical fields such as intelligent search, computer vision, and deep learning. Background Art

[0002] Common image materials can be roughly divided into three categories: tables, forms, and text. Each type of image material can be further divided into many subcategories, resulting in a wide variety of image materials and rich layouts. Summary of the Invention

[0003] The present disclosure provides a key information extraction method, device, electronic device and storage medium.

[0004] According to a first aspect of the present disclosure, a key information extraction method is provided, comprising:

[0005] Obtain the optical character recognition results of the image and obtain the target field information to be extracted;

[0006] Determining a target pipeline corresponding to the file type of the image from among a plurality of pre-configured pipelines corresponding to the file types;

[0007] The optical character recognition result is processed using the operator included in the target pipeline to obtain an extraction result corresponding to the target field information.

[0008] According to a second aspect of the present disclosure, there is provided a key information extraction device, comprising:

[0009] An acquisition module is used to obtain the optical character recognition result of the image and obtain the target field information to be extracted;

[0010] A determination module, configured to determine a target pipeline corresponding to the file type of the image from among pipelines corresponding to a plurality of pre-configured file types;

[0011] A processing module is used to process the optical character recognition result using the operator included in the target pipeline to obtain an extraction result corresponding to the target field information.

[0012] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the above-mentioned key information extraction method.

[0016] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the above-mentioned key information extraction method.

[0017] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the above-mentioned key information extraction method when executed by a processor.

[0018] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to better understand the present invention and do not constitute a limitation of the present invention.

[0020] Figure 1 This is a flow chart of a key information extraction method provided according to an embodiment of the present disclosure;

[0021] Figure 2a This is a schematic diagram of a document image provided according to an embodiment of the present disclosure;

[0022] Figure 2b is a schematic diagram of an image piece with a seal provided according to an embodiment of the present disclosure;

[0023] Figure 2c is a schematic diagram of a table-type image provided according to an embodiment of the present disclosure;

[0024] Figure 2d is a schematic diagram of a form-type image provided according to an embodiment of the present disclosure;

[0025] Figure 3 This is a flow chart of an extraction model training method provided according to an embodiment of the present disclosure;

[0026] Figure 4 This is a flowchart of the training and online reasoning of the extraction model provided in accordance with an embodiment of the present disclosure;

[0027] Figure 5 is a schematic diagram of the processing process of the strategic recall operator provided according to an embodiment of the present disclosure;

[0028] Figure 6 This is a schematic diagram of a seal determination logic process provided according to an embodiment of the present disclosure;

[0029] Figure 7is a schematic diagram of a picture of a missing square seal provided according to an embodiment of the present disclosure;

[0030] Figure 8 This is a schematic diagram of the processing process of the entity highlight coordinate calculation operator provided according to an embodiment of the present disclosure;

[0031] Figure 9 is a schematic diagram of a structure of an authentication framework provided according to an embodiment of the present disclosure;

[0032] Figure 10 This is a structural diagram of a key information extraction device provided according to an embodiment of the present disclosure;

[0033] Figure 11 This is a first block diagram of an electronic device for implementing the key information extraction method of the embodiment of the present disclosure;

[0034] Figure 12 This is a second block diagram of an electronic device used to implement the key information extraction method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0036] Common image files include pictures and PDF scans. These can be roughly divided into three categories: tables, forms, and documents. Each category can be further subdivided into many subcategories, resulting in a wide variety of image files and formats.

[0037] Table: has clear row and column information, header information, etc.

[0038] Forms: There is no complex table structure. Form cells contain large text contents, and key information may also be outside the form. The length of the text obtained by Optical Character Recognition (OCR) usually does not exceed 1,000 characters.

[0039] Documents: These documents consist of multiple images, and the text obtained by OCR is usually longer than 1,000 characters, such as contracts and agreements.

[0040] The key information in the image includes key fields and key values (kv). When extracting the kv from the above image, there are the following pain points:

[0041] 1) The material complexity is high, and the model for extracting kv is demanding.

[0042] As mentioned above, images come in a wide variety of types and formats, such as various types of invoices. Furthermore, a high percentage of key fields require customization, and audit scenarios place high demands on the effectiveness and performance of the model's key value extraction.

[0043] The requirements for the effect are as follows: calculated based on the average accuracy of the extracted kv, the minimum accuracy is 80%, and the accuracy is considered stable only when it reaches 90%, which also includes errors in optical character recognition (OCR).

[0044] Performance requirements: The response time for extracting KV from a single request model should be controlled within 100 milliseconds.

[0045] 2) The cost of model development is high.

[0046] The model development process for extracting the KV of a material includes model selection, model building, post-processing, and highlighting. Currently, the average development time for a material model is approximately four man-days. For materials with high complexity and demanding KV extraction, the model development time can be around eight man-days.

[0047] 3) Model selection and optimization are inconvenient.

[0048] There are numerous models and schemes for extracting KVs, and materials are diverse and rich in layout. To improve the KV extraction performance of the model for different materials, a lot of time needs to be spent on model selection and model tuning, especially in the later stages of optimization.

[0049] In order to reduce the R&D cost and ensure the effect and performance of KV extraction of complex materials, the embodiment of the present disclosure provides a key information extraction method, such as Figure 1 As shown, the following steps are included:

[0050] Step S11, obtaining the optical character recognition result of the image and obtaining the target field information to be extracted;

[0051] Step S12, determining a target pipeline corresponding to the file type of the image file from among a plurality of pipelines corresponding to pre-configured file types;

[0052] Step S13: Use the operator included in the target pipeline to process the optical character recognition result to obtain an extraction result corresponding to the target field information.

[0053] In the technical solution provided by the embodiments of the present disclosure, the operators required to process images of each file type are integrated into the corresponding pipeline for each file type. After obtaining the OCR results of the corresponding image, the OCR results are processed using the operators included in the corresponding pipeline to obtain the corresponding key information extraction results. In the technical solution provided by the embodiments of the present disclosure, the required operators are pre-integrated into the corresponding pipeline, eliminating the need to design independent extraction models for diverse and complex materials. This reduces R&D costs while ensuring the effectiveness and performance of KV extraction for complex materials.

[0054] For ease of description, the following description is made using electronic devices as the execution entity, which is not intended to be limiting.

[0055] In step S11, the image may be a picture or a PDF scan, etc. The image may be in the form of a table, a form, or a document. The OCR result may also be referred to as a perception result.

[0056] In the disclosed embodiment, the OCR results of the image can be stored in a storage device with storage capabilities, such as a public cloud, a private cloud service, a software development kit (SDK), or a Redis database. The electronic device obtains the OCR results of the image by calling an interface provided by the storage device.

[0057] Electronic devices can also capture images and then use machine vision technology to perform OCR on the images to obtain OCR results. The methods for obtaining OCR results for images vary depending on the file type, as follows:

[0058] a) The file type of the image is document, such as Figure 2a In this case, the step of obtaining the OCR result of the image may be: performing OCR on the image to obtain each text in the image, the probability of each text, the coordinates of each text, and the seal detection result; each text, the probability of each text, the coordinates of each text, and the seal detection result are the OCR result; the seal detection result includes at least one of the following: the position of the seal, the text in the seal, the shape of the seal, etc. The shape of the seal may include but is not limited to a round seal, an oval seal, a square seal, a diamond seal, etc. The image with the seal is as follows: Figure 2b shown.

[0059] In the embodiment of the present disclosure, the electronic device can obtain the seal detection result through the seal detection service;

[0060] b) The file type of the image is a table, such as Figure 2cIn this case, the step of obtaining the OCR result of the image may be: performing OCR on the image to obtain the text in each cell in the table in the image, the row and column information of each cell, and the text outside the table, where the text in each cell, the row and column information of each cell, and the text outside the table are the OCR results.

[0061] c) The file type of the image is form, such as Figure 2d In this case, the steps for obtaining the OCR result of the image may include: performing OCR on the image to obtain the text in each cell in the form, the row and column information of each cell, the text outside the form, and the seal detection result; and splicing the text in each cell, and the spliced text in each cell, the row and column information of each cell, the text outside the form, and the seal detection result are the OCR result.

[0062] In the embodiment of the present disclosure, the cells of the form contain large paragraphs of text content. For each cell, the electronic device can recognize multiple lines of text. By splicing these multiple lines of text, a line of text can be obtained, which is the spliced text of the cell. The spliced text is used to facilitate processing of the cell content.

[0063] In the embodiment of the present disclosure, the target field information is the field information of the kv that needs to be extracted. The field information may include but is not limited to the model extraction field, the uniform resource locator (URL) of the model reasoning service, and the strategy extraction field.

[0064] Model extraction fields are key fields that the model needs to extract. Model extraction fields can include, but are not limited to, the Chinese name of the field, the English name of the field, and the field data type. Using the extraction model, you can extract the key values of most fields. The extraction model can be a neural network-based Named Entity Recognition (NER) model or other neural network-based models, without limitation.

[0065] The strategy extraction fields are the key fields that need to be extracted by the extraction strategy. These key fields are the key fields that cannot be extracted by the extraction model, such as determining whether there is a seal.

[0066] In the above step S12, the electronic device is pre-configured with pipelines corresponding to multiple file types. As described in the above step S11, the file types of the image may include documents, forms and tables. Accordingly, the electronic device can pre-configure pipelines corresponding to documents, pipelines corresponding to forms and pipelines corresponding to tables.

[0067] After obtaining the OCR result of the image, the electronic device determines the pipeline corresponding to the file type of the image from the pre-configured pipelines corresponding to multiple file types as the target pipeline.

[0068] In the embodiment of the present disclosure, the pipeline corresponding to each file type includes one or more operators.

[0069] For images with diverse scenes, complex material layouts, and rich data types, there will be many repeated modules in the KV extraction process. The electronic device abstracts these modules as an operator, each operator implements an independent function, and each independent operator can be called a standardized operator. In the embodiment of the present disclosure, the electronic device can connect the various operators included in each pipeline in series in a preset order to form different pipelines to extract KV from different materials. This is suitable for KV extraction of materials with different layouts in different scenes. There is no need to build a targeted KV extraction framework, which saves R&D costs.

[0070] For different file types, the operators included in the pipeline are different, as follows:

[0071] a) The pipeline corresponding to the table may include at least one of the following operators: a table data structure construction operator, a table data structure merging operator, a row and column extraction operator, a cell positioning operator, and an entity highlight coordinate calculation operator.

[0072] b) The pipeline corresponding to the form and document may include at least one of the following operators: a text coordinate alignment operator, a model extraction operator, a strategy recall operator, a business formatting operator, and an entity highlight coordinate calculation operator.

[0073] After obtaining the target pipeline, the electronic device executes step S13, uses the operators included in the target pipeline to process the OCR results, and obtains the extraction results corresponding to the target field information, wherein the extraction results may include but are not limited to, key values (i.e., entities), page numbers of images where the key values are located in the image, and outputs of each operator after processing the OCR results.

[0074] After obtaining the extraction results, the electronic device can output the OCR results. For example, the electronic device can publish the extraction results through the FastAPI backend service framework so that users can view the extraction results, integrate the information, and perform corresponding business processing.

[0075] In the embodiment of the present disclosure, when the file type of the image piece is a document, the image piece includes multiple pictures. The pipeline corresponding to the document is used to process the OCR results of each picture separately to obtain the extraction results of each picture. Then, the extraction results of multiple pictures are aggregated to obtain the extraction results of the document.

[0076] In some embodiments, when the target pipeline includes a text coordinate alignment operator, in the above step S13, the electronic device can use the text coordinate alignment operator to process the OCR results according to the following steps: aligning the single characters and the coordinates of the single characters in the OCR results; wherein the alignment processing includes at least one of the following processing: removing invalid characters in the OCR results; converting punctuation marks into punctuation marks of a preset language.

[0077] In the embodiment of the present disclosure, the text coordinate alignment operator is applicable to the pipeline corresponding to documents and forms; the electronic device uses the OCR result as the input of the text coordinate alignment operator, and the text coordinate alignment operator processes the OCR result, aligns the single characters and the coordinates of the single characters in the OCR result one by one, removes invalid characters such as spaces, line breaks, tabs, etc. in the OCR result, and uniformly converts punctuation marks into punctuation marks of a preset language, etc., and outputs the full-text text string corresponding to the OCR result and the coordinates of each text character.

[0078] The electronic device uses the text coordinate alignment operator to process the OCR results, which can make the pipeline adaptable to the OCR results of documents or forms from different manufacturers and different interfaces, expand the scope of use of the embodiment of the present disclosure, and further reduce R&D costs.

[0079] In some embodiments, when the target field information includes a first target key field and a target URL, and the target pipeline includes a model extraction operator, the first target key field is the key field that the extraction model needs to extract, the target URL is the uniform resource locator of the extraction model online reasoning service, and the extraction model at the target URL is the first target extraction model. In the above step S13, the electronic device can use the model extraction operator to process the optical character recognition result according to the following steps: converting the optical character recognition result into a target document, where the target document is a document in a data format supported by the first target extraction model; inputting the target document and the first target key field into the first target extraction model to obtain the probability of each entity corresponding to the first target key field; and using the entity with the maximum probability as the first target key value corresponding to the first target key field. The online reasoning service is a service for extracting kv.

[0080] In the disclosed embodiment, the electronic device can configure a URL in the form of a service interface specified by the key information extraction inference engine, and use the URL to distinguish different extraction models. The extraction model can be a model obtained by secondary development of the Universal Information Extraction (UIE) model or the GlobalPointer model based on the paddle natural language processing (Natural Language Processing, paddleNLP) algorithm, using pre-trained models such as the ERNIE3.0 model. The model extraction operator is applicable to the pipeline corresponding to documents and forms.

[0081] The electronic device can use the OCR result and the target URL as input to the model extraction operator. The model extraction operator can process the OCR result according to the first target extraction model and output the kv corresponding to the OCR result, that is, the first target key field and the first target key value pair. The first target key value may include the start coordinates, end coordinates and probability that the entity corresponding to the first target key field is the real entity corresponding to the first target key field, etc.

[0082] In the disclosed embodiments, electronic devices can extract key information using NER, build a model extraction service, decouple it from the cognitive framework, perform offline training externally, and publish online reasoning services for the cognitive framework to call. Here, the cognitive framework is the electronic device that performs the key information extraction mentioned above.

[0083] In the embodiment of the present disclosure, in order to facilitate tracing the picture to which the kv belongs and further confirm whether the kv is correct, the electronic device can also use the identity (id) of the picture in the image as the input of the model extraction operator, and the output of the model extraction operator is associated with the picture id.

[0084] In some embodiments, to improve data processing efficiency, the electronic device may convert the first target extraction model from a dynamic graph to a static graph to obtain a second target extraction model. Based on this, the electronic device inputs the target document and the first target key field into the second target extraction model to obtain the probability of each entity corresponding to the first target key field.

[0085] In the disclosed embodiments, electronic devices can use the FastAPI backend service framework to provide online inference services for extraction models. In the disclosed embodiments, the configuration parameters support online inference services for central processing units (CPUs) and graphics processing units (GPUs). To improve data processing efficiency and reduce CPU burden, GPUs can be used to perform online inference services.

[0086] In some embodiments, the present disclosure also provides an extraction model training method, such as Figure 3 As shown, the following steps may be included:

[0087] Step S31: Obtain a sample text set, where the sample text set includes a plurality of annotated sample texts.

[0088] In the disclosed embodiment, the sample text set includes multiple sample texts that can be divided into a training set, a validation set, and a test set. The ratio of the sample texts included in the training set, validation set, and test set can be set according to actual needs, for example, 8:1:1. The representation of each sample text can be annotated with nested entities. For example, for the text "Peking University", the institution can be annotated with "Peking University" and the place name can be annotated with "Beijing".

[0089] Step S32: using a network search method to determine a preset number of group model parameters.

[0090] In the disclosed embodiments, model parameters may include hyperparameters such as learning rate, training rounds, and batch size (min-batch gradient descent may be used). The electronic device may perform a network search on the model parameters to obtain a preset number of sets of model parameters. In one example, the preset number may be 27, i.e., the number of network searches is 3*3*3=27, and the electronic device may obtain 27 sets of model parameters.

[0091] Step S33: Use the multiple annotated sample texts to train the extraction model that has used each set of model parameters separately.

[0092] The electronic device assigns each set of model parameters to the extraction model to obtain a preset number of extraction models; and uses multiple annotated sample texts to train the preset number of extraction models that have used the model parameters.

[0093] In an embodiment of the present disclosure, the electronic device may perform fine-tuning training on the extraction model based on parameters of a pre-trained model, such as the above-mentioned model parameters.

[0094] Step S34: Determine an extraction model for extracting key information from a preset number of extraction models obtained through training.

[0095] After the training is completed, the electronic device obtains a preset number of extraction models through training. According to a preset rule, the electronic device determines an extraction model from the preset number of extraction models obtained through training for subsequent online extraction of key information.

[0096] In some embodiments, the electronic device can determine the recall rate and precision rate of each extraction model obtained through training; based on the recall rate and precision rate of each extraction model obtained through training, determine the first measurement indicator of each extraction model obtained through training, wherein the largest first measurement indicator among the obtained first measurement indicators is used as the second measurement indicator; and use the extraction model corresponding to the second measurement indicator as the extraction model for extracting key information.

[0097] In the embodiment of the present application, the type of the first measurement indicator can be set according to actual needs. In one example, the first measurement indicator can be the F1 indicator. The F1 indicator can be used to measure the accuracy of the model, which takes into account both the precision and recall of the model.

[0098] The recall rate, precision rate and F1 of the above extraction model can be calculated by the following formula:

[0099] Recall rate = (number of fields identified and correctly identified / number of fields identified) * 100%;

[0100] Precision = (number of correctly identified fields / number of annotations) * 100%;

[0101] F1 = (2*recall*precision) / (recall+precision).

[0102] In the embodiment of the present disclosure, after the electronic device obtains the extraction model for extracting key information through training, it can provide the ID of the extraction model, the weight file of the extraction model, and evaluation indicators (such as the above-mentioned recall rate, precision rate and F1), etc., for easy backtracking.

[0103] The following combination Figure 4 The training and online inference process of the extraction model shown in the figure explains the processing process of the model extraction operator.

[0104] Step S41: Obtain the OCR result of the image.

[0105] During offline training:

[0106] Step S42: annotate the text in the OCR result.

[0107] In the embodiment of the present disclosure, the electronic device can use the doccano annotation tool to annotate the text in the OCR result to annotate all key-value information. Nested entity annotation is supported here.

[0108] Step S43: construct a training set, a validation set, and a test set.

[0109] In the embodiment of the present disclosure, the electronic device can use Prompt data to construct a training set, a validation set, and a test set, and convert the data format into a data format supported by the extraction model to obtain a data format document specified as input by the extraction model.

[0110] Step S44: model parameter adjustment, model fine-tuning, model evaluation and release.

[0111] 1) Model parameter tuning: Use grid search for hyperparameters such as learning rate, training rounds, and batch size.

[0112] 2) Model fine-tuning: Using the labeled data provided in step S43, the extraction model is fine-tuned based on the parameters of the pre-trained model.

[0113] 3) Model evaluation and release:

[0114] a, Model evaluation: Calculate recall, precision and F1;

[0115] b. Model release: From multiple sets of extraction models obtained through training, the extraction model with the highest F1 score is selected as the final extraction model and released.

[0116] Online reasoning stage:

[0117] Step S45: construct online prediction data.

[0118] In the embodiment of the present disclosure, the electronic device can use Prompt data to construct online prediction data, and convert the data format into a data format supported by the extraction model to obtain a data format document specified as input by the extraction model.

[0119] Step S46, reasoning.

[0120] 1) Converting a dynamic graph to a static graph: Convert the extracted model published in step S44 to a static graph. Then, use the FastAPI backend service framework to provide online inference services for the extracted model. Configuration parameters support inference services for both CPUs and GPUs.

[0121] 2) Inference deployment:

[0122] a. Input the unlabeled data provided in step S45 into the extraction model, predict the results of all entities, and return the probability of the corresponding entity.

[0123] The probability can be calculated as follows: for an entity, the extraction model obtains two probabilities, one is the probability of the starting position, recorded as start_prob, and the other is the probability of the ending position, recorded as end_prob. The final output probability is start_prob*end_prob, which is taken as the probability prob of the entity.

[0124] b. Reason about the text to get all the kv information.

[0125] In the disclosed embodiment, a key may correspond to multiple key values, and each value has a probability prob. For the same key, the electronic device can sort the probabilities of the values corresponding to the key, select the value with the highest probability as the entity of the key, and return the entity.

[0126] In the disclosed embodiment, the inconsistency between training and reasoning in a document is resolved. That is, a single page of text is used as a sample for training, but predictions are made for each page during reasoning, and the entities predicted for each page are finally aggregated. In addition, the entity with the highest probability is selected, ensuring that for a key, there is ultimately only one value.

[0127] In some embodiments, when the target field information includes a second target key field that needs to be extracted by the extraction strategy, and the target pipeline includes a strategy recall operator, the electronic device can use the strategy recall operator to process the OCR results according to the following steps: obtain the target extraction strategy corresponding to the second target key field; use the target extraction strategy to process the OCR results to obtain the second target key value corresponding to the second target key field.

[0128] In the disclosed embodiment, the extraction strategy can be a regular expression or other strategy. The electronic device can obtain the extraction strategy corresponding to the second target key field as the target extraction strategy, use the OCR result as the input of the target extraction strategy, and obtain the corresponding extraction result. This can compensate for the extraction of fields that the extraction model cannot extract, ensuring the reliability of key information extraction.

[0129] In some embodiments, if the second target key field is a text extraction field, that is, the second target key field is a field from which text needs to be extracted, the step of using the target extraction strategy to process the OCR result and obtain the second target key value corresponding to the second target key field can be: according to a preset matching mode, use the target extraction strategy to match the OCR result and the second target key field to obtain the second target key value corresponding to the second target key field.

[0130] The preset matching mode may indicate the number of matches, such as one or more matches. Here, a match once means that the match ends when a text matching the second target key field is found. The second target key value is the text matching the second target key field.

[0131] When the second target key field is a text extraction field, the target extraction strategy can be a regular expression. The electronic device can recall common key information through regular expressions. In the embodiment of the present disclosure, the strategy recall operator can be a built-in date regular recall operator. The electronic device uses regular expressions, matching patterns and OCR results as inputs of the strategy recall operator. After the strategy recall operator processes the input, it outputs matching text (i.e., the second target key value), start coordinates and end coordinates, etc. Taking the case where the strategy recall operator can be a built-in date regular recall operator as an example, the processing process of the strategy recall operator is as follows: Figure 5 shown.

[0132] Step S51: input the regular expression, matching pattern and OCR result into the strategy recall operator.

[0133] Step S52, determine whether the first match is successful; if so, use the re.search function to return the matching result; if not, use the re.finditer function to return an iterator to continue matching.

[0134] Among them, the re.search function can be expressed as re.search(hit_word, text), and the re.finditer function can be expressed as re..finditer(hit_word, text), where hit_word represents a regular expression and text represents the OCR result, that is, text.

[0135] Step S53: output the matching text, start coordinates and end coordinates.

[0136] In the disclosed embodiment, the problem of insufficient extraction recall rate of the extraction model is compensated, and key information is recalled through additional rules.

[0137] In some embodiments, if the second target key field is a seal determination field and the OCR result includes a seal detection result, the step of using the target extraction strategy to process the OCR result and obtain the second target key value corresponding to the second target key field can be: determining the first center point coordinates of the minimum bounding box of each second target key field and the second center point coordinates of the minimum bounding box of the target seal indicated by the seal detection result; calculating the Euclidean distance between the second center point coordinates and each first center point coordinate; and determining the target seal as the seal of the second target key field corresponding to the minimum Euclidean distance.

[0138] In the embodiment of the present disclosure, the seal determination logic is as follows: Figure 6As shown, the electronic device uses the seal detection result and the seal owner keyword (i.e., the second target key field) as input to the target extraction strategy. The target extraction strategy calculates the center point of the minimum bounding box, obtains the keyword center point coordinates and the seal center point coordinates, such as the first center point coordinates and the second center point coordinates, calculates the Euclidean distance between the two, and determines based on the Euclidean distance that the keyword closest to the seal is the seal owner. It then outputs whether each seal owner has a seal. Among them, the seal owner keywords can be "Party A (official seal)" and "Party B (official seal)", etc.

[0139] For example, Figure 7 In the picture of the missing seal shown, the seal detection result shows only one seal, namely the legal person seal, and the keywords of the seal owners are "Party A (official seal)" and "Party B (official seal)". The electronic device calculates the coordinates (x1, y1) of the center point of the minimum bounding box of "Party A (official seal)" in the picture; calculates the coordinates (x2, y2) of the center point of the minimum bounding box of "Party B (official seal)" in the picture; and calculates the coordinates (x, y) of the center point of the minimum bounding box of the seal in the picture. The electronic device calculates the Euclidean distance 1 from (x, y) to (x1, y1), and calculates the Euclidean distance 2 from (x, y) to (x2, y2). After calculation, it is found that the Euclidean distance 1 from (x, y) to (x1, y1) is smaller. Therefore, the electronic device can determine that the seal at (x, y) is the legal person seal of Party A indicated by (x1, y1), that is, the legal person seal of Party B is missing.

[0140] In the disclosed embodiment, when some fields of an image document involve seal detection, there may be a situation where a seal is missing. At this time, it is necessary to determine which party has not stamped. Through the extraction strategy, it can be determined which party has not stamped, further compensating for the problem of insufficient extraction recall rate of the extraction model.

[0141] In some embodiments, when the target pipeline includes a business formatting operator, the electronic device can use the business formatting operator to process the OCR result according to the following steps: processing the OCR result to obtain a third target key value; formatting the third target key value according to a preset data structure.

[0142] In the embodiment of the present disclosure, the formatting process may include but is not limited to:

[0143] 1) Convert the uppercase amount format to lowercase amount format and retain two decimal places;

[0144] 2) Clean lowercase amounts, filter out OCR errors, and retain two decimal places;

[0145] 3) Format percentage data such as interest rates and tax rates, retaining four decimal places

[0146] 4) Format various date formats into "year-month-day" date format or "month-day" date format.

[0147] In the embodiment of the present disclosure, some numerical values and date-type values that can be standardized are formatted to form a shareable structured data, which is convenient for storage and can be used even if a different programming language or business scenario is used, thereby expanding the application scenarios of the embodiment of the present disclosure. In addition, the key values are formatted to facilitate the review of materials and fields on the engineering side. For example: the date of material A is: October 1, 2020, and the date of material B is 2020.10.1, then when both are formatted as 2020-10-01, the review time is equal.

[0148] In some embodiments, when the target pipeline includes a construction operator for a table-type data structure, the electronic device may use the construction operator for the table-type data structure to convert the OCR result into data of the table-type data structure according to the following steps.

[0149] In the disclosed embodiment, the table in the OCR result can be a data structure such as HTML or JSON, and the table data structure can be a Pandas DataFrame data structure. The electronic device converts the OCR result into data in the DataFrame data structure. This unifies the table data structure and facilitates subsequent data operations.

[0150] In some embodiments, when the target pipeline further includes a merging operator for a table-type data structure, the electronic device may use the merging operator for the table-type data structure to process the OCR result according to the following steps: merging data of multiple table-type data structures.

[0151] In the disclosed embodiment, the OCR results may include tables across multiple pages. The electronic device may use a merge operator of a table-type data structure to process tables that span multiple pages and merge tables across multiple pages. For example, each page of the table is constructed to obtain data in a DataFrame data structure. For a multi-page table, the electronic device obtains data in multiple DataFrame data structures. Using a merge operator of a table-type data structure, the electronic device may merge the data in these multiple DataFrame data structures. Compared to merging OCR results across multiple pages, merging data in a table-type data structure is easier for electronic devices to implement, reducing the requirements for the electronic device.

[0152] In some embodiments, when the target pipeline also includes a row and column extraction operator, the electronic device can use the row and column extraction operator to process the OCR results according to the following steps: extract data in the rows and / or columns indicated by the target field information from the data in the tabular data structure.

[0153] In an embodiment of the present disclosure, the electronic device can use the key fields of the row and / or column (such as column name, row index, etc.) and the data of the tabular data structure as the input of the row and column extraction operator to extract the data of the entire row or column from the data of the tabular data structure.

[0154] In some embodiments, when the target pipeline also includes a cell locating operator, the electronic device can use the cell locating operator to process the OCR result according to the following steps: extracting data in the cell indicated by the target field information from the data in the tabular data structure.

[0155] In the disclosed embodiments, an electronic device can use row and column key fields (such as column name, row index, etc.) and data of a tabular data structure as inputs to a cell location operator to extract data of a specified cell from the data of the tabular data structure. The specified cell is the cell indicated by the row and column key fields, where the row and column key fields are information included in the target field information.

[0156] In some embodiments, when the target pipeline includes an entity highlight coordinate calculation operator, the electronic device can use the entity highlight coordinate calculation operator to process the OCR results according to the following steps: extract the target entity corresponding to the target field information in the OCR result; determine the coordinates of the minimum bounding box corresponding to the target entity in the image as the highlight display coordinates.

[0157] In an embodiment of the present disclosure, the electronic device can use the target field information as the input of the entity highlight coordinate calculation operator. The entity highlight coordinate calculation operator calculates the coordinates of the minimum bounding box corresponding to the target entity indicated by the target field information. The coordinates can be the coordinates of the four vertices of the minimum bounding box, which can locate the absolute position of the target entity in the image.

[0158] In the disclosed embodiment, when the entity highlight coordinate calculation operator is preceded by other operators in series, the input to the entity highlight coordinate calculation operator may also include the output of other operators, such as the entities extracted by the extraction model, entities extracted by the extraction strategy, business formatting results, and OCR results output by the aforementioned operators. To accurately locate the position of the target entity in the image, the electronic device may also use the ID of the image containing the target entity as the input to the entity highlight coordinate calculation operator, so that the output of the entity highlight coordinate calculation operator is associated with the image ID, thereby facilitating the tracing of the image to which the key value belongs.

[0159] The processing logic of the entity highlight coordinate calculation operator can be found in Figure 8As shown. The electronic device inputs the OCR results and the extracted entity information into the entity highlight coordinate calculation operator. Among them, the OCR results include the coordinates of each character and the text indicated by the character, and the entity information includes the entity's start coordinates, end coordinates, and key fields and key value pairs, such as Figure 8 The entity highlight coordinate calculation operator finds the character coordinates of the entity, determines line breaks and page breaks, and based on the determination results, calculates the minimum bounding box of the entity. For example, if the determination result is that there is no line break or page break, the entity's minimum bounding box is calculated; if the determination result is that there is a line break or page break, two minimum bounding boxes of the entity are calculated, one minimum bounding box for each partial entity on each line or page. The entity highlight coordinate calculation operator inputs the coordinates of the minimum bounding box.

[0160] In the embodiment of the present disclosure, the electronic device can be used for front-end highlighting and can realize cross-row highlighting, meeting the demand for highlighting key information in the picture from the front-end page.

[0161] The following combination Figure 9 The authentication framework shown illustrates the key information extraction method provided by the embodiment of the present disclosure. Figure 9 The authentication framework shown includes an input module, a pipeline, and an output module.

[0162] 1) Input module:

[0163] a. Input computer vision-based perception results, such as document OCR results, form OCR results, table OCR results, seal detection results, etc. The perception results can be obtained by the perception module and stored in the Redis database.

[0164] b. Enter the field information of the material to be extracted, which may include the fields that need to be extracted by the extraction strategy, the fields that need to be extracted by the extraction model, and the URL of the extraction model service.

[0165] 2) Pipeline: including form pipeline, document pipeline, and form pipeline.

[0166] The document pipeline and form pipeline include text coordinate alignment operators, model extraction operators, strategy recall operators, business formatting operators, and entity highlighting coordinate calculation operators.

[0167] The table pipeline includes operators for constructing table-type data structures, merging table-type data structures, extracting rows and columns, locating cells, and calculating entity highlight coordinates.

[0168] 3) Output module: extracts and formats key information from the returned materials for structured storage and front-end highlighting.

[0169] Specific output information may include:

[0170] a. Image page number: corresponds to the image page number of the kv, which can be used to distinguish multiple pages of images. The image page number can also become the image ID.

[0171] b. KV extraction results: original results of fields extracted by the cognitive framework and business formatted results of fields;

[0172] c, kv highlight coordinates: the coordinates of the bounding box of the original result of the field (i.e., the target entity) in the image, used for front-end highlighting;

[0173] d, Recognition probability: the probability of entities extracted by the extraction model.

[0174] above Figure 9 The description is relatively simple. Please refer to the above related description for details, which will not be repeated here.

[0175] The technical solution provided by the embodiments of the present disclosure can be used to extract key information from images and store the extracted key information in a structured manner, supporting visual display on the front-end page. It can process various image types such as forms, tables, and documents, use natural language processing methods to extract key information from the above images, and use computer vision methods to calculate the coordinates of the key information, so that the extracted key information can be highlighted on the front-end page. This can save 70% of labor costs in the delivery of document intelligence-related projects while maintaining industry-leading key information extraction results.

[0176] Corresponding to the above-mentioned key information extraction method, the embodiment of the present disclosure also provides a key information extraction device, such as Figure 10 As shown, including:

[0177] The acquisition module 101 is used to obtain the optical character recognition result of the image and obtain the target field information to be extracted;

[0178] The determination module 102 is used to determine the target pipeline corresponding to the file type of the image file from among the pipelines corresponding to the pre-configured multiple file types;

[0179] The processing module 103 is used to process the optical character recognition result using the operators included in the target pipeline to obtain the extraction result corresponding to the target field information.

[0180] In some embodiments, when the file type of the image is a document, the acquisition module 101 may be specifically configured to:

[0181] Perform optical character recognition on the image to obtain each text in the image, the probability of each text, the coordinates of each text, and the seal detection result; each text, the probability of each text, the coordinates of each text, and the seal detection result are the optical character recognition results;

[0182] The seal detection result includes at least one of the following: the position of the seal, the text inside the seal, and the shape of the seal.

[0183] In some embodiments, when the file type of the image is a table, the acquisition module 101 may be specifically configured to:

[0184] Optical character recognition is performed on the image to obtain the text in each cell in the table in the image, the row and column information of each cell, and the text outside the table. The text in each cell, the row and column information of each cell, and the text outside the table are the optical character recognition results.

[0185] In some embodiments, when the file type of the image is a form, the acquisition module 101 may be specifically configured to:

[0186] Perform optical character recognition on the image to obtain the text in each cell of the form in the image, the row and column information of each cell, the text outside the form, and the seal detection result; splice the text in each cell, and the spliced text in each cell, the row and column information of each cell, the text outside the form, and the seal detection result are the optical character recognition results;

[0187] The seal detection result includes at least one of the following: the position of the seal, the text inside the seal, and the shape of the seal.

[0188] In some embodiments, the plurality of file types includes forms, sheets, and documents;

[0189] The pipeline corresponding to the table includes at least one of the following operators: a table data structure construction operator, a table data structure merging operator, a row and column extraction operator, a cell positioning operator, and an entity highlight coordinate calculation operator;

[0190] The pipeline corresponding to forms and documents includes at least one of the following operators: text coordinate alignment operator, model extraction operator, strategy recall operator, business formatting operator, and entity highlight coordinate calculation operator.

[0191] In some embodiments, when the target pipeline includes a text coordinate alignment operator, the processing module 103 may be specifically configured to:

[0192] Aligning the coordinates of individual characters in the optical character recognition results;

[0193] The alignment process includes at least one of the following processes:

[0194] Remove invalid characters from optical character recognition results;

[0195] Convert punctuation marks to the punctuation marks of the preset language.

[0196] In some embodiments, when the target field information includes a first target key field and a target uniform resource locator, and the target pipeline includes a model extraction operator, the first target key field is a key field that needs to be extracted by the extraction model, the target uniform resource locator is a uniform resource locator of the extraction model online inference service, and the extraction model at the target uniform resource locator is the first target extraction model, the processing module 103 can be specifically used to:

[0197] Converting the optical character recognition result into a target document, where the target document is a document in a data format supported by the first target extraction model;

[0198] Input the target document and the first target key field into the first target extraction model to obtain the probability of each entity corresponding to the first target key field;

[0199] The entity with the highest probability is used as the first target key value corresponding to the first target key field.

[0200] In some embodiments, the key information extraction device may further include:

[0201] A graph conversion module is used to convert the first target extraction model into a static graph to obtain a second target extraction model;

[0202] The processing module 103 may be specifically configured to:

[0203] The target document and the first target key field are input into the second target extraction model to obtain the probability of each entity corresponding to the first target key field.

[0204] In some embodiments, the key information extraction device may further include a training module for training an extraction model, which may be specifically used to:

[0205] Obtaining a sample text set, where the sample text set includes multiple annotated sample texts;

[0206] Using a network search method, determine the preset number of group model parameters;

[0207] Using multiple annotated sample texts, train the extraction model using each set of model parameters separately;

[0208] From a preset number of extraction models obtained through training, an extraction model for extracting key information is determined.

[0209] In some embodiments, the training module may be used to:

[0210] Determine the recall and precision of each extracted model trained;

[0211] Determine a first measurement index of each extraction model obtained through training based on the recall rate and precision rate of each extraction model obtained through training, wherein the largest first measurement index among the obtained first measurement indexes is used as the second measurement index;

[0212] The extraction model corresponding to the second measurement indicator is used as the extraction model for extracting key information.

[0213] In some embodiments, when the target field information includes a second target key field that needs to be extracted by the extraction strategy, and the target pipeline includes a strategy recall operator, the processing module 103 may be specifically configured to:

[0214] Obtain the target extraction strategy corresponding to the second target key field;

[0215] The optical character recognition result is processed using a target extraction strategy to obtain a second target key value corresponding to the second target key field.

[0216] In some embodiments, if the second target key field is a text extraction field, the processing module 103 may be specifically configured to:

[0217] According to a preset matching mode, the optical character recognition result and the second target key field are matched using a target extraction strategy to obtain a second target key value corresponding to the second target key field.

[0218] In some embodiments, if the second target key field is a seal determination field and the optical character recognition result includes a seal detection result, the processing module 103 may be specifically configured to:

[0219] Determine the coordinates of a first center point of a minimum bounding box of each second target key field, and the coordinates of a second center point of the minimum bounding box of the target seal indicated by the seal detection result;

[0220] Calculate the Euclidean distance between the coordinates of the second center point and the coordinates of each first center point;

[0221] The target seal is determined as the seal of the second target key field corresponding to the minimum Euclidean distance.

[0222] In some embodiments, when the target pipeline includes a service formatting operator, the processing module 103 may be specifically configured to:

[0223] Processing the optical character recognition result to obtain a third target key value;

[0224] The third target key value is formatted according to the preset data structure.

[0225] In some embodiments, when the target pipeline includes a construction operator for a table-type data structure, the processing module 103 may be specifically configured to:

[0226] Convert the optical character recognition results into data in a tabular data structure.

[0227] In some embodiments, when the target pipeline further includes a merge operator of a table-type data structure, the processing module is specifically configured to:

[0228] Merge data from multiple tabular data structures.

[0229] In some embodiments, when the target pipeline further includes a row-column extraction operator, the processing module 103 may be specifically configured to:

[0230] Extract data in the row and / or column indicated by target field information from data in a tabular data structure.

[0231] In some embodiments, when the target pipeline further includes a cell location operator, the processing module 103 may be specifically configured to:

[0232] Extracts data from the cell indicated by the target field information from data in a table data structure.

[0233] In some embodiments, when the target pipeline includes an entity highlight coordinate calculation operator, the processing module 103 may be specifically configured to:

[0234] Extracting the target entity corresponding to the target field information in the optical character recognition result;

[0235] Determine the coordinates of the minimum bounding box corresponding to the target entity in the image as the highlight coordinates.

[0236] In some embodiments, the operators included in the pipeline are connected in series according to a preset order.

[0237] In some embodiments, the key information extraction device may further include:

[0238] Output module, used to output the extraction results.

[0239] In the technical solution provided by the embodiments of the present disclosure, the operators required to process images of each file type are integrated into the corresponding pipeline for each file type. After obtaining the OCR results of the corresponding image, the OCR results are processed using the operators included in the corresponding pipeline to obtain the corresponding key information extraction results. In the technical solution provided by the embodiments of the present disclosure, the required operators are pre-integrated into the corresponding pipeline, eliminating the need to design independent extraction models for diverse and complex materials. This reduces R&D costs while ensuring the effectiveness and performance of KV extraction for complex materials.

[0240] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0241] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0242] Figure 11 A schematic block diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0243] like Figure 11 As shown, the device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. Various programs and data required for the operation of the device 1100 can also be stored in the RAM 1103. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0244] Various components in device 1100 are connected to I / O interface 1105, including an input unit 1106, such as a keyboard and mouse; an output unit 1107, such as various types of displays and speakers; a storage unit 1108, such as a magnetic disk and optical disk; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. Communication unit 1109 allows device 1100 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0245] The computing unit 1101 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1101 performs the various methods and processes described above, such as the key information extraction method. For example, in some embodiments, the key information extraction method can be implemented as a computer software program that is tangibly included in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the computing unit 1101, one or more steps of the key information extraction method described above can be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to execute the key information extraction method in any other appropriate manner (eg, by means of firmware).

[0246] The present disclosure also provides an electronic device, such as Figure 12 As shown, including:

[0247] at least one processor 121; and

[0248] A memory 122 in communication with the at least one processor 121; wherein,

[0249] The memory 122 stores instructions that can be executed by the at least one processor 121. The instructions are executed by the at least one processor so that the at least one processor can perform any of the key information extraction methods described above.

[0250] An embodiment of the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute any of the key information extraction methods described above.

[0251] An embodiment of the present disclosure further provides a computer program product, including a computer program, which implements any of the above-described key information extraction methods when executed by a processor.

[0252] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0253] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0254] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0255] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0256] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0257] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0258] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0259] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A key information extraction method, comprising: Obtain the optical character recognition results of the image and obtain the target field information to be extracted; Determining a target pipeline corresponding to the file type of the image from among a plurality of pre-configured pipelines corresponding to the file types; Using the operator included in the target pipeline, processing the optical character recognition result to obtain an extraction result corresponding to the target field information; When the target field information includes a first target key field and a target uniform resource locator, and the target pipeline includes a model extraction operator, the first target key field is a key field that needs to be extracted by the extraction model, the target uniform resource locator is a uniform resource locator of an online inference service for the extraction model, and the extraction model at the target uniform resource locator is the first target extraction model, the optical character recognition result is processed using the model extraction operator according to the following steps: Converting the optical character recognition result into a target document, where the target document is a document in a data format supported by the first target extraction model; Inputting the target document and the first target key field into the first target extraction model to obtain the probability of each entity corresponding to the first target key field; Taking the entity with the highest probability as the first target key value corresponding to the first target key field; The method further comprises: Convert the first target extraction model from a dynamic image to a static image to obtain a second target extraction model; The step of inputting the target document and the first target key field into the first target extraction model to obtain the probability of each entity corresponding to the first target key field includes: Inputting the target document and the first target key field into a second target extraction model to obtain the probability of each entity corresponding to the first target key field; The extraction model is trained by the following steps: Acquire a sample text set, wherein the sample text set includes a plurality of sample texts with annotations; Using a network search method, determine the preset number of group model parameters; Using the multiple labeled sample texts, respectively training the extraction model using each set of model parameters; From a preset number of extraction models obtained through training, an extraction model for extracting key information is determined.

2. The method according to claim 1, wherein When the file type of the image is a document, the step of obtaining an optical character recognition result of the image includes: Performing optical character recognition on the image to obtain each text in the image, the probability of each text, the coordinates of each text, and a seal detection result; the each text, the probability of each text, the coordinates of each text, and the seal detection result are optical character recognition results; The seal detection result includes at least one of the following: the position of the seal, the text in the seal, and the shape of the seal.

3. The method according to claim 1, wherein When the file type of the image is a table, the step of obtaining an optical character recognition result of the image includes: Optical character recognition is performed on the image to obtain text in each cell in the table in the image, row and column information of each cell, and text outside the table, where the text in each cell, row and column information of each cell, and text outside the table are optical character recognition results.

4. The method according to claim 1, wherein When the file type of the image is a form, the step of obtaining an optical character recognition result of the image includes: Performing optical character recognition on the image to obtain text in each cell of the form in the image, row and column information of each cell, text outside the form, and a seal detection result; splicing the text in each cell, and the spliced text in each cell, row and column information of each cell, text outside the form, and seal detection result are optical character recognition results; The seal detection result includes at least one of the following: the position of the seal, the text in the seal, and the shape of the seal.

5. The method according to claim 1, wherein The plurality of file types include forms, schedules, and documents; The pipeline corresponding to the table includes at least one of the following operators: a table data structure construction operator, a table data structure merging operator, a row and column extraction operator, a cell positioning operator, and an entity highlight coordinate calculation operator; The pipeline corresponding to the form and document includes at least one of the following operators: a text coordinate alignment operator, a model extraction operator, a strategy recall operator, a business formatting operator, and an entity highlight coordinate calculation operator.

6. The method according to claim 5, wherein: When the target pipeline includes a text coordinate alignment operator, the text coordinate alignment operator is used to process the optical character recognition result according to the following steps: Aligning the coordinates of the individual characters in the optical character recognition result; The alignment process includes at least one of the following processes: Removing invalid characters from the optical character recognition result; Convert punctuation marks to the punctuation marks of the preset language.

7. The method according to claim 1, wherein The step of determining an extraction model for extracting key information from a preset number of extraction models obtained through training includes: Determine the recall and precision of each extracted model trained; Determine a first measurement index of each extraction model obtained through training based on the recall rate and precision rate of each extraction model obtained through training, wherein the largest first measurement index among the obtained first measurement indexes is used as the second measurement index; The extraction model corresponding to the second measurement indicator is used as the extraction model for extracting key information.

8. The method according to claim 5, wherein When the target field information includes a second target key field that needs to be extracted by the extraction strategy, and the target pipeline includes a strategy recall operator, the strategy recall operator is used to process the optical character recognition result according to the following steps: Obtaining a target extraction strategy corresponding to the second target key field; The optical character recognition result is processed using the target extraction strategy to obtain a second target key value corresponding to the second target key field.

9. The method according to claim 8, wherein If the second target key field is a text extraction field, the step of using the target extraction strategy to process the optical character recognition result to obtain a second target key value corresponding to the second target key field includes: According to a preset matching mode, the target extraction strategy is used to match the optical character recognition result and the second target key field to obtain a second target key value corresponding to the second target key field.

10. The method according to claim 8, wherein If the second target key field is a seal determination field, and the optical character recognition result includes a seal detection result, the step of using the target extraction strategy to process the optical character recognition result to obtain a second target key value corresponding to the second target key field includes: Determining the coordinates of a first center point of a minimum bounding box of each of the second target key fields and the coordinates of a second center point of the minimum bounding box of the target seal indicated by the seal detection result; Calculating the Euclidean distance between the coordinates of the second center point and the coordinates of each first center point; The target seal is determined to be the seal of the second target key field corresponding to the minimum Euclidean distance.

11. The method according to claim 5, wherein: When the target pipeline includes a service formatting operator, the service formatting operator is used to process the optical character recognition result according to the following steps: Processing the optical character recognition result to obtain a third target key value; The third target key value is formatted according to a preset data structure.

12. The method according to claim 5, wherein: When the target pipeline includes a construction operator for a table-type data structure, the construction operator for the table-type data structure is used to process the optical character recognition result according to the following steps: The optical character recognition result is converted into data of the table data structure.

13. The method according to claim 12, wherein: When the target pipeline further includes a merge operator of a table-type data structure, the optical character recognition result is processed using the merge operator of the table-type data structure according to the following steps: Merge the data of the plurality of table-type data structures.

14. The method according to claim 12, wherein: When the target pipeline further includes a row-column extraction operator, the row-column extraction operator is used to process the optical character recognition result according to the following steps: The data in the row and / or column indicated by the target field information is extracted from the data in the tabular data structure.

15. The method according to claim 12, wherein: When the target pipeline further includes a cell positioning operator, the cell positioning operator is used to process the optical character recognition result according to the following steps: The data in the cell indicated by the target field information is extracted from the data in the table data structure.

16. The method according to claim 5, wherein When the target pipeline includes an entity highlight coordinate calculation operator, the entity highlight coordinate calculation operator is used to process the optical character recognition result according to the following steps: Extracting a target entity corresponding to the target field information in the optical character recognition result; The coordinates of the minimum bounding box corresponding to the target entity in the image are determined as the highlight coordinates.

17. The method according to any one of claims 1 to 16, wherein: The operators included in the pipeline are connected in series according to a preset order.

18. The method according to any one of claims 1 to 16, further comprising: The extraction result is output.

19. A key information extraction device comprising: An acquisition module is used to obtain the optical character recognition result of the image and obtain the target field information to be extracted; A determination module, configured to determine a target pipeline corresponding to the file type of the image from among pipelines corresponding to a plurality of pre-configured file types; a processing module, configured to process the optical character recognition result using the operator included in the target pipeline to obtain an extraction result corresponding to the target field information; Wherein, when the target field information includes a first target key field and a target uniform resource locator, and the target pipeline includes a model extraction operator, the first target key field is a key field that needs to be extracted by the extraction model, the target uniform resource locator is a uniform resource locator of an online inference service of the extraction model, and the extraction model at the target uniform resource locator is the first target extraction model, the processing module is specifically used to: Converting the optical character recognition result into a target document, where the target document is a document in a data format supported by the first target extraction model; Inputting the target document and the first target key field into the first target extraction model to obtain the probability of each entity corresponding to the first target key field; Taking the entity with the highest probability as the first target key value corresponding to the first target key field; The device further comprises: A graph conversion module, configured to convert the first target extraction model into a static graph to obtain a second target extraction model; The processing module is specifically used to: Inputting the target document and the first target key field into a second target extraction model to obtain the probability of each entity corresponding to the first target key field; The device also includes a training module for training to obtain an extraction model, specifically for: Acquire a sample text set, wherein the sample text set includes a plurality of sample texts with annotations; Using a network search method, determine the preset number of group model parameters; Using the multiple labeled sample texts, respectively training the extraction model using each set of model parameters; From a preset number of extraction models obtained through training, an extraction model for extracting key information is determined.

20. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 18.

21. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-18.

22. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 18.

Images