Key information extraction method and device, equipment and storage medium

Abstract

The invention discloses a key information extraction method and device, equipment and a storage medium, and relates to the technical field of information extraction, and the method comprises the steps: carrying out the character detection of a to-be-extracted picture data set, and obtaining a character detection result; performing character recognition on the to-be-extracted picture data set based on the character detection result to obtain a character recognition result; performing mode alignment on the character recognition result and a manual labeling result corresponding to the to-be-extracted picture data set to obtain mode alignment metadata; and obtaining a key information extraction result corresponding to the to-be-extracted picture data set based on the mode alignment metadata, a preset text information classification model and a preset text information relation matching model. By applying the technical scheme, the technical problem that in the prior art, when OCR key information is extracted, due to the fact that the accuracy of information labeling through a manual labeling mode and / or a detection model labeling mode is not high, the accuracy of key information extraction is not high is solved.

Description

Technical Field

[0001] The present application relates to the technical field of information extraction, and in particular to a key information extraction method, device, equipment and storage medium. Background Art

[0002] OCR (Optical Character Recognition) refers to the process of detecting and identifying the shape, color and other information of printed or handwritten characters, and then translating this information into computer text using character recognition methods. There are many industrial applications in the field of OCR, from the simple detection and recognition of text information in the past to the need to extract and integrate key information from many recognized text characters today, namely OCR Key Information Extraction (KIE), such as extracting key information such as name and address from ID cards and converting them into computer text.

[0003] In actual application, the process of OCR key information extraction usually includes three stages: annotation, modeling and prediction. In the annotation stage, users can select the key information to be extracted and provide the contact information between the information, and then these annotation information will be input into the modeling process; in the modeling stage, two models will be trained to classify the recognized text information and match the text information with category labels, and finally obtain the information extraction result through the prediction process. At present, information annotation can usually be performed by manual annotation or detection model annotation. However, there are certain defects in information annotation by manual annotation and / or detection model annotation, which leads to low accuracy of information annotation, and ultimately affects the accuracy of key information extraction. Summary of the invention

[0004] The main purpose of this application is to provide a key information extraction method, device, equipment and storage medium, aiming to solve the technical problem in the prior art that when performing OCR key information extraction, the accuracy of key information extraction is low due to the low accuracy of information annotation through manual annotation and / or detection model annotation.

[0005] To achieve the above purpose, the present application proposes a key information extraction method, which comprises:

[0006] Perform text detection on the image data set to be extracted to obtain text detection results;

[0007] Based on the text detection result, text recognition is performed on the image data set to be extracted to obtain a text recognition result;

[0008] Performing pattern alignment on the text recognition result and the manual annotation result corresponding to the image data set to be extracted to obtain pattern alignment metadata;

[0009] Based on the pattern alignment metadata, the preset text information classification model, and the preset text information relationship matching model, the key information extraction result corresponding to the image data set to be extracted is obtained.

[0010] In one embodiment, the step of performing pattern alignment on the text recognition result and the manual annotation result corresponding to the image data set to be extracted to obtain pattern alignment metadata includes:

[0011] Determine first metadata based on the text recognition result and the manual annotation result corresponding to the to-be-extracted image data set, wherein the first metadata is text content in which the text recognition result and the manual annotation result have the same text content and the degree of overlap of the annotation boxes is greater than a preset overlap threshold;

[0012] Determine second metadata in the text recognition result, where the second metadata is text content that exists in all image samples in the text recognition result and that appears a number of times that meets a preset number requirement;

[0013] Acquire third metadata based on the difference between the text recognition result and the manual annotation result;

[0014] The first metadata, the second metadata, and the third metadata are merged to obtain pattern-aligned metadata.

[0015] In one embodiment, the step of determining the first metadata based on the text recognition result and the manual annotation result corresponding to the image data set to be extracted includes:

[0016] Obtaining a first annotation box in the text recognition result and a second annotation box in the manual annotation result corresponding to the image data set to be extracted;

[0017] Determine a first target annotation frame and a second target annotation frame that have the same text content in the first annotation frame and the second annotation frame;

[0018] Determine an area of ​​region overlap and an area of ​​region union between the first target annotation box and the second target annotation box;

[0019] Determine the overlap between the first target annotation box and the second target annotation box based on the region overlap area and the region union area;

[0020] If the overlap degree of the annotation boxes is greater than a preset overlap degree threshold, the text contents in the first target annotation box and the second target annotation box are recorded as first metadata.

[0021] In one embodiment, the step of acquiring third metadata based on the difference between the text recognition result and the manual annotation result includes:

[0022] Determining a difference recording method according to the difference content between the text recognition result and the manual annotation result;

[0023] Recording the difference content in the difference recording manner to obtain a difference metadata record;

[0024] The difference metadata records are deduplicated using a preset difference deduplication rule to obtain third metadata.

[0025] In one embodiment, the step of determining the difference recording method according to the difference between the text recognition result and the manual annotation result includes:

[0026] Determine a detection frame in the image data set to be extracted based on the text recognition result;

[0027] Determining a difference type corresponding to a difference content between the text recognition result and the manual annotation result based on the detection frame and the annotation frame corresponding to the manual annotation result;

[0028] A difference recording method is determined according to the difference type.

[0029] In one embodiment, the step of obtaining the key information extraction result corresponding to the image data set to be extracted based on the pattern alignment metadata, the preset text information classification model, and the preset text information relationship matching model includes:

[0030] Based on the pattern alignment metadata, content splitting is performed on the difference content in the text recognition result to obtain a splitting correction record;

[0031] Inputting the split and revised records into a preset text information classification model to obtain information classification prediction results;

[0032] Correcting the content of the question type text of the information classification prediction result by using the pattern alignment metadata to obtain a question correction record;

[0033] Inputting the problem correction record into a preset text information relationship matching model to obtain an information relationship matching prediction result;

[0034] The answer type text of the information relationship matching prediction result is content-corrected through the pattern alignment metadata to obtain the key information extraction result corresponding to the image data set to be extracted.

[0035] In one embodiment, before the step of performing text detection on the image data set to be extracted and obtaining the text detection result, the step further includes:

[0036] Determine the coordinate position of the annotation box in the image data set to be extracted, the annotation box category and the annotation box link information;

[0037] Generate a plurality of annotation point records based on the image identifier in the to-be-extracted image data set, the annotation box coordinate position, the annotation box category and the annotation box link information;

[0038] The image data set to be extracted is annotated based on each annotation point record to obtain a manual annotation result corresponding to the image data set to be extracted.

[0039] In addition, to achieve the above purpose, the present application also proposes a key information extraction device, the device comprising:

[0040] A text detection module is used to perform text detection on the image data set to be extracted and obtain text detection results;

[0041] A text recognition module, used to perform text recognition on the image data set to be extracted based on the text detection result to obtain a text recognition result;

[0042] A pattern alignment module, used to perform pattern alignment on the text recognition result and the manual annotation result corresponding to the image data set to be extracted, so as to obtain pattern alignment metadata;

[0043] The information extraction module is used to obtain the key information extraction result corresponding to the image data set to be extracted based on the pattern alignment metadata, the preset text information classification model, and the preset text information relationship matching model.

[0044] In addition, to achieve the above-mentioned purpose, the present application also proposes a key information extraction device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the key information extraction method described above.

[0045] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the key information extraction method described above are implemented.

[0046] The present application provides a key information extraction method, which discloses performing text detection on a picture data set to be extracted to obtain a text detection result; performing text recognition on the picture data set to be extracted based on the text detection result to obtain a text recognition result; performing pattern alignment on the text recognition result and the manual annotation result corresponding to the picture data set to be extracted to obtain pattern alignment metadata; obtaining the key information extraction result corresponding to the picture data set to be extracted based on the pattern alignment metadata, a preset text information classification model, and a preset text information relationship matching model; compared with the low accuracy of information annotation through manual annotation and / or detection model annotation in the prior art, resulting in low accuracy of key information extraction, the present invention can perform pattern alignment on the text recognition result and the manual annotation result of the picture data set to be extracted to obtain pattern alignment metadata, and obtain the key information extraction result based on the pattern alignment metadata, the preset text information classification model, and the preset text information relationship matching model, so that the manual annotation result and the model recognition result can be combined and the key information can be extracted through the model, thereby solving the technical problem in the prior art that when performing OCR key information extraction, the accuracy of key information extraction is low due to the low accuracy of information annotation through manual annotation and / or detection model annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0049] Figure 1 A flowchart of the first embodiment of the key information extraction method of the present application is provided;

[0050] Figure 2 This is a schematic diagram of the overall process of OCR key information extraction in the key information extraction method of this application;

[0051] Figure 3 A flowchart of adding post-processing OCR key information extraction to the key information extraction method of this application;

[0052] Figure 4 This is an example diagram of character detection in the key information extraction method of this application;

[0053] Figure 5This is a diagram showing the stages of OCR key information extraction in the key information extraction method of this application;

[0054] Figure 6 This is an example of manually annotating an image in the key information extraction method of this application;

[0055] Figure 7 This is an example diagram of model annotation of images in the key information extraction method of this application;

[0056] Figure 8 The overall annotation and training flow chart of the key information extraction method for this application;

[0057] Fig. 9 A schematic diagram of a process flow provided for the second embodiment of the key information extraction method of the present application;

[0058] Fig.10 A diagram showing the types of differences in the key information extraction method of this application;

[0059] Fig.11 A flowchart of the third embodiment of the key information extraction method of the present application is provided;

[0060] Fig.12 The overall prediction flow chart of the key information extraction method of this application;

[0061] Fig.13 This is a schematic diagram of the module structure of the key information extraction device of the embodiment of the present application;

[0062] Fig.14 Schematic diagram of the device structure of the hardware operating environment involved in the key information extraction method in the embodiment of the present application.

[0063] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0064] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0065] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0066] The main solution of the embodiment of the present application is: perform text detection on the image data set to be extracted to obtain a text detection result; perform text recognition on the image data set to be extracted based on the text detection result to obtain a text recognition result; perform pattern alignment on the text recognition result and the manual annotation result corresponding to the image data set to be extracted to obtain pattern alignment metadata; obtain the key information extraction result corresponding to the image data set to be extracted based on the pattern alignment metadata, a preset text information classification model, and a preset text information relationship matching model.

[0067] In the prior art, when extracting OCR key information, information annotation is usually performed by manual annotation or detection model annotation. If manual annotation is used, the information obtained in the text detection process during prediction will not be combined in a way that humans understand, which may easily lead to differences between the text combination form used during modeling and the text combination form during prediction, thereby affecting the model prediction effect. If detection model annotation is used, the detection model cannot recognize human intentions, resulting in low annotation accuracy.

[0068] The present application provides a solution, which enables pattern alignment of text recognition results and manual annotation results of a picture data set to be extracted, obtains pattern alignment metadata, and obtains key information extraction results based on the pattern alignment metadata, a preset text information classification model, and a preset text information relationship matching model, so that key information can be extracted by combining the manual annotation results and the model recognition results, thereby solving the technical problem in the prior art that when performing OCR key information extraction, the accuracy of key information extraction is not high due to the low accuracy of information annotation through manual annotation and / or detection model annotation.

[0069] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a key information extraction device, etc. The following takes the key information extraction device as an example (hereinafter referred to as the device) to illustrate this embodiment and the following embodiments.

[0070] Based on this, the present application embodiment provides a key information extraction method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the key information extraction method of the present application.

[0071] In this embodiment, the key information extraction method includes steps S10 to S40:

[0072] Step S10: Perform text detection on the image data set to be extracted to obtain text detection results.

[0073] In practical applications, refer to Figure 2 , Figure 2 Schematic diagram of the overall process of OCR key information extraction in the key information extraction method of this application. Figure 2 As shown in FIG. 1 , the process of OCR key information extraction usually includes three processes: text detection, text recognition and information extraction. The information extraction process is sometimes divided into two processes: semantic entity recognition (SER) and relation extraction (RE). Text detection can be used to identify the area with text in the image; text recognition can be used to identify the image area where the text is detected as computer text; semantic entity recognition can be used to classify the detected text content, which can usually be divided into several fixed categories, such as title, question, answer, and others. Specifically, "name" can be identified as a question, and "Zhang San" can be identified as an answer. Relation extraction can be used to match the relationship between texts to form information with specific understandable meanings. For example, "name" and "Zhang San" are matched, and "Zhang San" is the content corresponding to the key information "name".

[0074] It should be noted that in actual application scenarios, many problems are often encountered, and these problems require specific processing methods to improve recognition accuracy. For example, the distance between characters is too large, resulting in the content that should have been merged together being recognized as multiple separate objects; blurred images result in characters being recognized as similar characters, etc. Therefore, in actual industrial applications, a "post-processing" process is often added at the end. Figure 3 , Figure 3 This is a flowchart of adding post-processing OCR key information extraction to the key information extraction method of this application. In practical applications, the detection results of text detection in the key information extraction process are usually several detection boxes, each of which may contain several characters. These detection boxes are not arranged and combined in a way that humans understand, but are combined by the detection model according to certain position rules. Figure 4 , Figure 4 This is an example diagram of character detection in the key information extraction method of this application. Figure 4The content should be combined into two detection boxes according to human understanding, namely "certification agency" and "XXX company", but the model usually cannot understand this potential meaning and can only combine by line or character spacing, resulting in the detection of four text boxes. At this time, in order to improve the accuracy of the overall task, a post-processing process can be added to recombine the contents of the four detection boxes into appropriate content. At this time, due to the lack of prompt information required by the user for the result, it is often necessary to manually use the program to implement this customized post-processing process, which requires a lot of manpower costs. If the user is required to provide prompt information for post-processing, the user's learning cost will increase. At the same time, if the user cannot understand the specific technical means of post-processing, the accuracy may not be guaranteed. Therefore, the key information extraction method proposed in this scheme can detect text on the image by combining the model with the manual annotation results and the model recognition results, thereby reducing labor costs and improving the accuracy and efficiency of image detection.

[0075] In practical applications, refer to Figure 5 , Figure 5 This is a diagram showing the stages of OCR key information extraction in the key information extraction method of this application. Figure 5 As shown in the figure, the process of OCR key information extraction can include three stages: annotation, modeling, and prediction. During annotation, the user can select the key information to be extracted and provide the connection information between the information. Then, these annotation information will be input into the modeling process. The model will be trained to obtain two models, one is the SER model, which is used to classify the recognized text information; the other is the RE model, which is used to match the relationship between the text information with category labels. In the prediction process, it will go through the following steps respectively: Figure 3 The five processes shown finally result in the extraction.

[0076] It should be understood that the above-mentioned image data set to be extracted can be a data set consisting of any images that need to be extracted for key information. In this embodiment, K images that need to be extracted for key information can be collected in advance to construct the image data set S to be extracted, wherein each image in the image data set S to be extracted has a unique image id number.

[0077] It should be noted that the above text detection result is a result composed of the image information (including: image id, detection box coordinate position, etc.) obtained after performing text detection on the image. In this embodiment, a text detection model can be used to perform text detection on the image dataset S to be extracted to obtain a text detection result, wherein the text detection result includes K detection items, each detection item includes a number of detection point records, and each detection point record is in a dictionary format, recording the image id and the coordinate position information of the detection box.

[0078] Step S20: performing text recognition on the image data set to be extracted based on the text detection result to obtain a text recognition result.

[0079] It should be noted that the text recognition for the image data set to be extracted can be a process of identifying the text content information in the image, and accordingly, the above text recognition result can be a result composed of the text content information in each image in the image data set to be extracted. In this embodiment, a text recognition model can be used to perform text recognition on the image data set S to be extracted. The text detection result in step S10 is required for text recognition, and then the text recognition result is obtained, wherein the text recognition result can include K recognition entries, each recognition entry includes a number of text recognition result records, and the format of each text recognition result record is a dictionary format, which records the image id, the coordinate position of the detection box, and the recognized text content information.

[0080] Step S30: performing pattern alignment on the text recognition result and the manual annotation result corresponding to the image data set to be extracted to obtain pattern alignment metadata.

[0081] It should be understood that manual labeling can be a process of labeling images according to human labelers’ understanding of human patterns. Figure 6 , Figure 6 This is an example of manually annotating an image in the key information extraction method of this application. The content to be annotated can be annotated as follows using the manual annotation method: Figure 6 The form shown, where Figure 6 The blue annotation box is the question, the yellow annotation box is the answer, and the red link line indicates the relationship between the two. The annotation content of this annotation mode is combined according to human understanding, even if the content of the same annotation box spans across lines, is too wide or too tight, etc. In addition, information that users do not need to extract does not need to be annotated, such as Figure 6"Irrelevant content" in the image. Therefore, modeling with this form of annotated content is simple and intuitive, and the annotation workload is relatively small. However, since the information obtained during the text detection process during prediction will not be combined in a way that humans understand, the text combination form used during modeling will be different from the text combination form used during prediction, which will affect the model prediction effect. At the same time, since the annotator did not annotate the unannotated information, this part of information is missing during the modeling process, but during the prediction process, all the text in the picture will be detected during text detection, so this part of information needs to be identified during prediction. Since this part of information is missing during the modeling process, the SER model's ability to judge categories will be reduced. In addition, if the user annotates information that does not need to be extracted during annotation, the workload of the annotator will increase significantly, and the additional annotated information will also affect the model prediction effect due to the difference in the form of the results obtained from text detection.

[0082] It should be noted that, referring to Figure 7 , Figure 7 This is an example of model annotation of images in the key information extraction method of this application. The method of using the detection model for annotation can annotate the content to be annotated as follows: Figure 7 The form shown, where Figure 7 The category of the blue labeled box is question, the category of the yellow labeled box is answer, the category of the purple labeled box is others, and the red link line indicates the relationship between the two. The labeled boxes of this annotation mode are generally detected by the text detection model used during reasoning, and then the annotator labels and connects the detection results. Its advantage is that it is labeled according to the detection characteristics of the detection model itself, so that the text combination form used in modeling and the text combination form used in prediction have a high consistency, which can improve the model prediction performance. However, since the annotation information does not contain the form of the final result, for example Figure 7 The annotation content does not provide information that "certification" and "institution" are part of the same part, so the final model output result can only be identified and reassembled by manually customized post-processing. In addition, due to the fragmentation of the test results, the annotator needs to annotate a large number of connecting lines, which reduces the annotation efficiency; due to the fragmentation of the test results, if one part is misidentified by the SER model, the entire result will be wrong. For example Figure 7 If the category prediction of any of the five digits in the amount is wrong, the position association information cannot be used to correct the result; since the text detection model cannot recognize human intentions, content that should not be merged may be merged into one detection frame, such as Figure 7It can be seen that the effect of image annotation using a single manual annotation method or a model annotation method is not good. Therefore, this solution can combine the manual annotation results with the model detection results, thereby improving the image annotation effect and further improving the subsequent extraction of key image information.

[0083] It should be noted that the above manual annotation results may be the results obtained after annotating the image in a human-understandable annotation mode. In this embodiment, the manual annotation results may include K annotation entries, each of which includes a number of annotation point records, where each annotation point record is in a dictionary format, recording the image id, the coordinate position of the annotation box, the category of the annotation box, and the link information between the annotation boxes. Since the annotation process is performed in a human-understandable manner, the obtained annotation information includes the result format that the user expects to obtain when the final prediction output is obtained. Figure 6 Taking the "certification body" in as an example, the result format expected by the user is: {"certification body": "XXX company"}.

[0084] It should be noted that the above-mentioned pattern alignment metadata may be metadata obtained by pattern-aligning text contents in the text recognition result and the manual annotation result.

[0085] Step S40: obtaining a key information extraction result corresponding to the image data set to be extracted based on the pattern alignment metadata, the preset text information classification model, and the preset text information relationship matching model.

[0086] It should be noted that the preset text information classification model may be a model for classifying recognized text information, such as a SER model. The preset text information relationship matching model may be a model for matching relationships between text information with category labels, such as a RE model.

[0087] In this embodiment, the metadata information in the pattern alignment metadata can be used to train the SER model and the RE model, and the key information of the pictures in the picture data set to be extracted can be extracted through the trained SER model and the RE model.

[0088] In the specific implementation, refer to Figure 8 , Figure 8 This is the overall annotation and training flow chart of the key information extraction method of this application. Figure 8As shown, when extracting key information from the image dataset S to be extracted, firstly, the image dataset S to be extracted can be manually annotated using a human-understandable pattern to obtain a manual annotation result. At the same time, a text detection model can be used to perform text detection on the image dataset S to be extracted to obtain a text detection result, and a text recognition model can be used to perform text recognition on the image dataset S to be extracted based on the text detection result to obtain a text recognition result C. Then, the manual annotation result and the text recognition result are pattern-aligned to obtain pattern alignment metadata M. Then, the device can use the metadata information in the pattern alignment metadata M to split the content with differences in the text recognition result C to obtain a split correction record D, and input the split correction record D into the SER model for model training to obtain a SER model, and then use the metadata information in the pattern alignment metadata M to correct the content with differences in the text of the problem type in the split correction record D to obtain a problem correction record E, and input the problem correction record E into the RE model for model training to obtain an RE model. Finally, the key information in the image dataset S to be extracted can be extracted through the trained SER model and RE model to obtain a key information extraction result.

[0089] The present embodiment provides a key information extraction method, which discloses performing text detection on a picture data set to be extracted to obtain a text detection result; performing text recognition on the picture data set to be extracted based on the text detection result to obtain a text recognition result; performing pattern alignment on the text recognition result and the manual annotation result corresponding to the picture data set to be extracted to obtain pattern alignment metadata; obtaining the key information extraction result corresponding to the picture data set to be extracted based on the pattern alignment metadata, a preset text information classification model, and a preset text information relationship matching model; compared with the low accuracy of information annotation through manual annotation and / or detection model annotation in the prior art, resulting in low accuracy of key information extraction, the present embodiment can perform pattern alignment on the text recognition result and the manual annotation result of the picture data set to be extracted to obtain pattern alignment metadata, and obtain the key information extraction result based on the pattern alignment metadata, the preset text information classification model, and the preset text information relationship matching model, so that the manual annotation result and the model recognition result can be combined and the key information can be extracted through the model, thereby solving the technical problem in the prior art that the accuracy of key information extraction is low due to the low accuracy of information annotation through manual annotation and / or detection model annotation when performing OCR key information extraction.

[0090] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Fig. 9 , Fig. 9A flow chart of the second embodiment of the key information extraction method of the present application is provided.

[0091] In this embodiment, step S30 includes steps S301 to S304:

[0092] Step S301: Determine first metadata based on the text recognition result and the manual annotation result corresponding to the image data set to be extracted, wherein the first metadata is the text content in the text recognition result and the manual annotation result that is the same and the annotation box overlap is greater than a preset overlap threshold.

[0093] It should be noted that the above-mentioned annotation box overlap can be a value used to characterize the degree of overlap of the annotation boxes in the text recognition result and the manual annotation result. The above-mentioned preset overlap threshold can be a threshold used to characterize that the degree of overlap of the annotation boxes in the text recognition result and the manual annotation result is large. For example, in this embodiment, the preset overlap threshold can be set to 75%. At this time, the above-mentioned first metadata, i.e., the text content recorded in the text recognition result is consistent with that recorded in the manual annotation result and the overlap value of the annotation boxes of the two is greater than 75%.

[0094] Specifically, the step S301 includes: obtaining a first annotation box in the text recognition result and a second annotation box in the manual annotation result corresponding to the image data set to be extracted; determining a first target annotation box and a second target annotation box having the same text content in the first annotation box and the second annotation box; determining a region overlap area and a region union area between the first target annotation box and the second target annotation box; determining an annotation box overlap degree between the first target annotation box and the second target annotation box based on the region overlap area and the region union area; if the annotation box overlap degree is greater than a preset overlap degree threshold, recording the text content in the first target annotation box and the second target annotation box as first metadata.

[0095] It should be understood that the first annotation box and the second annotation box may be annotation boxes for annotating text content in the text recognition result and the manual annotation result, respectively. Correspondingly, the first target annotation box and the second target annotation box may be annotation boxes for the same text content in the text recognition result and the manual annotation result, respectively.

[0096] It can be understood that the above-mentioned region overlap area may be the area of ​​the two overlapping regions of the first target marking box and the second target marking box; the above-mentioned region union area may be the union area of ​​the two regions of the first target marking box and the second target marking box.

[0097] In this embodiment, the overlap of the first target annotation frame and the second target annotation frame can be divided by the area of ​​the union of the two regions to obtain the above-mentioned annotation frame overlap, that is, the area of ​​the region overlap can be divided by the area of ​​the union of the regions to obtain the annotation frame overlap. Then, the device can compare the annotation frame overlap with a preset overlap threshold, and if the annotation frame overlap is greater than the preset overlap threshold, the text content in the first target annotation frame and the second target annotation frame can be recorded in the first metadata.

[0098] Step S302: determining second metadata in the text recognition result, where the second metadata is text content that exists in all image samples in the text recognition result and whose occurrence times meet a preset number requirement.

[0099] It should be understood that the above-mentioned second metadata can record the text content that appears only once in each image sample in the text recognition result in the second metadata, that is, in this embodiment, the above-mentioned preset number requirement can be set to once, so that the content that appears fixedly in the image can be saved as the second metadata.

[0100] Step S303: Acquire third metadata based on the difference between the text recognition result and the manual annotation result.

[0101] It is understandable that the third metadata may be the content where there is a difference between the text recognition result and the manual annotation result.

[0102] Furthermore, the step S303 includes:

[0103] Step S303a: determining a difference recording method according to the difference content between the text recognition result and the manual annotation result.

[0104] It should be understood that the above-mentioned difference recording method can be a method used to record the difference between the text recognition result and the manual annotation result. In practical applications, the difference recording method is related to the needs of the actual application scenario. Different application scenarios encounter different difference problems, so different recording methods are required. The precision of the difference recording is related to the precision of the prediction demand.

[0105] Specifically, the step S303a includes: determining a detection frame in the image data set to be extracted based on the text recognition result; determining a difference type corresponding to the difference content between the text recognition result and the manual annotation result based on the detection frame and the annotation frame corresponding to the manual annotation result; and determining a difference recording method according to the difference type.

[0106] It should be noted that, referring to Fig.10 , Fig.10This is a diagram showing the types of differences in the key information extraction method of this application. Fig.10 There are several specific difference types and recording methods shown in the figure. The blue box is the detection box obtained by the detection model, that is, the detection box in the above-mentioned image dataset to be extracted; the orange box is the annotation box obtained by manual annotation, that is, the annotation box corresponding to the above-mentioned manual annotation result.

[0107] It should be noted that the difference types corresponding to the difference contents between the text recognition result and the manual annotation result in this embodiment may include: Fig.10 There are four types of differences shown, among which difference a means that the content annotated by the user spans across lines, while the result of text detection is that each line is an independent detection box; difference b means that the content annotated by the user is two independent parts, while text detection results in only one detection box, which includes the two parts of the content annotated by the user; difference c means that the content annotated by the user is horizontally continuous characters, but because the character spacing is large, text detection results in multiple independent character boxes; difference d means that because the spacing between different contents is too small, the detection box detected by the text contains content that does not need to be extracted.

[0108] In practical applications, Fig.10 The difference type shown can be judged based on the overlap between the annotation box and the detection box and the specific content. For example, in difference a and difference c, the overlap can be defined as the overlap area of ​​the two boxes divided by the area of ​​the detection box. If the overlap is greater than 80%, it means that the content of the detection box belongs to the content of the annotation box. If multiple different detection boxes belong to the annotation box, then it meets the requirements. Fig.10 Further analysis of whether the multiple annotation boxes are in different rows or the same row can distinguish whether Fig.10 One of the situations shown in difference a and difference c.

[0109] Step S303b: Record the difference content in the difference recording manner to obtain a difference metadata record.

[0110] It should be noted that after determining the difference type corresponding to the difference content between the text recognition result and the manual annotation result, the difference data can be recorded in a corresponding difference recording method to obtain the above-mentioned difference metadata. For example, taking difference type a as an example, the metadata record can be expressed as: {"difference_type":"a","annotation_type":"question","user_annotation":"certification agency","user_annotation_position":[0.1,0.2,0.15,0.22],"model_detected":["certification","agency"],"user_annotation_position":[0.1,0.2,0.15,0.22],"model_detected_position":[[0.1,0.2,0.15,0.22],[0.1,0.2,0.15,0.22]],"multi_lines":"true","lines_spacing":0.8}. Among them, difference_type records the type of difference; annotation_type records the type of the annotation box, usually header, question, answer, and other, which respectively represent the title, question, answer, and other types; user_annotation records the text content entered by the user during annotation; user_annotation_position records the coordinates of the annotation box during user annotation; model_detected records the text content recognized by the recognition model, which is in the form of a list because there are multiple texts; model_detected_position records the coordinates of the text box detected by the detection model, which is in the form of a list because there are multiple detection boxes; multi_lines indicates whether the annotation box has multiple lines, which is a Boolean value. If there are multiple lines, it is true, otherwise it is false; lines_spacing indicates the line spacing, and the unit is the average height of the detection box. Similar metadata records can be constructed for other types of differences.

[0111] Step S303c: Deduplication processing is performed on the difference metadata records using a preset difference deduplication rule to obtain third metadata.

[0112] It should be noted that the difference between each image sample may be different, so the difference data can be deduplicated. In this embodiment, the difference metadata records with the same difference type, the same annotation box type, the same annotation content, and the position box overlap IOU (Intersection over Union) value greater than 85% can be deduplicated according to the preset difference deduplication rules, and a record with a larger difference value is retained during deduplication to obtain the above-mentioned third metadata.

[0113] Step S304: merging the first metadata, the second metadata and the third metadata to obtain pattern-aligned metadata.

[0114] It can be understood that, in this embodiment, the first metadata, the second metadata and the third metadata can be merged to obtain a pattern-aligned metadata.

[0115] In this embodiment, it is disclosed that first metadata is determined based on a text recognition result and a manual annotation result corresponding to a to-be-extracted image data set, the first metadata being text content in which the text content in the text recognition result and the manual annotation result is the same and the overlap of the annotation box is greater than a preset overlap threshold; second metadata in the text recognition result is determined, the second metadata being text content in which each image sample in the text recognition result exists and the number of occurrences meets a preset number requirement; third metadata is obtained based on the difference content between the text recognition result and the manual annotation result; the first metadata, the second metadata and the third metadata are merged to obtain pattern alignment metadata. Since this embodiment can perform pattern alignment on the manual annotation content and the model detection content to obtain the pattern alignment metadata, the pattern alignment metadata can be used for model prediction later, thereby ensuring that the result form of the model output is consistent with the human-understandable pattern while ensuring the model training and prediction effect, greatly reducing the customized post-processing process required due to the inconsistency between the two patterns, and improving the accuracy of key information extraction.

[0116] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the above embodiments can be referred to the above introduction, and will not be described in detail later. Fig.11 , Fig.11 A flowchart of the third embodiment of the key information extraction method of this application is provided.

[0117] In this embodiment, step S40 includes steps S401 to S405:

[0118] Step S401: splitting the difference contents in the text recognition result based on the pattern alignment metadata to obtain a split correction record.

[0119] In this embodiment, the metadata information in the pattern alignment metadata M can be used to align the text recognition result C with the metadata information in the pattern alignment metadata M. Fig.10 The content of the cases where difference b and difference d are similar is split to obtain the above split correction record.

[0120] Step S402: input the split and corrected record into a preset text information classification model to obtain an information classification prediction result.

[0121] It should be understood that, in this embodiment, the split correction record can be input into a preset text information classification model for model training, and the preset text information classification model can output a prediction result, namely the above-mentioned information classification prediction result.

[0122] Step S403: Correct the content of the question type text of the information classification prediction result by using the pattern alignment metadata to obtain a question correction record.

[0123] In this embodiment, the metadata information in the pattern alignment metadata M can be used to split and correct the text of the problem type in the record. Fig.10 The contents of the cases similar to difference a and difference c are corrected to obtain the above-mentioned problem correction record.

[0124] Step S404: input the problem correction record into a preset text information relationship matching model to obtain an information relationship matching prediction result.

[0125] It should be understood that in this embodiment, the problem correction record can be input into the preset text information relationship matching model for model training. At this time, the preset text information relationship matching model can output a prediction result, that is, the above-mentioned information relationship matching prediction result.

[0126] Step S405: performing content correction on the answer type text of the information relationship matching prediction result through the pattern alignment metadata to obtain the key information extraction result corresponding to the image data set to be extracted.

[0127] In this embodiment, the metadata information in the pattern alignment metadata M can be used to correct the content that is different in the text of the answer type in the information relationship matching prediction result, and obtain the answer correction record, so as to realize customized post-processing, and finally output the key information extraction processing corresponding to the image data set to be extracted. This embodiment adds three links of content splitting, question correction and answer correction to the conventional prediction process of the model, and applies the pattern alignment metadata to these three links respectively, so that the final output result can be matched to the output form that can be understood by humans, thereby improving the effect of model training, while maintaining consistency with the output form that can be understood by humans, and reducing the R&D investment in the customized post-process.

[0128] In the specific implementation, refer to Fig.12 , Fig.12 This is the overall prediction flow chart of the key information extraction method of this application. Fig.12 As shown, first, the device can use the text detection model to detect text on the image P to obtain the text detection result B'. The text detection result B' contains several detection point records, where each detection point record is in a dictionary format and records the image id and the coordinate position information of the detection box. Then, the text recognition model can be used to perform text recognition on the image P based on the text detection result B' to obtain the text recognition result C'. The text recognition result C' contains several text recognition result records, where each text recognition result record is in a dictionary format and records the image id, the coordinate position of the detection box, and the recognized text content information. Afterwards, the metadata information in the pattern alignment metadata M obtained during training can be used to align the text recognition result C' with the text recognition result C'. Fig.10 The content of the case where the difference b and the difference d are similar is split, and the split correction record D' is input into the SER model for model prediction to obtain the SER prediction result F'. After that, the metadata information in the pattern alignment metadata M can be used to predict the text of the problem type in the SER prediction result F'. Fig.10 The content of the similar cases of difference a and difference c in the above example is corrected to obtain the problem correction record E', and then the problem correction record E' is input into the RE model for model prediction to obtain the RE model prediction result G'. Finally, the metadata information in the pattern alignment metadata M can be used to correct the different content in the text of the answer type in the RE model prediction result G' to obtain the answer correction record H'. If there are still some problems that are difficult to correct after the correction process of the above steps, customized post-processing can be implemented according to the specific problem, and finally a customized post-processing output O' is obtained, and finally the key information extraction result of the image is output.

[0129] In the present embodiment, it is disclosed that the difference content in the text recognition result is split based on pattern alignment metadata to obtain a split correction record; the split correction record is input into a preset text information classification model to obtain an information classification prediction result; the question type text of the information classification prediction result is content-corrected by the pattern alignment metadata to obtain a question correction record; the question correction record is input into a preset text information relationship matching model to obtain an information relationship matching prediction result; the answer type text of the information relationship matching prediction result is content-corrected by the pattern alignment metadata to obtain a key information extraction result corresponding to the image data set to be extracted. Since the present embodiment applies the pattern alignment metadata to the three links of content splitting, question correction and answer correction in the model training process, part of the recognition content can be corrected, thereby improving the effectiveness of the model training. At the same time, the use of automated correction and matching in the present embodiment can reduce manpower investment.

[0130] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the key information extraction method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0131] This application also provides a key information extraction device, please refer to Fig.13 , the key information extraction device comprises:

[0132] A text detection module 10, used to perform text detection on the image data set to be extracted and obtain a text detection result;

[0133] A text recognition module 20, configured to perform text recognition on the image data set to be extracted based on the text detection result to obtain a text recognition result;

[0134] A pattern alignment module 30 is used to perform pattern alignment on the text recognition result and the manual annotation result corresponding to the image data set to be extracted to obtain pattern alignment metadata;

[0135] The information extraction module 40 is used to obtain the key information extraction result corresponding to the image data set to be extracted based on the pattern alignment metadata, the preset text information classification model, and the preset text information relationship matching model.

[0136] The key information extraction device provided by the present application adopts the key information extraction method in the above embodiment, which can solve the technical problem in the prior art that when performing OCR key information extraction, the accuracy of key information extraction is not high due to the low accuracy of information annotation through manual annotation and / or detection model annotation. Compared with the prior art, the beneficial effects of the key information extraction device provided by the present application are the same as the beneficial effects of the key information extraction method provided by the above embodiment, and the other technical features in the key information extraction device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0137] The present application provides a key information extraction device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the key information extraction method in the above-mentioned embodiment one.

[0138] Reference below Fig.14 , which shows a schematic diagram of the structure of a key information extraction device suitable for implementing the embodiment of the present application. The key information extraction device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.14 The key information extraction device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0139] like Fig.14As shown, the key information extraction device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the key information extraction device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the key information extraction device to communicate wirelessly or wired with other devices to exchange data. Although the key information extraction device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0140] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0141] The key information extraction device provided by the present application adopts the key information extraction method in the above embodiment, which can solve the technical problem of key information extraction. Compared with the prior art, the beneficial effects of the key information extraction device provided by the present application are the same as the beneficial effects of the key information extraction method provided by the above embodiment, and other technical features in the key information extraction device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0142] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0143] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0144] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the key information extraction method in the above-mentioned embodiment.

[0145] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0146] The computer-readable storage medium may be included in the key information extraction device; or may exist independently without being assembled into the key information extraction device.

[0147] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the key information extraction device, the key information extraction device enables the following: to perform text detection on the image data set to be extracted to obtain a text detection result; to perform text recognition on the image data set to be extracted based on the text detection result to obtain a text recognition result; to perform pattern alignment on the text recognition result and the manual annotation result corresponding to the image data set to be extracted to obtain pattern alignment metadata; and to obtain the key information extraction result corresponding to the image data set to be extracted based on the pattern alignment metadata, a preset text information classification model, and a preset text information relationship matching model.

[0148] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0149] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0150] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0151] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned key information extraction method, and can solve the technical problem in the prior art that when performing OCR key information extraction, the accuracy of key information extraction is not high due to the low accuracy of information annotation through manual annotation and / or detection model annotation. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the key information extraction method provided by the above-mentioned embodiment, and will not be repeated here.

[0152] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A key information extraction method, characterized in that: The method includes: Perform text detection on the image data set to be extracted to obtain text detection results; Based on the text detection result, text recognition is performed on the image data set to be extracted to obtain a text recognition result; Performing pattern alignment on the text recognition result and the manual annotation result corresponding to the image data set to be extracted to obtain pattern alignment metadata; Based on the pattern alignment metadata, the preset text information classification model, and the preset text information relationship matching model, the key information extraction result corresponding to the image data set to be extracted is obtained.

2. The method according to claim 1, characterized in that The step of performing pattern alignment on the text recognition result and the manual annotation result corresponding to the image data set to be extracted to obtain pattern alignment metadata includes: Determine first metadata based on the text recognition result and the manual annotation result corresponding to the to-be-extracted image data set, wherein the first metadata is text content in which the text recognition result and the manual annotation result have the same text content and the degree of overlap of the annotation boxes is greater than a preset overlap threshold; Determine second metadata in the text recognition result, where the second metadata is text content that exists in all image samples in the text recognition result and that appears a number of times that meets a preset number requirement; Acquire third metadata based on the difference between the text recognition result and the manual annotation result; The first metadata, the second metadata, and the third metadata are merged to obtain pattern-aligned metadata.

3. The method according to claim 2, characterized in that The step of determining the first metadata based on the text recognition result and the manual annotation result corresponding to the image data set to be extracted includes: Obtaining a first annotation box in the text recognition result and a second annotation box in the manual annotation result corresponding to the image data set to be extracted; Determine a first target annotation frame and a second target annotation frame that have the same text content in the first annotation frame and the second annotation frame; Determine an area of ​​region overlap and an area of ​​region union between the first target annotation box and the second target annotation box; Determine the overlap between the first target annotation box and the second target annotation box based on the region overlap area and the region union area; If the overlap degree of the annotation boxes is greater than a preset overlap degree threshold, the text contents in the first target annotation box and the second target annotation box are recorded as first metadata.

4. The method according to claim 2, characterized in that The step of acquiring third metadata based on the difference between the text recognition result and the manual annotation result includes: Determining a difference recording method according to the difference content between the text recognition result and the manual annotation result; Recording the difference content in the difference recording manner to obtain a difference metadata record; The difference metadata records are deduplicated using a preset difference deduplication rule to obtain third metadata.

5. The method according to claim 4, characterized in that The step of determining the difference recording method according to the difference between the text recognition result and the manual annotation result includes: Determine a detection frame in the image data set to be extracted based on the text recognition result; Determining a difference type corresponding to a difference content between the text recognition result and the manual annotation result based on the detection frame and the annotation frame corresponding to the manual annotation result; A difference recording method is determined according to the difference type.

6. The method according to any one of claims 1 to 5, characterized in that The step of obtaining the key information extraction result corresponding to the image data set to be extracted based on the pattern alignment metadata, the preset text information classification model, and the preset text information relationship matching model includes: Based on the pattern alignment metadata, content splitting is performed on the difference content in the text recognition result to obtain a splitting correction record; Inputting the split and revised records into a preset text information classification model to obtain information classification prediction results; Correcting the content of the question type text of the information classification prediction result by using the pattern alignment metadata to obtain a question correction record; Inputting the problem correction record into a preset text information relationship matching model to obtain an information relationship matching prediction result; The answer type text of the information relationship matching prediction result is content-corrected through the pattern alignment metadata to obtain the key information extraction result corresponding to the image data set to be extracted.

7. The method according to any one of claims 1 to 5, characterized in that Before the step of performing text detection on the image dataset to be extracted and obtaining the text detection result, the step further includes: Determine the coordinate position of the annotation box in the image data set to be extracted, the annotation box category and the annotation box link information; Generate a plurality of annotation point records based on the image identifier in the to-be-extracted image data set, the annotation box coordinate position, the annotation box category and the annotation box link information; The image data set to be extracted is annotated based on each annotation point record to obtain a manual annotation result corresponding to the image data set to be extracted.

8. A key information extraction device, characterized in that: The device comprises: A text detection module is used to perform text detection on the image data set to be extracted and obtain text detection results; A text recognition module, used to perform text recognition on the image data set to be extracted based on the text detection result to obtain a text recognition result; A pattern alignment module, used to perform pattern alignment on the text recognition result and the manual annotation result corresponding to the image data set to be extracted, so as to obtain pattern alignment metadata; The information extraction module is used to obtain the key information extraction result corresponding to the image data set to be extracted based on the pattern alignment metadata, the preset text information classification model, and the preset text information relationship matching model.

9. A key information extraction device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the key information extraction method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the key information extraction method according to any one of claims 1 to 7 are implemented.

Images