A method, device, storage medium and equipment for extracting structured document information

Abstract

The present application discloses a method, device, storage medium and equipment for structured extraction of document information, the method comprising: first obtaining a target image where the target document is located; and extracting target visual features of the target image; then inputting the target visual features into a semantic information extraction model for OCR information extraction to obtain target OCR information, and encoding it to obtain a target semantic encoding vector; then inputting the target visual features and the target semantic encoding vector into a pre-built multimodal large language model to predict the structured information extraction result corresponding to the target document. It can be seen that since the present application adopts a general structured information extraction method combining OCR information with a multimodal large language model, and uses visual features and OCR information as the extraction basis for spatial and textual semantics respectively, it solves the problem of lack of spatial information when performing structured extraction through pure text, thereby effectively improving the accuracy of structured extraction of document information.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a method, device, storage medium and equipment for extracting structured document information. Background Art

[0002] With the advent of the information age, the amount of information that needs to be processed has increased exponentially. Documents can be found everywhere in various fields of people's daily lives, such as contracts, bills, teaching aids, and test papers. How to extract valuable key information from various unstructured and structured documents more quickly and accurately has become an important research topic.

[0003] At present, there are usually two ways to achieve structured extraction of document information: one is to use predefined rules and patterns to extract key information from documents, but this method relies on the accuracy and comprehensiveness of the rules. It works well in documents with specific formats and structures, but when faced with diverse and complex text formats, the design and maintenance of rules becomes difficult and inflexible, which may lead to inaccurate extraction results or omission of key information. The second is to use machine learning algorithms to automatically extract useful information from unstructured texts. Although this method has great advantages in processing complex texts and various scenarios, it also has problems such as data scarcity, model overfitting, and difficulty in processing long texts. It can be seen that the existing methods for achieving structured extraction of document information cannot achieve ideal results. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to provide a method, device, storage medium and equipment for structured extraction of document information, which can improve the accuracy of structured extraction of document information.

[0005] The present application provides a method for extracting structured document information, including:

[0006] Acquire a target image where the target document to be extracted is located; and extract target visual features of the target image;

[0007] Inputting the target visual features into a semantic information extraction model to extract optical character recognition (OCR) information to obtain target OCR information corresponding to the target document; and encoding the target OCR information to obtain a target semantic encoding vector;

[0008] The target visual features and the target semantic encoding vector are input into a pre-built multimodal large language model to predict the information structured extraction result corresponding to the target document.

[0009] In a possible implementation manner, after extracting the target visual features of the target image, the method further includes:

[0010] The target visual feature is resampled to obtain the resampled target visual feature.

[0011] In a possible implementation, the step of inputting the target visual feature into a semantic information extraction model to extract optical character recognition (OCR) information to obtain target OCR information corresponding to the target document includes:

[0012] The resampled target visual features are input into a semantic information extraction model to extract OCR information, thereby obtaining target OCR information corresponding to the target document.

[0013] In a possible implementation, the step of inputting the target visual feature and the target semantic encoding vector into a pre-built multimodal large language model to predict the structured information extraction result corresponding to the target document includes:

[0014] The resampled target visual features and target semantic encoding vectors are input into a pre-built multimodal large language model to predict and obtain a structured information extraction result corresponding to the target document.

[0015] In a possible implementation, the semantic information extraction model is constructed as follows:

[0016] Acquire a first sample image where a first target document is located;

[0017] The initial semantic information extraction model is trained using the first sample image, the regression loss function and the first cross entropy loss function to obtain the semantic information extraction model.

[0018] In a possible implementation, the initial semantic information extraction model is an OCR information detection head SpottingHead.

[0019] In a possible implementation, the multimodal large language model is constructed as follows:

[0020] Acquire a second sample image where a second target document is located;

[0021] Pre-training the initial multimodal large language model using the second sample image and the second cross entropy loss function to obtain a pre-trained multimodal large language model;

[0022] The pre-trained multimodal large language model is fine-tuned using a structured question answering (QA) task to obtain the multimodal large language model.

[0023] In a possible implementation, the target OCR information corresponding to the target document includes the contents of all serialized texts in the target document and their corresponding bounding box information.

[0024] The present application also provides a document information structure extraction device, including:

[0025] A first acquisition unit is used to acquire a target image where a target document to be extracted is located; and extract a target visual feature of the target image;

[0026] The first input unit is used to input the target visual feature into a semantic information extraction model to extract optical character recognition (OCR) information to obtain target OCR information corresponding to the target document; and encode the target OCR information to obtain a target semantic encoding vector;

[0027] The second input unit is used to input the target visual features and the target semantic encoding vector into a pre-built multimodal large language model to predict the information structured extraction result corresponding to the target document.

[0028] In a possible implementation manner, the device further includes:

[0029] The resampling unit is used to resample the target visual features to obtain the resampled target visual features.

[0030] In a possible implementation, the first input unit is specifically used to:

[0031] The resampled target visual features are input into a semantic information extraction model to extract OCR information, thereby obtaining target OCR information corresponding to the target document.

[0032] In a possible implementation, the second input unit is specifically used to:

[0033] The resampled target visual features and target semantic encoding vectors are input into a pre-built multimodal large language model to predict and obtain a structured information extraction result corresponding to the target document.

[0034] In a possible implementation manner, the device further includes:

[0035] A second acquisition unit, used for acquiring a first sample image where the first target document is located;

[0036] A training unit is used to train an initial semantic information extraction model using the first sample image, a regression loss function and a first cross entropy loss function to obtain the semantic information extraction model.

[0037] In a possible implementation, the initial semantic information extraction model is an OCR information detection head SpottingHead.

[0038] In a possible implementation manner, the device further includes:

[0039] A third acquisition unit, used for acquiring a second sample image where a second target document is located;

[0040] A pre-training unit, configured to pre-train an initial multimodal large language model using the second sample image and the second cross entropy loss function to obtain a pre-trained multimodal large language model;

[0041] The fine-tuning unit is used to fine-tune the pre-trained multimodal large language model by using a structured question answering (QA) task to obtain the multimodal large language model.

[0042] In a possible implementation, the target OCR information corresponding to the target document includes the contents of all serialized texts in the target document and their corresponding bounding box information.

[0043] The embodiment of the present application also provides a document information structured extraction device, including: a processor, a memory, and a system bus;

[0044] The processor and the memory are connected via the system bus;

[0045] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor executes any one of the implementations of the above-mentioned document information structured extraction method.

[0046] An embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes any one of the implementation methods of the above-mentioned document information structured extraction method.

[0047] The embodiment of the present application also provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes any one of the implementations of the above-mentioned document information structured extraction method.

[0048] The embodiment of the present application provides a method, device, storage medium and equipment for document information structure extraction, firstly obtaining the target image where the target document to be extracted is located; and extracting the target visual features of the target image; then inputting the target visual features into the semantic information extraction model for optical character recognition (OCR) information extraction to obtain the target OCR information corresponding to the target document; and encoding the target OCR information to obtain the target semantic encoding vector; then, inputting the target visual features and the target semantic encoding vector into the pre-built multimodal large language model to predict the information structure extraction result corresponding to the target document. It can be seen that since the present application adopts a general information structure extraction method combining OCR information and multimodal large language model, in which visual features and OCR information are used as the extraction basis of space and text semantics respectively, the problem of lack of spatial information when the existing structured extraction is performed only through pure text is solved, so that the accuracy of document information structure extraction can be effectively improved, and it has strong adaptability and can process diversified and complex text format data. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 A flowchart of a method for extracting structured document information provided by an embodiment of the present application;

[0051] Figure 2 An example diagram of the process of extracting structured information from a target document provided in an embodiment of the present application;

[0052] Figure 3 A schematic diagram of the composition of a document information structured extraction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] At present, the research on structured extraction of document information mainly focuses on how to extract key information from text recognized by OCR. Traditional rule-based extraction methods mainly rely on rule engines and regular expressions to extract information in specific fields. Although rule-based methods work better in documents with specific formats and structures, the design and maintenance of rules become difficult and inflexible when faced with diverse and complex text formats. The other is an extraction method based on machine learning, which classifies document information through a series of feature engineering. The specific implementation process of these two extraction methods is as follows:

[0054] (1) Rule-based extraction method.

[0055] The rule-based structured information extraction (KIE) method refers to the use of predefined rules and patterns to extract key information from text. These rules are usually based on the structure of the text, keywords, contextual relationships, etc., and are applicable to specific fields or document types. Specifically, rule definition, rules are usually defined by regular expressions, keyword lists, contextual constraints, etc. For example, specific regular expression rules can be designed for invoice numbers, dates, amounts, etc. Pattern matching, the system scans the document according to the defined rules and identifies matching text fragments. These rules can be based on the format of the text (such as the "YYYY-MM-DD" format of the date) or contextual relationships (such as the amount is usually followed by "total" or "expense"). Information extraction, once a matching text fragment is found, the system will extract and classify it into predetermined fields or tags. For example, the invoice number, date, and amount extracted from the invoice will be sorted into the corresponding fields.

[0056] It can be seen that the rule-based approach relies on the accuracy and comprehensiveness of the rules. The rules must be precisely designed for the specific format and content of the target document, otherwise the extraction results may be inaccurate or key information may be missed. Its advantages are simple implementation and strong interpretability, but its limitations are that it is difficult to adapt to changes in document formats and process complex unstructured texts.

[0057] (2) Extraction method based on machine learning.

[0058] The machine learning-based structured information extraction (KIE) method refers to the use of machine learning algorithms to automatically extract useful information from unstructured text. These methods are highly flexible and adaptable and can handle a variety of complex documents and text structures. Before applying machine learning methods, text data needs to be preprocessed, including noise removal, word segmentation, and tagging. These steps help convert the original text into a format that the model can process. The second step of the method is to extract features from the text. Machine learning models need to extract features from text. Common features include word vectors (such as Word2Vec), contextual information, syntactic structure, etc. Existing methods usually use pre-trained language models (such as BERT) to generate context-related word embeddings. Model training mainly includes three parts: named entity recognition (NER), relation extraction (RE), and information extraction: named entity recognition (NER), which is used to identify specific entities in text (such as names, places, dates). Common models include conditional random fields (CRF), long short-term memory networks (LSTM), and BERT-based models. Relation extraction (RE) is used to identify the relationship between entities. The model usually includes convolutional neural network (CNN), recurrent neural network (RNN) and attention mechanism. Although the information extraction method based on machine learning has great advantages in processing complex texts and various scenarios, it also has problems such as data scarcity, model overfitting, and difficulty in processing long texts.

[0059] Therefore, whether it is a rule-based key information structured extraction method or a machine learning-based information structured extraction method, its ability to solve general document scenarios (which can also be understood as cross-domain scenarios) and complex layouts, that is, information structuring of text content is very lacking and cannot achieve the desired effect. For example, in the text "Name: Zhang is absent" that appears in the document, the current extraction method will structure it into "{key: name, value: Zhang is absent}". For another example, assuming that "Summary [line break] xxxx" appears in the document, the current extraction method cannot parse out the structured information contained in this text.

[0060] Therefore, how to improve the accuracy of structured extraction of document information and thus improve user experience is a technical problem that needs to be solved urgently.

[0061] To address the above-mentioned defects, the present application provides a method for structured extraction of document information, which first obtains a target image containing a target document to be extracted; and extracts target visual features of the target image; then inputs the target visual features into a semantic information extraction model to extract OCR information, and obtains target OCR information corresponding to the target document; and encodes the target OCR information to obtain a target semantic encoding vector; then, the target visual features and the target semantic encoding vector are input into a pre-constructed multimodal large language model to predict the structured information extraction result corresponding to the target document.

[0062] It can be seen that since the present application adopts a general information structured extraction method that combines OCR information with a multimodal large language model, in which visual features and OCR information are used as the basis for spatial and textual semantic extraction respectively, the problem of lack of spatial information in the existing structured extraction through pure text is solved, thereby effectively improving the accuracy of structured extraction of document information, and has strong adaptability, and can handle diverse and complex text format data, thereby improving user experience.

[0063] For example, taking the text "Name: Zhang is absent" in the document as an example, the current extraction method will structure it into "{key: name, value: Zhang is absent}". The document information structured extraction method provided in this application combines the accuracy of OCR information with the learning ability of a multimodal large model to detect the semantic and spatial structure information of the document at a fine-grained level. For the above example, this application will take into account that when the key is "name", the value generally does not contain "absence"; for example, the spatial structure of the key and value may be in an "up-down" relationship.

[0064] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0065] First embodiment

[0066] See also Figure 1 , is a flow chart of a method for extracting structured document information provided in this embodiment, the method comprising the following steps:

[0067] S101: Acquire a target image containing a target document to be extracted; and extract target visual features of the target image.

[0068] In this embodiment, any document that uses this embodiment to implement structured information extraction is defined as a target document to be extracted, and the image where the target document is located is defined as a target image. In addition, this embodiment does not limit the language type of the target document, for example, the target document can be a Chinese document, or an English document, etc.; this embodiment does not limit the source of the target document, for example, the target document can be teaching aids, homework and test papers from the education industry, or from other industries such as medicine and food; this embodiment does not limit the type of the target document, for example, the target document can be a company's charter contract, or a report form of a patient's visit to a hospital, etc.

[0069] In addition, this embodiment does not limit the type of the target image. For example, the target image can be a color image composed of three primary colors of red (R), green (G), and blue (B), or a grayscale image, etc. Moreover, this embodiment does not limit the resolution of the target image. For example, the target image can be a 720*480 RGB image, or a high-resolution 1920*1280 RGB image, etc. This embodiment does not limit the method for obtaining the target image. For example, the target image can be obtained by photographing the document with a mobile phone camera, etc.

[0070] Furthermore, after obtaining the target image containing the target document to be extracted, in order to improve the extraction accuracy, existing or future image visual feature extraction methods can be used, such as the ResNet series backbone based on convolutional neural networks, or a series of neural network frameworks based on transformers, etc., to extract the visual features of the target image (that is, extract the spatial vector representation of the target image, and the specific format is not limited and can be set according to actual conditions and experience values), and define it as the target visual feature to execute subsequent steps S102 and S103, so as to increase the spatial information in the subsequent structured information extraction as part of the extraction basis, thereby improving the accuracy of the extraction results.

[0071] For example: Assume the target image is Figure 2 The image of the "Color Image Report" shown on the left is input as follows Figure 2 After the visual feature extraction module (Vision Encoder) shown in FIG. 1 (such as a series of neural network frameworks based on transformers, etc.), the target visual features of the target image can be extracted as “image tokens”, such as Figure 2 shown.

[0072] S102: Input the target visual features into the semantic information extraction model to extract optical character recognition (OCR) information to obtain target OCR information corresponding to the target document; and encode the target OCR information to obtain a target semantic encoding vector.

[0073] In this embodiment, after obtaining the target image of the target document to be extracted and extracting the target visual features of the target image through step S101, in order to improve the accuracy of information structure extraction for the target document, the target visual features can be further input into a pre-built semantic information extraction model to extract OCR information, and the OCR information corresponding to the target document is obtained, and it is defined as the target OCR information. Among them, this application does not limit the specific content of the target OCR information, which may include but is not limited to the content of all texts serialized in the target document (such as in a reading order from top to bottom and from left to right) and their corresponding coordinate box bounding box information (such as the coordinates of the upper left corner and the coordinates of the lower right corner of the coordinate box, etc.). Furthermore, the contents of the target OCR information can be spliced ​​together, and then the spliced ​​target OCR information can be encoded using existing or future text encoding methods to obtain a semantic encoding vector, which is defined as a target semantic encoding vector. The present application does not limit the specific format of the target semantic encoding vector, and it can be set according to actual conditions and experience values ​​to execute subsequent step S103, so that it can be used as semantic information for subsequent structured information extraction, as another part of the extraction basis, to improve the accuracy of the extraction results.

[0074] It should be noted that the reason why this application uses the above-mentioned target OCR information as part of the extraction basis is because such tasks have certain versatility and universality, which is of great help to this application in solving document information structured extraction tasks in unlimited scenarios.

[0075] In addition, an optional implementation method, in order to extract information into a serialized text form, the present application adopts a technical solution similar to a large language model (LLM). First, an existing or future resampling method is used to resample the target visual features to obtain a fixed-length target visual feature after resampling. For example, the target visual feature can be downsampled to obtain a target visual feature with a length of 512, thereby reducing the amount of data processing and improving processing efficiency. Then, the resampled target visual feature is input into the semantic information extraction model for OCR information extraction to obtain the serialized target OCR information corresponding to the target document, and then the serialized target OCR information is spliced ​​and encoded to obtain a serialized target semantic encoding vector.

[0076] For example: Assume that the target image is still Figure 2 The image of the "Color Image Report" shown on the left is located and entered as Figure 2After the visual feature extraction module (Vision Encoder) shown in FIG. 1 , the target visual features of the target image are extracted as “image tokens” (such as Figure 2 On this basis, the target visual features can be resampled first (such as Figure 2 The Resample box shown in the figure is processed to obtain the fixed-length target visual features after resampling, such as Figure 2 The four rectangular boxes indicated by the arrows on the right side of the Resample box shown in the figure correspond to the "image tokens", thereby achieving dimensionality reduction of the "image tokens" and reducing the amount of data processing. Then the resampled target visual features (such as Figure 2 The "image tokens" corresponding to the four rectangular boxes indicated by the arrows on the right side of the Resample box shown in the figure are input as follows Figure 2 The semantic information extraction module (Spotting Head module) shown in FIG. 1 can obtain the target OCR information as follows: Figure 2 The "Color Ultrasound Report" shown in the upper right corner <x 11 ,y 11 ,x 12 ,y 12 >Name <x 21 ,y 21 ,x 22 ,y 22 > Zhang <x 31 ,y 31 ,x 32 ,y 32 >..." Then, combine these target OCR information and input them into Figure 2 The text encoding module (Text Encoder) shown in FIG. 1 can obtain the serialized target semantic encoding vector, such as Figure 2 The four rectangular boxes indicated by arrows below the Text Encoder box shown correspond to the "text tokens".

[0077] Among them, x 11 ,y 11 and x 12 ,y 12 They represent the coordinates of the upper left corner and lower right corner of the bounding box where the "Color Ultrasound Report" is located. 21 ,y 21 and x 22 ,y 22 They represent the coordinates of the upper left corner and lower right corner of the bounding box where the "name" is located.31 ,y 31 and x 32 ,y 32 They respectively represent the coordinates of the upper left corner and the lower right corner of the bounding box where "Zhang" is located.

[0078] It should be noted that in order to improve the accuracy of structured extraction of target document information, this application will pre-use a large number of sample images containing sample documents (here defined as the first sample image containing the first target document) to construct a semantic information extraction model with better OCR information extraction effect. In addition, this application does not limit the specific network composition structure of the semantic information extraction model, which can be selected and set according to actual conditions. A preferred implementation method is as follows: Figure 2 As shown, the pre-built semantic information extraction model may be, but is not limited to, an OCR information detection head (Spotting Head) composed of a transformer structure, which refers to a module in an OCR system for identifying and extracting specific text information in an image.

[0079] Next, this embodiment will introduce the construction process of the semantic information extraction model, wherein an optional implementation method is that the construction process of the semantic information extraction model may specifically include: first obtaining a first sample image where the first target document is located, and then using the first sample image, a regression loss function and a first cross-entropy (CE) loss function to train the initial semantic information extraction model to obtain a semantic information extraction model.

[0080] Specifically, in this implementation, in order to construct a semantic information extraction model, a lot of preparatory work needs to be done in advance. First, a large number of documents need to be collected as first sample documents, and the images of the first sample documents are obtained as first sample images to form model training data. For example, a large number of ultrasound reports or students' test paper recording data can be collected in advance as first sample documents, and then these first sample documents are photographed by a mobile phone camera to obtain a large number of first sample images to form model training data, and the text content in all text lines in these sample images and their corresponding coordinate box bounding box information (such as the coordinates of the upper left corner and the coordinates of the lower right corner of the coordinate box, etc.) are manually marked. Label recognition results. Then, the initial semantic information extraction model can be trained based on these first sample images and their corresponding label recognition results, regression loss function and first cross entropy loss function, and then a semantic information extraction model can be generated.

[0081] An optional implementation method is that the initial semantic information extraction model may be (but not limited to) an OCR information detection head (Spotting Head).

[0082] Specifically, when training the model, a first sample image can be extracted from the training data in turn as the model input, and the corresponding label recognition result (i.e., the serialized text displayed in the reading order + the coordinates of the coordinate box bounding box and other OCR information) can be used as the output, and multiple rounds of model training can be performed. Each round of training includes two stages: (1) the text line detection stage, i.e., outputting all the text line bounding boxes and coordinate information in the first sample image; (2) the text recognition stage, i.e., cutting the input first sample image according to the detected text line bounding box and coordinate information, and using the cut text image for text recognition. The text line and text content recognition results obtained in each round of training are compared with the corresponding manual annotation results, and the model parameters are updated according to the difference between the two until the preset conditions are met, such as the values ​​of the regression loss function and the first cross entropy loss function are very small and basically unchanged, then the model parameter update is stopped, the training of the semantic information extraction model is completed, and a trained semantic information extraction model is generated.

[0083] Among them, in the text line detection stage, the regression loss function is used for training, and the specific calculation formula is as follows:

[0084] L D =|b label -b pred |

[0085] Among them, b label b represents the text content in all text lines in the first sample image manually annotated and the corresponding coordinate frame bounding box information; b pred Represents the text content in all text lines in the first sample image output by the model and its corresponding bounding box information.

[0086] In the text recognition stage, the first cross entropy loss function is used for training. The specific calculation formula is as follows:

[0087] L R = -Y log P

[0088] Among them, P represents the probability distribution of the model output; Y represents the label in one-hot form. And the recognition result of the model can be obtained using argmax(P).

[0089] In this way, after obtaining the text line detection results and the text recognition results based on the OCR information detection head (Spotting Head), these target OCR information can be spliced ​​together in the form of text description, such as Figure 2The target OCR information shown is combined into text form: "Color Ultrasound Report<x11,y11,x12,y12> Name:<x21,y21,x22,y22> Zhang<x31,y31,x32,y32> ", and then encode it to obtain the target semantic encoding vector, which is then input into the Multi-Modal Language Model (MLLM) through subsequent steps for structured information extraction.

[0090] S103: Input the target visual features and the target semantic encoding vector into a pre-built multimodal large language model to predict the information structured extraction result corresponding to the target document.

[0091] In this embodiment, the target image containing the target document to be extracted is obtained through step S101 and the target visual features of the target image are extracted, and the target OCR information is encoded through step S102 to obtain the target semantic coding vector. In order to improve the accuracy of structured information extraction for the target document, the target visual features and the target semantic coding vector can be further concatenated and input into a pre-built multimodal large language model MLLM, so as to predict the structured information extraction result corresponding to the target document.

[0092] In addition, an optional implementation method, in order to extract information into a serialized text form, the present application adopts an existing or future resampling (resampler) method to resample the target visual features, and obtain the resampled target visual features of a fixed length. For example, the target visual features are downsampled to obtain target visual features of a length of 512. The resampled target visual features and the target semantic encoding vectors can be concatenated and then input into the multimodal large language model MLLM for prediction, so that the information structured extraction results corresponding to the target document can be obtained more quickly and accurately.

[0093] For example: Assume that the target image is still Figure 2 The image of the "Color Image Report" shown on the left is located and entered as Figure 2 After the visual feature extraction module (Vision Encoder) shown in FIG. 1 , the target visual features of the target image are extracted as “image tokens” (such as Figure 2 On this basis, the target visual features are first resampled (such as Figure 2 The Resample box shown in the figure is processed to obtain the fixed-length target visual features after resampling, such as Figure 2The four rectangular boxes indicated by the arrows on the right side of the Resample box shown in the figure correspond to the "image tokens". Then the resampled target visual features (such as Figure 2 The "image tokens" corresponding to the four rectangular boxes indicated by the arrows on the right side of the Resample box shown in the figure are input as follows Figure 2 The semantic information extraction module (Spotting Head module) shown in FIG. 1 obtains the target OCR information as follows: Figure 2 The "Color Ultrasound Report" shown in the upper right corner <x 11 ,y 11 ,x 12 ,y 12 >Name <x 21 ,y 21 ,x 22 ,y 22 > Zhang <x 31 ,y 31 ,x 32 ,y 32 >..." Then combine these target OCR information and input them into Figure 2 The text encoding module (Text Encoder) shown in FIG. 1 obtains the serialized target semantic encoding vector, such as Figure 2 The four rectangular boxes indicated by arrows below the Text Encoder box shown correspond to the "text tokens".

[0094] At the same time, in getting Figure 2 After the six rectangular boxes indicated by the arrows on the right side of the Vision Encoder shown in FIG. 1 are obtained, the target visual features can be resampled to obtain the resampled fixed-length target visual features, such as Figure 2 The four rectangular boxes indicated by the arrows below the Resample box shown in the figure correspond to the "image tokens". Then the resampled target visual features (such as Figure 2 After the “image tokens” corresponding to the four rectangular boxes indicated by the arrows below the Resample box are concatenated with the serialized target semantic encoding vector output by the text encoding module (TextEncoder) on the right, they can be input as follows Figure 2 The multi-modal language model (Multi-ModalLanguage Model) shown in Figure 2 obtains the information structured extraction result corresponding to the target document, such as Figure 2 The multimodal large language model box shown in the figure is indicated by the arrow on the right side of the "{text: Color ultrasound report <x 11 ,y 11 ,x 12 ,y 12>}{key:name <x 21 ,y 21 ,x 22 ,y 22 >,value:Zhang <x 31 ,y 31 ,x 32 ,y 32 >}…”.

[0095] In this way, compared with the existing extraction method that only inputs the OCR information of the plain text into the layoutLanguage Model (such as the pre-trained document understanding model based on the Transformer architecture) to extract the document information in a structured manner, that is, only using text to describe the spatial position of the text in the image will cause a large degree of spatial drift, which is not conducive to the perception of space. Figure 2 The image tokens+text tokens shown in the figure are used as input, and the multimodal large language model MLLM is used to accurately perceive the input target document in terms of semantics and space. It can more accurately output the structured information at the chapter level, thereby improving the accuracy of structured extraction of target document information.

[0096] It should be noted that in order to improve the accuracy of structured extraction of target document information, the present application will pre-utilize a large number of sample images containing sample documents (defined here as the second sample image where the second target document is located) to construct a multimodal large language model with better structured extraction of document information. In addition, the present application does not limit the specific network composition structure of the multimodal large language model, which can be selected and set according to actual conditions.

[0097] Next, this embodiment will introduce the construction process of the multimodal large language model, wherein an optional implementation method is that the construction process of the multimodal large language model may specifically include: first obtaining the second sample image where the second target document is located, and then using the second sample image and the second cross entropy loss function to pre-train the initial multimodal large language model to obtain the pre-trained multimodal large language model. Then, using the structured information question answering (QA) task, the pre-trained multimodal large language model is subjected to supervised fine-tuning (Supervised Fine-Tuning, SFT) processing to obtain the multimodal large language model.

[0098] Specifically, in this implementation, in order to construct a multimodal large language model, a lot of preparatory work needs to be done in advance. First, a large number of documents need to be collected as second sample documents, and the image where the second sample document is located is obtained as the second sample image (which can be the same as or different from the first sample image where the first target document is located) to constitute the model training data. For example, a large number of ultrasound reports or students' test paper recording data can be collected in advance as the second sample documents, and then these second sample documents are photographed by a mobile phone camera to obtain a large number of second sample images to constitute the model training data, and the text content in all text lines in these sample images and their corresponding coordinate frame bounding box information (such as the coordinates of the upper left corner and the coordinates of the lower right corner of the coordinate frame, etc.), as well as structured information are manually marked as label recognition results. Then, the initial multimodal large language model can be trained based on these second sample images and their corresponding label recognition results and the second cross entropy (CE) loss function to generate a multimodal large language model.

[0099] Specifically, when training the model, a second sample image can be extracted from the training data as the model input, and the corresponding label recognition result can be used as the output. Multiple rounds of model pre-training are performed. Each round of pre-training is to enhance the multimodal large model's perception of semantics and space. The training task includes document element structured extraction (layout), that is, outputting document layout information. This task mainly outputs the layout information of the second sample image, still based on Figure 2 Taking the picture shown in the figure (where the target image can be regarded as the second sample image) as an example, the output is "{Title: Color Ultrasound Report<x11,y11,x12,y12>}{text:name:<x21,y21,x22,y22> Zhang<x31,y31,x32,y32> The model output results obtained from each round of pre-training are compared with the corresponding manual annotation results, and the model parameters are updated according to the difference between the two until the preset conditions are met. For example, the value of the second cross entropy (CE) loss function is very small and basically unchanged, then the updating of the model parameters is stopped, the pre-training of the multimodal large language model is completed, and a pre-trained multimodal large language model is generated.

[0100] Furthermore, in order to enhance the multimodal model's perception of the semantics and space of structured information, this application also uses structured information question answering (QA) tasks. That is, randomly input the text and coordinates of a key or value, and ask questions to the pre-trained multimodal model. The pre-trained multimodal model needs to output the text and coordinates of the value or key. This training task allows the pre-trained multimodal model to perceive the semantic logic and spatial logic of structured extraction in a more fine-grained manner. For example, based on Figure 2The picture shown (the target image can be regarded as the second sample image) can be asked to the pre-trained multimodal large model: "{name:<x21,y21,x22,y22>}”, the answer of the pre-trained multimodal large model can be “{Zhang<x31,y31,x32,y32> Or, ask the pre-trained multimodal model “{Zhang<x31,y31,x32,y32>}”, the answer of the pre-trained multimodal large model can be “{name:<x21,y21,x22,y22> In this way, by using this type of structured information question-answering (QA) task to fine-tune the pre-trained multimodal large language model, the pre-trained model can learn the semantic and spatial associations of structured information, further improving the extraction effect of the model.

[0101] It can be seen that the process of training the model in this application includes two stages: pre-training and SFT. In the pre-training stage, the main purpose is to enhance the model's perception of the semantics and space of the document, which requires a large amount of OCR-related data such as Figure 2 The visual feature extraction module (Vision Encoder), semantic information extraction model (Spotting Head), and multi-modal language model (Multi-ModelLanguage Model) shown in the figure are trained. The training task of the semantic information extraction model (SpottingHead) only includes one task, OCR information detection. The training task of the multi-modal language model (Multi-ModelLanguageModel) includes two tasks: OCR information and document element structured extraction (layout). In the SFT stage, you can use precise labeled data for structured information extraction (the specific acquisition method and content are not limited). In addition to OCR information and document element structured extraction (layout), the training task of the multi-modal language model (Multi-Model Language Model) also includes structured information extraction tasks and structured information QA tasks to further improve the extraction effect of the model.

[0102] In this way, by executing the above steps S101-S103, visual features and OCR information are used as the extraction basis for spatial and textual semantics respectively, which solves the problem of lack of spatial information when the existing large language model (LLM) only performs structured extraction through text (that is, directly inputting OCR text information to LLM and using LLM to extract structured information from the input document will lack the perception of the spatial structure of the document). For example, the document "Summary: The inspection is basically normal\nThis report only represents the results of this sample" is directly input to LLM, and the text layout of the document in the image is as follows:

[0103] Summary: The examination is basically normal

[0104] This report only represents the results of this sample

[0105] In this regard, the result of using LLM to extract structured information on the document is "{key: summary, value: the inspection is basically normal. This report only represents the results of this sample}". In fact, the text "This report only represents the results of this sample" has nothing to do with the previous text. However, through the document information structured extraction method provided in this application, after extracting structured information on the document, a more accurate extraction result can be obtained, specifically: "{key: summary, value: the inspection is basically normal}, {text: This report only represents the results of this sample}".

[0106] In summary, the present embodiment provides a method for structural extraction of document information, which first obtains the target image where the target document to be extracted is located; and extracts the target visual features of the target image; then inputs the target visual features into the semantic information extraction model to extract OCR information, and obtains the target OCR information corresponding to the target document; and encodes the target OCR information to obtain the target semantic encoding vector; then, the target visual features and the target semantic encoding vector are input into the pre-built multimodal large language model to predict the information structural extraction result corresponding to the target document. It can be seen that since the present application adopts a general information structural extraction method combining OCR information with a multimodal large language model, in which visual features and OCR information are used as the extraction basis for spatial and textual semantics respectively, the problem of lack of spatial information in the existing structured extraction through pure text is solved, thereby effectively improving the accuracy of document information structural extraction, and has strong adaptability, and can handle diverse and complex text format data.

[0107] Second embodiment

[0108] This embodiment will introduce a device for extracting structured document information. For related content, please refer to the above method embodiment.

[0109] See also Figure 3 , is a schematic diagram of the composition of a document information structure extraction device provided in this embodiment, the device 300 includes:

[0110] The first acquisition unit 301 is used to acquire a target image where the target document to be extracted is located; and extract target visual features of the target image;

[0111] The first input unit 302 is used to input the target visual feature into a semantic information extraction model to extract optical character recognition (OCR) information to obtain target OCR information corresponding to the target document; and encode the target OCR information to obtain a target semantic encoding vector;

[0112] The second input unit 303 is used to input the target visual features and the target semantic encoding vector into a pre-built multimodal large language model to predict and obtain the structured information extraction result corresponding to the target document.

[0113] In an implementation of this embodiment, the device further includes:

[0114] The resampling unit is used to resample the target visual features to obtain the resampled target visual features.

[0115] In an implementation of this embodiment, the first input unit 302 is specifically used to:

[0116] The resampled target visual features are input into a semantic information extraction model to extract OCR information, thereby obtaining target OCR information corresponding to the target document.

[0117] In an implementation of this embodiment, the second input unit 303 is specifically used to:

[0118] The resampled target visual features and target semantic encoding vectors are input into a pre-built multimodal large language model to predict and obtain a structured information extraction result corresponding to the target document.

[0119] In an implementation of this embodiment, the device further includes:

[0120] A second acquisition unit, used for acquiring a first sample image where the first target document is located;

[0121] A training unit is used to train an initial semantic information extraction model using the first sample image, a regression loss function and a first cross entropy loss function to obtain the semantic information extraction model.

[0122] In an implementation of this embodiment, the initial semantic information extraction model is an OCR information spotting head.

[0123] In an implementation of this embodiment, the device further includes:

[0124] A third acquisition unit, used for acquiring a second sample image where a second target document is located;

[0125] A pre-training unit, configured to pre-train an initial multimodal large language model using the second sample image and the second cross entropy loss function to obtain a pre-trained multimodal large language model;

[0126] The fine-tuning unit is used to fine-tune the pre-trained multimodal large language model by using a structured question answering (QA) task to obtain the multimodal large language model.

[0127] In an implementation of this embodiment, the target OCR information corresponding to the target document includes the contents of all serialized characters in the target document and their corresponding coordinate box bounding box information.

[0128] Furthermore, the embodiment of the present application also provides a document information structured extraction device, including: a processor, a memory, and a system bus;

[0129] The processor and the memory are connected via the system bus;

[0130] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor executes any one of the implementation methods of the above-mentioned document information structured extraction method.

[0131] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes any one of the implementation methods of the above-mentioned document information structured extraction method.

[0132] Furthermore, an embodiment of the present application also provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute any one of the implementation methods of the above-mentioned document information structured extraction method.

[0133] It can be known from the description of the above implementation mode that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0134] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0135] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0136] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for extracting structured document information, characterized in that: include: Obtain the target image where the target document to be extracted is located; and extracting target visual features of the target image; Inputting the target visual features into a semantic information extraction model to extract optical character recognition (OCR) information to obtain target OCR information corresponding to the target document; and encoding the target OCR information to obtain a target semantic encoding vector; The target visual features and the target semantic encoding vector are input into a pre-built multimodal large language model to predict the information structured extraction result corresponding to the target document.

2. The method according to claim 1, characterized in that After extracting the target visual features of the target image, the method further includes: The target visual feature is resampled to obtain the resampled target visual feature.

3. The method according to claim 2, characterized in that The step of inputting the target visual feature into a semantic information extraction model to extract optical character recognition (OCR) information to obtain target OCR information corresponding to the target document includes: The resampled target visual features are input into a semantic information extraction model to extract OCR information, thereby obtaining target OCR information corresponding to the target document.

4. The method according to claim 2, characterized in that: The step of inputting the target visual features and the target semantic encoding vector into a pre-built multimodal large language model to predict and obtain a structured information extraction result corresponding to the target document includes: The resampled target visual features and target semantic encoding vectors are input into a pre-built multimodal large language model to predict and obtain a structured information extraction result corresponding to the target document.

5. The method according to claim 1, characterized in that The semantic information extraction model is constructed as follows: Acquire a first sample image where a first target document is located; The initial semantic information extraction model is trained using the first sample image, the regression loss function and the first cross entropy loss function to obtain the semantic information extraction model.

6. The method according to claim 5, characterized in that The initial semantic information extraction model is an OCR information detection head Spotting Head.

7. The method according to claim 1, characterized in that The multimodal large language model is constructed as follows: Acquire a second sample image where a second target document is located; Pre-training the initial multimodal large language model using the second sample image and the second cross entropy loss function to obtain a pre-trained multimodal large language model; The pre-trained multimodal large language model is fine-tuned using a structured question answering (QA) task to obtain the multimodal large language model.

8. The method according to any one of claims 1 to 7, characterized in that: The target OCR information corresponding to the target document includes the contents of all serialized characters in the target document and the corresponding coordinate box bounding box information.

9. A document information structured extraction device, characterized in that: include: A first acquisition unit is used to acquire a target image where a target document to be extracted is located; and extracting target visual features of the target image; The first input unit is used to input the target visual feature into a semantic information extraction model to extract optical character recognition (OCR) information to obtain target OCR information corresponding to the target document; and encode the target OCR information to obtain a target semantic encoding vector; The second input unit is used to input the target visual features and the target semantic encoding vector into a pre-built multimodal large language model to predict the information structured extraction result corresponding to the target document.

10. A document information structured extraction device, characterized in that: include: Processor, memory, system bus; The processor and the memory are connected via the system bus; The memory is used to store one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the method according to any one of claims 1 to 8.

Images