Document snapshot information extraction method and device, equipment and storage medium

Abstract

The application discloses a document snapshot information extraction method and device, equipment and a storage medium, relates to the technical field of multi-modal learning, and comprises the following steps: acquiring a preset image-text pair data set, performing autoregressive training on a pre-trained model, and obtaining a successfully trained pre-trained model; extracting the network structure of the successfully trained pre-trained model as a new word table network structure, updating the original word table network structure of a preset multi-modal large model, and determining an updated preset multi-modal large model; acquiring a preset visual question and answer data set and a first information extraction instruction input by a user, fine-tuning the updated preset multi-modal large model, and obtaining a preset multi-modal large model that has been fine-tuned; receiving a target document snapshot and a second information extraction instruction input by the user, and obtaining an information extraction result output by the fine-tuned preset multi-modal large model. The application can improve the accuracy of information extraction in complex document snapshots.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal learning, in particular to a document snapshot information extraction method and device, equipment and a storage medium. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, multi-modal learning, as a technology that can integrate and process multiple types of data, has shown extensive application potential in many fields, especially in the intersection of computer vision and natural language processing. It not only enhances the system's understanding ability of complex information, but also promotes the development of new interaction methods.

[0003] In the process of document digital transformation, optical character recognition (OCR) is a key technology that can convert text in pictures into editable and searchable electronic text format. Although current OCR technology performs well in converting ordinary text, its accuracy and stability still need to be improved when dealing with official documents with complex backgrounds or formats. For example, when dealing with official document snapshots, since these documents often contain complex formats, charts, seals and other elements, OCR technology often has difficulty in accurately recognizing and correctly parsing, resulting in low accuracy of document snapshot information extraction.

[0004] Therefore, how to improve the accuracy of information extraction in complex document snapshots is a problem that needs to be solved at present.

[0005] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0006] The main purpose of the present application is to provide a document snapshot information extraction method, device, equipment and storage medium, aiming to solve the technical problem of how to improve the accuracy of information extraction in complex document snapshots.

[0007] To achieve the above purpose, the present application provides a document snapshot information extraction method, which comprises:

[0008] obtaining a preset image-text pair data set, taking the preset image-text pair data set as training data of a pre-training model, performing autoregressive training on the pre-training model, and obtaining a successfully trained pre-training model, wherein the preset image-text pair data set includes a document image-text pair data set, a chart image-text pair data set and a negative natural image-text pair data set, and the pre-training model is used to generate a new word table according to the preset image-text pair data set;

[0009] Extract the network structure of the successfully trained pre-trained model as a new vocabulary network structure, use the new vocabulary network structure to update the original vocabulary network structure of the preset multimodal large model, and determine the updated preset multimodal large model;

[0010] Obtain a preset visual question-answering dataset and a first information extraction instruction input by the user. Use the preset visual question-answering dataset as the training dataset for the updated preset multimodal large model. Use the first information extraction instruction as the large model instruction for the updated preset multimodal large model. Fine-tune the updated preset multimodal large model to obtain the preset multimodal large model after fine-tuning.

[0011] The system receives a target document snapshot and a second information extraction instruction from the user, inputs the target document snapshot and the second information extraction instruction into the fine-tuned preset multimodal large model, and obtains the information extraction result output by the fine-tuned preset multimodal large model.

[0012] In one embodiment, prior to the step of obtaining the preset image-text pair dataset, the following steps are included:

[0013] Collect raw document data containing plain text, data formulas, and tables from external data sources, and extract plain text data, data formula data, and table data from each of the raw document data.

[0014] Based on multiple predefined templates, the plain text data, the data formula data, and the table data are rendered to obtain a rendered first document file, wherein each predefined template corresponds to a different document type;

[0015] Each of the first document files is converted into a document image, and each of the first document files is converted into a second document file in text format. Each document image is paired with its corresponding second document file to generate a document image-text pair dataset.

[0016] Multiple original charts are obtained from the external data source. The actual values ​​of the chart text are extracted from each of the original charts. The actual values ​​of the chart text are converted into JSON data. The JSON data is used as a chart image text pair dataset.

[0017] Multiple natural images are obtained from the external data source, and predefined descriptive text is randomly assigned to each natural image. Each natural image is paired with its corresponding predefined descriptive text to generate a negative natural image-text pair dataset.

[0018] The document image-text pair dataset, the chart image-text pair dataset, and the negative natural image-text pair dataset are used as the preset image-text pair dataset.

[0019] In one embodiment, the preset multimodal large model includes a multimodal pre-trained model and a large language model. The step of updating the original vocabulary network structure of the preset multimodal large model using the new vocabulary network structure and determining the updated preset multimodal large model includes:

[0020] Obtain the original image embedding layer corresponding to the original vocabulary network structure of the multimodal pre-trained model;

[0021] A new image embedding layer is created in the multimodal pre-trained model based on the new vocabulary network structure;

[0022] The original image embedding layer and the new image embedding layer are concatenated to determine the parallelized image embedding layer, wherein the output vector of the parallelized image embedding layer is obtained by concatenating the output vector of the original image embedding layer and the output vector of the new image embedding layer;

[0023] The image embedding layer of the multimodal pre-trained model is updated according to the parallelized image embedding layer to obtain the updated multimodal pre-trained model;

[0024] The updated preset multimodal large model is determined based on the updated multimodal pre-trained model and the large language model.

[0025] In one embodiment, the step of using the preset visual question-answering dataset as the training dataset for the updated preset multimodal large model, using the first information extraction instruction as the large model instruction for the updated preset multimodal large model, and fine-tuning the updated preset multimodal large model to obtain the fine-tuned preset multimodal large model includes:

[0026] The preset visual question-answering dataset is used as the model input of the updated multimodal pre-trained model, the output vector of the updated multimodal pre-trained model is used as the model input of the large language model, and the first information extraction instruction is used as the large model instruction of the large language model. The updated multimodal pre-trained model and the large language model are jointly iteratively fine-tuned to obtain the preset multimodal large model after fine-tuning.

[0027] In one embodiment, the step of jointly iteratively fine-tuning the updated multimodal pre-trained model and the large language model to obtain the pre-set multimodal large model after fine-tuning includes:

[0028] The visual question-answering data in the preset visual question-answering dataset is input into the updated multimodal pre-trained model to obtain the output vector of the updated multimodal pre-trained model;

[0029] The output vector and the first information extraction instruction are input into the large language model to obtain the output result of the large language model.

[0030] Obtain the actual result corresponding to the output result from the preset visual question answering dataset, and calculate the loss value based on the difference between the output result and the actual result;

[0031] The parameters of the updated multimodal pre-trained model and the large language model to be fine-tuned are updated by backpropagation based on the loss value until the model converges, and then the final fine-tuned model parameters are determined.

[0032] The preset multimodal large model at the end of fine-tuning is determined based on the final fine-tuning model parameters.

[0033] In one embodiment, the step of calculating the loss value based on the difference between the output result and the true result includes:

[0034] Calculate the normalized edit distance and the balanced F-score between the output and the true result;

[0035] Obtain the preset first weight of the normalized edit distance and the preset second weight of the balanced F score;

[0036] The normalized edit distance and the balanced F-score are weighted and calculated according to the preset first weight and the preset second weight to obtain the loss value between the output result and the true result.

[0037] In one embodiment, the model parameters to be fine-tuned include the model parameters of the original image embedding layer, the model parameters of the new image embedding layer, and the model parameters of the large language model.

[0038] Furthermore, to achieve the above objectives, this application also proposes a document snapshot information extraction device, which includes:

[0039] The training module is used to acquire a preset image-text pair dataset, use the preset image-text pair dataset as training data for a pre-trained model, perform autoregressive training on the pre-trained model, and obtain the successfully trained pre-trained model. The preset image-text pair dataset includes a document image-text pair dataset, a chart image-text pair dataset, and a negative natural image-text pair dataset. The pre-trained model is used to generate a new vocabulary based on the preset image-text pair dataset.

[0040] The update module is used to extract the network structure of the successfully trained pre-trained model as a new vocabulary network structure, and use the new vocabulary network structure to update the original vocabulary network structure of the preset multimodal large model, thereby determining the updated preset multimodal large model.

[0041] The fine-tuning module is used to obtain a preset visual question-answering dataset and a first information extraction instruction input by the user, use the preset visual question-answering dataset as the training dataset of the updated preset multimodal large model, use the first information extraction instruction as the large model instruction of the updated preset multimodal large model, and fine-tune the updated preset multimodal large model to obtain the preset multimodal large model after fine-tuning.

[0042] The output module is used to receive a target document snapshot and a second information extraction instruction input by the user, input the target document snapshot and the second information extraction instruction into the fine-tuned preset multimodal large model, and obtain the information extraction result output by the fine-tuned preset multimodal large model.

[0043] In addition, to achieve the above objectives, this application also proposes a document snapshot information extraction device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the document snapshot information extraction method as described above.

[0044] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the document snapshot information extraction method described above.

[0045] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the document snapshot information extraction method described above.

[0046] This application obtains a pre-defined image-text pair dataset and uses it as training data for a pre-trained model. The pre-trained model undergoes autoregressive training to obtain a successfully trained model. This model can generate a new vocabulary based on the input image-text pair data, expanding its vocabulary range and learning richer lexical representations, thus better understanding the relationships between different types of images and text. The network structure of the successfully trained model is extracted as the new vocabulary network structure. This new vocabulary network structure is then used to update the original vocabulary network structure of the pre-defined multimodal large model. The updated pre-defined multimodal large model is determined, and the updated vocabulary provides richer and more accurate lexical representations, enabling the model to handle complex visual-linguistic processing. The model performs better on specific visual question-answering tasks. It acquires a pre-defined visual question-answering dataset and a first information extraction instruction input by the user. The pre-defined visual question-answering dataset is used as the training dataset for an updated pre-defined multimodal large model, and the first information extraction instruction is used as the large model instruction for the updated pre-defined multimodal large model. The updated pre-defined multimodal large model is then fine-tuned to obtain a finely tuned model. Through this fine-tuning, the model achieves better performance on specific visual question-answering tasks. It also receives a target document snapshot and a second information extraction instruction input by the user. These are input into the finely tuned pre-defined multimodal large model to obtain the information extraction result output by the finely tuned model. This application utilizes diverse image-text pair datasets and autoregressive training to enable the model to better understand the relationship between images and text, improving cross-modal understanding capabilities. Through vocabulary updates and fine-tuning, the model's performance on specific tasks is significantly improved, especially in document and chart-related visual question-answering tasks. This makes the model both efficient and accurate in extracting information from complex documents, enabling it to quickly respond to user needs and provide high-quality results. Attached Figure Description

[0047] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0048] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart illustrating an embodiment of the document snapshot information extraction method for this application.

[0050] Figure 2A new vocabulary network structure diagram provided for Embodiment 1 of the document snapshot information extraction method of this application;

[0051] Figure 3 A model training framework diagram for the first stage of model training provided in Embodiment 1 of this application;

[0052] Figure 4 This is a diagram illustrating the two-stage model training framework provided in Embodiment 1 of this application.

[0053] Figure 5 This is a schematic diagram of the module structure of the document snapshot information extraction device according to an embodiment of this application;

[0054] Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the document snapshot information extraction method in this application embodiment.

[0055] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0056] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0057] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0058] Based on this, embodiments of this application provide a method for extracting document snapshot information, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the document snapshot information extraction method of this application.

[0059] In this embodiment, the document snapshot information extraction method includes steps S10 to S40:

[0060] Step S10: Obtain a preset image-text pair dataset, use the preset image-text pair dataset as training data for the pre-trained model, perform autoregressive training on the pre-trained model, and obtain a successfully trained pre-trained model. The preset image-text pair dataset includes a document image-text pair dataset, a chart image-text pair dataset, and a negative natural image-text pair dataset. The pre-trained model is used to generate a new vocabulary based on the preset image-text pair dataset.

[0061] It should be noted that the document image-text pair dataset contains document images (such as scanned PDF pages, tables, etc.) and their corresponding text descriptions; the chart image-text pair dataset contains chart images (such as bar charts, line charts, pie charts, etc.) and their corresponding text descriptions; and the negative natural image-text pair dataset contains natural images unrelated to documents and charts and their corresponding text descriptions. These three types of image-text pair datasets are merged to form a comprehensive dataset, which is used as training data for the pre-trained model. This data is used to enhance the robustness of the model, enabling it to distinguish between relevant and irrelevant image-text pairs. Autoregressive training is a sequence generation method where the model predicts elements in a sequence one by one, with each prediction depending on previously generated elements. The pre-trained model is based on the ViTDet encoder in Meta's open-source SAM, plus a small decoder-only transformer. This model will be further fine-tuned or trained on the aforementioned image-text pair dataset to learn how to generate corresponding text descriptions from images. In multimodal tasks, autoregressive training can be extended to the joint generation of images and text. Through autoregressive training, the model is able to generate new vocabularies from input images and text. Specifically, the model learns how to extract features from images and text and generate new lexical representations. After training, the model can generate a new vocabulary containing richer lexical representations that better captures the relationships between images and text. The new vocabulary network structure refers to the specific architecture used to generate new visual vocabulary. Once the model is trained, its internal structure (especially the decoder part) can be used to build a system specifically designed for generating new visual vocabulary.

[0062] Specifically, in this embodiment, the network structure for training the new vocabulary is trained using the SAM-base model. This embodiment uses a vocabulary network and a small decoder-only transformer to generate the desired new visual vocabulary (i.e., the new vocabulary) through autoregression. The ViTDet image encoder (SAM-base) pre-trained with meta's open-source SAM is used as the main part of the new vocabulary network (the new vocabulary network structure).

[0063] Because of the mismatch between input and output resolutions, the input image resolution of SAM-base is (1024×1024), while the output stride is 16. Therefore, the feature shape of the last layer is (64×64×256), which cannot be aligned with the output of CLIP (Contrastive Language)-L (N×C is 256×1024). Therefore, the features need to be adjusted to align with the output of the CLIP-L model.

[0064] The new vocabulary network structure is as follows: Figure 2 As shown, two convolutional layers are added to the network structure to address the mismatch between the SAM-base output features and the CLIP-L output. The input image size is 3×1024×1024. First, it passes through the SAM-imageencoder, generating a feature shape of 64×64×256. Then, a convolutional layer (Conv1) with a kernel size of 3×3 is added to transform the feature shape to 32×32×512. Next, another convolutional layer (Conv2) is added, also with a kernel size of 3×3, to further transform the feature shape to 16×16×1024. Finally, the output features are flattened to 256×1024 to align with the image token shape of CLIP-VIT.

[0065] During training, three types of image-text pairs (document image-document text, chart image-JSON text, and landscape-text) were input into the model to train the new vocabulary network structure using an autoregressive approach. The loss value was calculated using the OPT-125M model, and the gradient was calculated using backpropagation to adjust the embedding linear layer and the new vocabulary network structure. The new vocabulary network structure was trained and the model parameters were updated using a batch size of 512, an epoch of 3, an AdamW optimizer, and a learning rate of 5e-5.

[0066] Understandably, by acquiring diverse image-text pair datasets and training them autoregressively, a new vocabulary network structure was generated. This not only expanded the model's vocabulary but also enhanced its cross-modal understanding capabilities. The new vocabulary provides richer lexical representations, enabling the model to perform better when handling complex vision-language tasks. This step lays a solid foundation for subsequent vocabulary updates and fine-tuning, ensuring the efficiency and accuracy of the entire information extraction process.

[0067] like Figure 3 As shown in the first stage, the model receives document data, chart data, and natural images, all of which have a common dimension of 3×1024×1024. These three types of data are fed into a new vocabulary network structure, which converts the data into 256*1024 vectors. The vectors are then fed into the embedding layer, where they are converted into 256*4096 vectors. These vectors are then fed into the OPT-125M model to generate new vocabulary results. The generated results are then returned to the OPT-125M model through autoregression.

[0068] Step S20: Extract the network structure of the successfully trained pre-trained model as the new vocabulary network structure, use the new vocabulary network structure to update the original vocabulary network structure of the preset multimodal large model, and determine the updated preset multimodal large model.

[0069] It should be noted that the pre-set multimodal large model is usually a multimodal model that has already been pre-trained on a large amount of data, such as CLIP (Contrastive Language–Image Pre-training) or similar models based on the Transformer architecture. These models have been pre-trained on a large amount of image-text pairs and have preliminary cross-modal understanding capabilities.

[0070] The new vocabulary network structure generated through autoregressive training in step S10 contains richer lexical representations and an expanded vocabulary range. After vocabulary updating, the vocabulary range of the multimodal large model is expanded, enabling it to better handle diverse text content.

[0071] Specifically, to quickly introduce new visual features, the newly generated visual vocabulary needs to be merged with the original CLIP vocabulary. By parallelizing the processing of these two visual vocabularys and using separate input embedding layers (simple linear layers), it can be ensured that the concatenated image token channels are aligned with the input of the Large Language Model (LLM). The original CLIP vocabulary is 1024-dimensional, and the linear layer transforms the 1024-dimensional input into a 2048-dimensional output. By processing the newly generated visual vocabulary in parallel, the two 2048-dimensional outputs are concatenated to form a 4096-dimensional representation, which is perfectly aligned with the input of the LLM (Qwen-7B or Vicuna-7B).

[0072] Understandably, by acquiring a pre-defined multimodal large model and updating its original vocabulary network structure based on the new vocabulary network structure, the updated pre-defined multimodal large model is determined. This step not only expands the model's vocabulary range but also provides richer lexical representations, enabling the model to perform better when handling complex visual-language tasks. The updated vocabulary allows the model to better adapt to specific task requirements, thereby improving the accuracy and practicality of the entire information extraction process.

[0073] In one feasible implementation, step S20, which involves pre-setting a multimodal large model including a multimodal pre-trained model and a large language model, updating the original vocabulary network structure of the pre-set multimodal large model using a new vocabulary network structure, and determining the updated pre-set multimodal large model, includes steps A10 to A50:

[0074] Step A10: Obtain the original image embedding layer corresponding to the original vocabulary network structure of the multimodal pre-trained model.

[0075] It is important to note that, firstly, the specific architecture of the currently used multimodal pre-trained model needs to be clarified. This typically includes the image encoder, the text encoder, and the interaction mechanisms between them. The part of the image encoder responsible for converting the input image into a high-dimensional vector representation, i.e., the image embedding layer, needs to be identified.

[0076] Among these, the multimodal pre-trained model is typically the CLIP model. CLIP is a multimodal pre-trained model developed by OpenAI that can learn the alignment relationships between images and text. CLIP-ViT is a variant of the CLIP model, where ViT (vision transformer) is a vision model based on the Transformer architecture. In CLIP, "vocabulary" usually refers to the vocabulary used to represent parts of text. This vocabulary contains all the words or sub-word units that the model can understand and process. CLIP uses a relatively large vocabulary to cover a wide range of linguistic expressions.

[0077] Step A20: Create a new image embedding layer in the multimodal pre-trained model based on the new vocabulary network structure.

[0078] It's important to note that a new image embedding layer is added to the existing multimodal pre-trained model. The newly generated visual vocabulary has a dimension of 1024, and the output dimension of the new image embedding layer is set to 2048, consistent with the original image embedding layer. A new linear layer is created with an input dimension of 1024 and an output dimension of 2048. This new linear layer will serve as the new image embedding layer, used to process the newly generated visual vocabulary. A linear layer is a commonly used fully connected layer that performs matrix multiplication operations, transforming the input tensor into a higher-dimensional output tensor.

[0079] Step A30: The original image embedding layer and the new image embedding layer are concatenated to determine the parallelized image embedding layer. The output vector of the parallelized image embedding layer is obtained by concatenating the output vector of the original image embedding layer and the output vector of the new image embedding layer.

[0080] Understandably, the output vectors of the original image embedding layer and the new image embedding layer are concatenated to form a higher-dimensional feature representation. This step not only preserves the original model's ability to understand image data but also introduces new visual features, enhancing the model's expressive and generalization capabilities. In this way, the model can better understand and process complex multimodal data, providing stronger support for subsequent tasks.

[0081] Step A40 involves updating the image embedding layer of the multimodal pre-trained model based on the parallelized image embedding layer, resulting in the updated multimodal pre-trained model.

[0082] It should be noted that the parallelized image embedding layer is composed of the original image embedding layer and the new image embedding layer in parallel. In fact, the original image embedding layer is not actually "deleted". Instead, it is used as part of the parallelization process. The parallelized image embedding layer, composed of the original image embedding layer and the new image embedding layer in parallel, replaces the original single original image embedding layer.

[0083] Understandably, parallelized image embedding layers are integrated into the multimodal pre-trained model to replace the original image embedding layers. In this way, the model can utilize both old and new visual features simultaneously, thereby improving its expressive power and generalization ability. This step not only preserves the functionality of the original model but also enhances its ability to understand complex multimodal data. Finally, through testing and validation, it is ensured that the updated model functions correctly and achieves the expected results.

[0084] Step A50: Determine the updated preset multimodal large model based on the updated multimodal pre-trained model and the large language model.

[0085] This implementation updates the pre-defined multimodal large model based on the new vocabulary network structure. This not only preserves the original multimodal pre-trained model's ability to understand images but also incorporates additional information from newly generated visual vocabulary. The updated pre-defined multimodal large model thus constructed possesses stronger expressive power and better generalization performance, making it suitable for more complex multimodal tasks.

[0086] Step S30: Obtain the preset visual question answering dataset and the first information extraction instruction input by the user; use the preset visual question answering dataset as the training dataset of the updated preset multimodal large model; use the first information extraction instruction as the large model instruction of the updated preset multimodal large model; fine-tune the updated preset multimodal large model to obtain the preset multimodal large model after fine-tuning.

[0087] It's important to note that visual question answering datasets typically contain images and associated questions and answers. Examples include VQA (Visual Question Answering) v2.0 and VQA-CP v2. The dataset consists of: images (various types of images, such as document images and charts); questions (natural language questions related to the images); and answers (the responses to the corresponding questions, which can be in text form or coordinates pointing to a location in the image). The first information extraction instruction provides the user with specific information extraction requirements, such as "extract key data from a table" or "identify trends in a chart."

[0088] During fine-tuning, you can choose to freeze the model's lower-level feature extraction layers and only fine-tune the higher-level task headers and other relevant layers. After fine-tuning, the model performs better on specific visual question-answering tasks and can answer image-related questions more accurately.

[0089] Understandably, by acquiring a pre-defined visual question-answering dataset and initial information extraction instructions from the user, the dataset is used as training data for the updated multimodal large model, and the instructions serve as guidance for fine-tuning the model. This step not only improves the model's performance on specific tasks but also enhances its flexibility and accuracy, enabling it to better meet specific user needs. The fine-tuned model is both efficient and accurate in extracting information from complex documents, quickly responding to user needs and providing high-quality results.

[0090] In one feasible implementation, step S30, which uses a preset visual question-answering dataset as the training dataset for the updated preset multimodal large model, uses the first information extraction instruction as the large model instruction for the updated preset multimodal large model, and fine-tunes the updated preset multimodal large model to obtain the fine-tuned preset multimodal large model, includes step S31:

[0091] Step S31: Use the preset visual question answering dataset as the model input of the updated multimodal pre-trained model, use the output vector of the updated multimodal pre-trained model as the model input of the large language model, use the first information extraction instruction as the large model instruction of the large language model, and perform joint iterative fine-tuning on the updated multimodal pre-trained model and the large language model to obtain the preset multimodal large model after fine-tuning.

[0092] It should be noted that visual question answering datasets are typically VQA datasets. The joint iterative fine-tuning process includes: inputting the image into the updated multimodal pre-trained model to obtain high-dimensional feature representations; passing these feature representations as input to the large language model; calculating the loss based on the output of the large language model and the ground truth labels; and updating the parameters of the multimodal pre-trained model and the large language model through the backpropagation algorithm.

[0093] like Figure 4 As shown in the second stage, the multimodal pre-trained model receives visual question-answering data and simultaneously feeds the visual question-answering data into the original vocabulary network structure and the new vocabulary network structure, converting them into 256*1024 vectors respectively. Both image vectors pass through the embedding layer and are converted into 256*2048 vectors respectively. The two image vectors are concatenated to form a 256*2048 vector. The concatenated vector is fed into the large language model. At the same time, the first information extraction instruction is fed into the large language model, freezing the labeled blocks (the frozen parts) and training the parts in the unlabeled blocks.

[0094] In one feasible implementation, step S31, which involves jointly iteratively fine-tuning the updated multimodal pre-trained model and the large language model to obtain the pre-defined multimodal large model after fine-tuning, includes steps B10 to B50:

[0095] Step B10: Input the visual question-answering data from the preset visual question-answering dataset into the updated multimodal pre-trained model to obtain the output vector of the updated multimodal pre-trained model.

[0096] Visual question answering data is input into the original vocabulary network structure and the new vocabulary network structure of the updated multimodal pre-trained model, which run in parallel, to obtain the output vector of the concatenated multimodal pre-trained model.

[0097] Specifically, the model uses parallelized image embedding layers (such as the original image embedding layer and the new image embedding layer). The image is processed through these two embedding layers to generate two feature vectors. These two image feature vectors are concatenated into a higher-dimensional feature vector. The resulting high-dimensional feature vector contains rich information from the image and text and can be used as input to a large language model.

[0098] Step B20: Input the output vector and the first information extraction instruction into the large language model to obtain the output result of the large language model.

[0099] The output vector and the first information extraction instruction are input into the large language model, which will generate the output result based on these inputs.

[0100] Step B30: Obtain the actual result corresponding to the output result from the preset visual question answering dataset, and calculate the loss value based on the difference between the output result and the actual result.

[0101] The model reads the image-question pairs and their corresponding standard answers from the pre-set visual question-answering dataset for the current batch, inputs the model's output and the actual results into the selected loss function, and calculates the loss value.

[0102] In one feasible implementation, step B30, which calculates the loss value based on the difference between the output result and the actual result, includes steps C10 to C30:

[0103] Step C10: Calculate the normalized edit distance and balanced F-score between the output and the true results.

[0104] It's important to note that normalized edit distance measures the similarity between two strings based on the minimum number of single-character editing operations (insertion, deletion, or replacement) required to transform one string into the other. The balanced F-score, also known as the F1 score, measures the overlap between the model's predicted and actual answers. It is the harmonic mean of precision and recall. Precision is the proportion of correctly predicted positive samples out of all predicted positive samples, and recall is the proportion of correctly predicted positive samples out of all actual positive samples. Specifically, the edit distance is calculated using dynamic programming, and then divided by the length of the longer string to obtain the normalized edit distance. The predicted and actual answers are segmented into words, the number of co-occurring words is calculated, precision and recall are calculated using the formula, and then the F1 score is calculated.

[0105] Step C20: Obtain the preset first weight of the normalized edit distance and the preset second weight of the balanced F score.

[0106] It should be noted that the preset first weight of the normalized edit distance and the preset second weight of the balanced F score need to be determined based on experience or experiments.

[0107] Step C30: The normalized edit distance and F1 score are weighted according to the preset first weight and preset second weight to obtain the loss value between the output result and the actual result.

[0108] The normalized edit distance and F1 score are weighted and summed according to preset weights to obtain the final loss value. Since the normalized edit distance is better when it is smaller, while the F1 score is better when it is larger, it is necessary to take the complement of the F1 score (i.e., 1-F1 score).

[0109] This implementation constructs a comprehensive loss function by calculating the normalized edit distance and F1 score, and combining them with preset weights. This allows for a more comprehensive evaluation of the model's performance and guides the optimization of model parameters.

[0110] Step B40: Update the parameters of the multimodal pre-trained model and the large language model to be fine-tuned based on the backpropagation of the loss value until the model converges, and then determine the final fine-tuned model parameters.

[0111] Based on the loss value calculated during forward propagation, the gradient of each learnable parameter is calculated using an automatic differentiation mechanism (such as autograd in PyTorch). The selected optimization algorithm is used to update the model parameters. The model weights are updated according to the calculated gradients, learning rate, and any other possible hyperparameters. The changes in the loss value are periodically checked to determine whether the predetermined convergence condition has been met.

[0112] In one feasible implementation, the model parameters to be fine-tuned include the model parameters of the original image embedding layer, the model parameters of the new image embedding layer, and the model parameters of the large language model.

[0113] During fine-tuning, most network parameters are frozen, and only the unfrozen parameters are updated to adapt the model to new tasks or datasets. For example, in this application, the two network structures in the visual encoder of the multimodal pre-trained model and the concatenated output vector are frozen, and only the model parameters of the original image embedding layer, the new image embedding layer, and the large language model are updated. See the specific references. Figure 4 In this process, the marked blocks are frozen, while the unmarked blocks are updated.

[0114] Step B50: Determine the preset multimodal large model at the end of fine-tuning based on the final fine-tuning model parameters.

[0115] This implementation method, after updating the multimodal pre-trained model and integrating it with the large language model, fine-tunes the entire system using a specific task dataset and task instructions. The updated multimodal pre-trained model can extract richer and more representative features, while the large language model can better understand the contextual relationships between images and text, generating high-quality answers. The joint optimization of the multimodal pre-trained model and the large language model ensures that they work together to improve the overall system performance. Finally, the fine-tuned preset multimodal large model performs well on specific tasks, providing more accurate and reliable solutions.

[0116] Step S40: Receive the target document snapshot and the second information extraction instruction input by the user, input the target document snapshot and the second information extraction instruction into the fine-tuned preset multimodal large model, and obtain the information extraction result output by the fine-tuned preset multimodal large model.

[0117] It should be noted that the target document snapshot is an image of the document provided by the user from which information needs to be extracted; this can be a scanned PDF page, table, chart, etc. The second information extraction instruction is the specific information extraction requirement provided by the user, such as "extract key data from the table," "identify trends in the chart," or "extract the document title."

[0118] The preprocessed image is input into the visual encoder of the model, and the preprocessed instruction embedding vector is input into the language encoder of the model. The image features and instruction embedding vector are processed to form a comprehensive input representation.

[0119] This embodiment acquires a pre-defined image-text pair dataset and uses it as training data for a pre-trained model. The pre-trained model undergoes autoregressive training to obtain a successfully trained model. This model can generate a new vocabulary based on the input image-text pair data, expanding its vocabulary range and learning richer lexical representations, thus better understanding the relationships between different types of images and text. The network structure of the successfully trained model is extracted as the new vocabulary network structure. This new vocabulary network structure is used to update the original vocabulary network structure of the pre-defined multimodal large model, determining the updated pre-defined multimodal large model. The updated vocabulary provides richer and more accurate lexical representations, enabling the model to handle complex visual-linguistic processing. The model performs better on specific visual question-answering tasks. It acquires a preset visual question-answering dataset and a first information extraction instruction input by the user. The preset visual question-answering dataset is used as the training dataset for the updated preset multimodal large model, and the first information extraction instruction is used as the large model instruction for the updated preset multimodal large model. The updated preset multimodal large model is fine-tuned to obtain the fine-tuned preset multimodal large model. Through fine-tuning the updated multimodal large model, the model achieves better performance on specific visual question-answering tasks. It receives a target document snapshot and a second information extraction instruction input by the user. The target document snapshot and the second information extraction instruction are input into the fine-tuned preset multimodal large model to obtain the information extraction result output by the fine-tuned preset multimodal large model. This embodiment, through diverse image-text pair datasets and autoregressive training, enables the model to better understand the relationship between images and text, improving cross-modal understanding capabilities. Through vocabulary updates and fine-tuning, the model's performance on specific tasks is significantly improved, especially in document and chart-related visual question-answering tasks. This makes the model both efficient and accurate in the information extraction process of complex documents, enabling it to quickly respond to user needs and provide high-quality results.

[0120] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. Furthermore, before step S10, the document snapshot information extraction method further includes steps S01 to S06:

[0121] Step S01: Collect multiple raw document data containing plain text, data formulas, and tables from external data sources, and extract plain text data, data formula data, and table data from each raw document data.

[0122] Step S02: Based on multiple predefined templates, render plain text data, data formula data, and tabular data to obtain the first rendered document file. The document types corresponding to each predefined template are different.

[0123] Step S03: Convert each first document file into a document image, convert each first document file into a second document file in a preset format, and pair each document image with the corresponding second document file to generate a document image text pair dataset.

[0124] It should be noted that the external data source is arXiv, a preprint server covering academic papers in multiple fields such as physics, mathematics, and computer science. The original document data is in .tex source files. The default format is the unified Matpix Markdown format. Predefined templates can be divided into different types, such as academic paper templates, technical report templates, and corporate report templates, each with different compositions and key content. Rendering refers to filling the extracted plain text data into the corresponding positions in the template.

[0125] Specifically, we collected a large number of .tex source files from arXiv. We used regular expressions to extract tables, mathematical formulas, and plain text from these files. We collected over 10 templates to perform batch rendering, and then used pdflatex to re-render the extracted content into PDF format based on the templates. We converted the actual text values ​​(i.e., the extracted text content) of each document page to Mathpix Markdown style for standardized formatting. Through this process, we obtained 500,000 English pages and 400,000 Chinese pages.

[0126] Step S04: Obtain multiple original charts from external data sources, extract the actual text values ​​from each original chart, convert each chart text value into JSON (JavaScript Object Notation) data, and use each JSON data as a chart image text pair dataset.

[0127] Specifically, matplotlib and pyecharts were chosen as the chart rendering tools to create 250,000 matplotlib-style or pyecharts-style charts in both Chinese and English. The actual text values ​​of the charts (such as titles, x-axis and y-axis labels) were randomly selected from a natural language processing corpus downloaded from the internet. The actual text values ​​of each chart (i.e., the text content in the chart) were converted into JSON format to standardize the chart text format and facilitate subsequent processing and use.

[0128] Step S05: Obtain multiple natural images from an external data source, randomly assign predefined descriptive text to each natural image, pair each natural image with its corresponding predefined descriptive text, and generate a negative natural image-text pair dataset.

[0129] Step S06: Use the document image-text pair dataset, the chart image-text pair dataset, and the negative natural image-text pair dataset as the preset image-text pair dataset.

[0130] Specifically, to ensure that newly introduced visual vocabulary does not introduce noise when processing natural images, negative natural image-text pairs need to be constructed. The COCO (Common Objects in Context) dataset is a large-scale image recognition, segmentation, and annotation dataset containing a large number of natural images and corresponding annotation information. This dataset covers a variety of scenes and objects, making it suitable for constructing a negative sample dataset. 120,000 images were extracted from the COCO dataset, covering different scenes and content, and a fixed text description was randomly selected for each image. The text portion was randomly selected from the following sentences: "This is a natural image"; "This is a natural picture"; "This is a natural photograph"; "This is a natural image"; "This is a lens from nature".

[0131] By introducing negative samples, we ensure that the model is not affected by newly introduced visual vocabulary when processing natural images, reducing the possibility of misidentification. Negative samples help improve the model's ability to distinguish between different types of images and enhance its robustness.

[0132] This embodiment generates various types of text-image pair datasets by collecting and processing raw document data, chart data, and natural image data from external data sources. These datasets are not only diverse and rich but also consistent and structured, providing high-quality data support for subsequent model training and applications. Through these steps, the system can generate professional and practical documents and charts, and the introduction of negative samples enhances the model's robustness and generalization ability, thereby significantly improving the model's performance in multimodal information processing tasks.

[0133] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the document snapshot information extraction method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0134] This application also provides a document snapshot information extraction device, please refer to... Figure 5 The document snapshot information extraction device includes:

[0135] Training module 10 is used to acquire a preset image-text pair dataset, use the preset image-text pair dataset as training data for a pre-trained model, perform autoregressive training on the pre-trained model, and obtain the successfully trained pre-trained model. The preset image-text pair dataset includes a document image-text pair dataset, a chart image-text pair dataset, and a negative natural image-text pair dataset. The pre-trained model is used to generate a new vocabulary based on the preset image-text pair dataset.

[0136] Update module 20 is used to extract the network structure of the successfully trained pre-trained model as a new vocabulary network structure, and use the new vocabulary network structure to update the original vocabulary network structure of the preset multimodal large model, thereby determining the updated preset multimodal large model.

[0137] The fine-tuning module 30 is used to obtain a preset visual question-answering dataset and a first information extraction instruction input by the user, use the preset visual question-answering dataset as the training dataset of the updated preset multimodal large model, use the first information extraction instruction as the large model instruction of the updated preset multimodal large model, and fine-tune the updated preset multimodal large model to obtain the preset multimodal large model after fine-tuning.

[0138] The output module 40 is used to receive the target document snapshot and the second information extraction instruction input by the user, input the target document snapshot and the second information extraction instruction into the fine-tuned preset multimodal large model, and obtain the information extraction result output by the fine-tuned preset multimodal large model.

[0139] The document snapshot information extraction device provided in this application, employing the document snapshot information extraction method in the above embodiments, can solve the technical problem of how to improve the accuracy of information extraction from complex document snapshots. Compared with the prior art, the beneficial effects of the document snapshot information extraction device provided in this application are the same as those of the document snapshot information extraction method provided in the above embodiments, and other technical features in the document snapshot information extraction device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0140] This application provides a document snapshot information extraction device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the document snapshot information extraction method in the first embodiment described above.

[0141] The following is for reference. Figure 6The diagram illustrates a structural schematic of a document snapshot information extraction device suitable for implementing embodiments of this application. The document snapshot information extraction device in this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The document snapshot information extraction device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0142] like Figure 6 As shown, the document snapshot information extraction device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the document snapshot information extraction device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the document snapshot information extraction device to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show document snapshot information extraction devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0143] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0144] The document snapshot information extraction device provided in this application, employing the document snapshot information extraction method in the above embodiments, can solve the technical problem of how to improve the accuracy of information extraction from complex document snapshots. Compared with the prior art, the beneficial effects of the document snapshot information extraction device provided in this application are the same as those of the document snapshot information extraction method provided in the above embodiments, and other technical features in this document snapshot information extraction device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0145] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0146] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0147] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the document snapshot information extraction method in the above embodiments.

[0148] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described document snapshot information extraction method, thereby solving the technical problem of how to improve the accuracy of information extraction from complex document snapshots. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the document snapshot information extraction method provided in the above embodiments, and will not be repeated here.

[0149] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the document snapshot information extraction method described above.

[0150] The computer program product provided in this application solves the technical problem of how to improve the accuracy of information extraction from complex document snapshots. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the document snapshot information extraction method provided in the above embodiments, and will not be repeated here.

[0151] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for extracting document snapshot information, characterized in that, The document snapshot information extraction method includes: A preset image-text pair dataset is obtained, and the preset image-text pair dataset is used as the training data for a pre-trained model. The pre-trained model is then subjected to autoregressive training to obtain the successfully trained pre-trained model. The preset image-text pair dataset includes a document image-text pair dataset, a chart image-text pair dataset, and a negative natural image-text pair dataset. The pre-trained model is used to generate a new vocabulary based on the preset image-text pair dataset. Extract the network structure of the successfully trained pre-trained model as a new vocabulary network structure, use the new vocabulary network structure to update the original vocabulary network structure of the preset multimodal large model, and determine the updated preset multimodal large model; Obtain a preset visual question-answering dataset and a first information extraction instruction input by the user. Use the preset visual question-answering dataset as the training dataset for the updated preset multimodal large model. Use the first information extraction instruction as the large model instruction for the updated preset multimodal large model. Fine-tune the updated preset multimodal large model to obtain the preset multimodal large model after fine-tuning. The system receives a target document snapshot and a second information extraction instruction from the user, inputs the target document snapshot and the second information extraction instruction into the fine-tuned preset multimodal large model, and obtains the information extraction result output by the fine-tuned preset multimodal large model.

2. The document snapshot information extraction method as described in claim 1, characterized in that, Before the step of obtaining the preset image-text pair dataset, the following steps are included: Collect raw document data containing plain text, data formulas, and tables from external data sources, and extract plain text data, data formula data, and table data from each of the raw document data. Based on multiple predefined templates, the plain text data, the data formula data, and the table data are rendered to obtain a rendered first document file, wherein each predefined template corresponds to a different document type; Each of the first document files is converted into a document image, and each of the first document files is converted into a second document file in text format. Each document image is paired with its corresponding second document file to generate a document image-text pair dataset. Multiple original charts are obtained from the external data source. The actual values ​​of the chart text are extracted from each of the original charts. The actual values ​​of the chart text are converted into JSON data. The JSON data is used as a chart image text pair dataset. Multiple natural images are obtained from the external data source, and predefined descriptive text is randomly assigned to each natural image. Each natural image is paired with its corresponding predefined descriptive text to generate a negative natural image-text pair dataset. The document image-text pair dataset, the chart image-text pair dataset, and the negative natural image-text pair dataset are used as the preset image-text pair dataset.

3. The document snapshot information extraction method as described in claim 1, characterized in that, The preset multimodal large model includes a multimodal pre-trained model and a large language model. The step of updating the original vocabulary network structure of the preset multimodal large model using the new vocabulary network structure and determining the updated preset multimodal large model includes: Obtain the original image embedding layer corresponding to the original vocabulary network structure of the multimodal pre-trained model; A new image embedding layer is created in the multimodal pre-trained model based on the new vocabulary network structure; The original image embedding layer and the new image embedding layer are concatenated to determine the parallelized image embedding layer, wherein the output vector of the parallelized image embedding layer is obtained by concatenating the output vector of the original image embedding layer and the output vector of the new image embedding layer; The image embedding layer of the multimodal pre-trained model is updated according to the parallelized image embedding layer to obtain the updated multimodal pre-trained model; The updated preset multimodal large model is determined based on the updated multimodal pre-trained model and the large language model.

4. The document snapshot information extraction method as described in claim 3, characterized in that, The steps of using the preset visual question-answering dataset as the training dataset for the updated preset multimodal large model, using the first information extraction instruction as the large model instruction for the updated preset multimodal large model, and fine-tuning the updated preset multimodal large model to obtain the fine-tuned preset multimodal large model include: The preset visual question-answering dataset is used as the model input of the updated multimodal pre-trained model, the output vector of the updated multimodal pre-trained model is used as the model input of the large language model, and the first information extraction instruction is used as the large model instruction of the large language model. The updated multimodal pre-trained model and the large language model are jointly iteratively fine-tuned to obtain the preset multimodal large model after fine-tuning.

5. The document snapshot information extraction method as described in claim 4, characterized in that, The step of jointly iteratively fine-tuning the updated multimodal pre-trained model and the large language model to obtain the pre-set multimodal large model after fine-tuning includes: The visual question-answering data in the preset visual question-answering dataset is input into the updated multimodal pre-trained model to obtain the output vector of the updated multimodal pre-trained model; The output vector and the first information extraction instruction are input into the large language model to obtain the output result of the large language model. Obtain the actual result corresponding to the output result from the preset visual question answering dataset, and calculate the loss value based on the difference between the output result and the actual result; The parameters of the updated multimodal pre-trained model and the large language model to be fine-tuned are updated by backpropagation based on the loss value until the model converges, and then the final fine-tuned model parameters are determined. The preset multimodal large model at the end of fine-tuning is determined based on the final fine-tuning model parameters.

6. The document snapshot information extraction method as described in claim 5, characterized in that, The step of calculating the loss value based on the difference between the output result and the actual result includes: Calculate the normalized edit distance and the balanced F-score between the output and the true result; Obtain the preset first weight of the normalized edit distance and the preset second weight of the balanced F score; The normalized edit distance and the balanced F-score are weighted and calculated according to the preset first weight and the preset second weight to obtain the loss value between the output result and the true result.

7. The document snapshot information extraction method as described in claim 5, characterized in that, The model parameters to be fine-tuned include the model parameters of the original image embedding layer, the model parameters of the new image embedding layer, and the model parameters of the large language model.

8. A document snapshot information extraction device, characterized in that, The document snapshot information extraction device includes: The training module is used to acquire a preset image-text pair dataset, use the preset image-text pair dataset as training data for a pre-trained model, perform autoregressive training on the pre-trained model, and obtain the successfully trained pre-trained model. The preset image-text pair dataset includes a document image-text pair dataset, a chart image-text pair dataset, and a negative natural image-text pair dataset. The pre-trained model is used to generate a new vocabulary based on the preset image-text pair dataset. The update module is used to extract the network structure of the successfully trained pre-trained model as a new vocabulary network structure, and use the new vocabulary network structure to update the original vocabulary network structure of the preset multimodal large model, thereby determining the updated preset multimodal large model. The fine-tuning module is used to obtain a preset visual question-answering dataset and a first information extraction instruction input by the user, use the preset visual question-answering dataset as the training dataset of the updated preset multimodal large model, use the first information extraction instruction as the large model instruction of the updated preset multimodal large model, and fine-tune the updated preset multimodal large model to obtain the preset multimodal large model after fine-tuning. The output module is used to receive a target document snapshot and a second information extraction instruction input by the user, input the target document snapshot and the second information extraction instruction into the fine-tuned preset multimodal large model, and obtain the information extraction result output by the fine-tuned preset multimodal large model.

9. A document snapshot information extraction device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the document snapshot information extraction method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the document snapshot information extraction method as described in any one of claims 1 to 7.

Images