Document snapshot information extraction method and device, equipment and storage medium

Abstract

The invention discloses a document snapshot information extraction method and device, equipment and a storage medium, and relates to the technical field of multi-modal learning, and the method comprises the steps: obtaining a preset image-text pair data set, carrying out the autoregression training of a pre-training model, and obtaining a successfully trained pre-training model; extracting a network structure of the successfully trained pre-training model as a new word list network structure, updating an original word list network structure of the preset multi-modal large model, and determining the updated preset multi-modal large model; obtaining a preset visual question and answer data set and a first information extraction instruction input by a user, and performing fine tuning on the updated preset multi-modal large model to obtain the preset multi-modal large model after fine tuning is finished; and receiving a target document snapshot input by the user and a second information extraction instruction, and obtaining an information extraction result output by the fine-tuned preset multi-modal large model. According to the method, the accuracy of information extraction in the complex document snapshot can be improved.

Description

Technical Field

[0001] The present application relates to the field of multimodal learning technology, and in particular to a document snapshot information extraction method, device, equipment and storage medium. Background Art

[0002] With the rapid development of artificial intelligence technology, multimodal learning, as a technology that can integrate and process various types of data, has shown broad application potential in many fields, especially in the intersection of computer vision and natural language processing. It not only enhances the system's ability to understand complex information, but also promotes the development of new interaction methods.

[0003] In the process of document digital transformation, optical character recognition (OCR) is a key technology that can convert text in images into editable and searchable electronic text formats. Although the current OCR technology has shown a high accuracy rate in converting ordinary text, its accuracy and stability still need to be improved when processing official documents with complex backgrounds or formats. For example, when processing official document snapshots, since these documents usually contain complex formats, charts, seals and other elements, OCR technology often has difficulty in accurately identifying and correctly parsing, resulting in low accuracy in document snapshot information extraction.

[0004] Therefore, how to improve the accuracy of information extraction in complex document snapshots is a problem that needs to be solved urgently.

[0005] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0006] The main purpose of this application is to provide a document snapshot information extraction method, device, equipment and storage medium, aiming to solve the technical problem of how to improve the accuracy of information extraction in complex document snapshots.

[0007] To achieve the above objectives, the present application proposes a document snapshot information extraction method, the document snapshot information extraction method comprising: Obtain a preset image-text pair data set, use the preset image-text pair data set as training data for a pre-trained model, perform autoregressive training on the pre-trained model, and obtain the successfully trained pre-trained model, wherein the preset image-text pair data set includes a document image-text pair data set, a chart image-text pair data set, and a negative natural image-text pair data set, and the pre-trained model is used to generate a new vocabulary according to the preset image-text pair data set; Extracting the network structure of the successfully trained pre-trained model as a new vocabulary network structure, using the new vocabulary network structure to update the original vocabulary network structure of the preset multimodal large model, and determining the updated preset multimodal large model; Obtaining a preset visual question answering dataset and a first information extraction instruction input by a user, using the preset visual question answering dataset as a training dataset of the updated preset multimodal large model, using the first information extraction instruction as a large model instruction of the updated preset multimodal large model, and fine-tuning the updated preset multimodal large model to obtain the preset multimodal large model after fine-tuning; Receive a target document snapshot and a second information extraction instruction input by a user, input the target document snapshot and the second information extraction instruction into the fine-tuned preset multimodal large model, and obtain the information extraction result output by the fine-tuned preset multimodal large model.

[0008] In one embodiment, before the step of obtaining a preset image-text pair data set, the following steps are included: Collecting a plurality of original document data including plain text, data formulas and tables from an external data source, and extracting plain text data, data formula data and table data from each of the original document data; Based on a plurality of predefined templates, the plain text data, the data formula data and the table data are rendered to obtain a rendered first document file, wherein each of the predefined templates corresponds to a different document type; Converting each of the first document files into a document image, and converting each of the first document files into a second document file in text format, and pairing each of the document images with a corresponding second document file to generate a document image-text pair data set; Acquire multiple original charts from the external data source, extract the chart text true value from each of the original charts, convert each of the chart text true value into JSON data, and use each of the JSON data as a chart image text pair data set; Acquire multiple natural images from the external data source, randomly assign predefined description text to each of the natural images, pair each of the natural images with the corresponding predefined description text, and generate a negative natural image-text pair data set; The document image-text pair dataset, the chart image-text pair dataset and the negative natural image-text pair dataset are used as the preset image-text pair datasets.

[0009] In one embodiment, the preset multimodal large model includes a multimodal pre-trained model and a large language model, and the step of updating the original vocabulary network structure of the preset multimodal large model using the new vocabulary network structure to determine the updated preset multimodal large model includes: Obtaining an original image embedding layer corresponding to the original vocabulary network structure of the multimodal pre-trained model; Creating a new image embedding layer in the multimodal pre-trained model according to the new vocabulary network structure; The original image embedding layer and the new image embedding layer are spliced ​​to determine a parallelized image embedding layer, wherein an output vector of the parallelized image embedding layer is obtained by splicing an output vector of the original image embedding layer and an output vector of the new image embedding layer; Updating the image embedding layer of the multimodal pre-trained model according to the parallelized image embedding layer to obtain an updated multimodal pre-trained model; The updated preset multimodal large model is determined according to the updated multimodal pre-trained model and the large language model.

[0010] In one embodiment, the step of using the preset visual question answering dataset as a training dataset of the updated preset multimodal large model, using the first information extraction instruction as a large model instruction of the updated preset multimodal large model, and fine-tuning the updated preset multimodal large model to obtain the preset multimodal large model after fine-tuning includes: The preset visual question answering dataset is used as the model input of the updated multimodal pre-trained model, the output vector of the updated multimodal pre-trained model is used as the model input of the large language model, the first information extraction instruction is used as the large model instruction of the large language model, the updated multimodal pre-trained model and the large language model are jointly iteratively fine-tuned, and the preset multimodal large model after fine-tuning is obtained.

[0011] In one embodiment, the step of jointly iteratively fine-tuning the updated multimodal pre-trained model and the large language model to obtain the preset multimodal large model after fine-tuning includes: Inputting the visual question answering data in the preset visual question answering dataset into the updated multimodal pre-trained model to obtain an output vector of the updated multimodal pre-trained model; Inputting the output vector and the first information extraction instruction into the large language model to obtain an output result of the large language model; Obtaining a true result corresponding to the output result from the preset visual question answering dataset, and calculating a loss value according to a difference between the output result and the true result; Back-propagating the loss value to update the updated multimodal pre-trained model and the model parameters to be fine-tuned of the large language model until the model converges, and then determining the final fine-tuning model parameters; The preset multimodal large model after fine-tuning is determined according to the final fine-tuning model parameters.

[0012] In one embodiment, the step of calculating the loss value according to the difference between the output result and the true result comprises: Calculating a normalized edit distance and a balanced F score between the output result and the true result; Obtaining a preset first weight of the normalized edit distance and a preset second weight of the balanced F-score; The normalized edit distance and the balanced F-score are weightedly calculated according to the preset first weight and the preset second weight to obtain a loss value between the output result and the true result.

[0013] In one embodiment, the model parameters to be fine-tuned include model parameters of the original image embedding layer, model parameters of the new image embedding layer, and model parameters of the large language model.

[0014] In addition, to achieve the above purpose, the present application also proposes a document snapshot information extraction device, the document snapshot information extraction device comprising: A training module, used to obtain a preset image-text pair data set, use the preset image-text pair data set as training data for a pre-trained model, perform autoregressive training on the pre-trained model, and obtain the successfully trained pre-trained model, wherein the preset image-text pair data set includes a document image-text pair data set, a chart image-text pair data set, and a negative natural image-text pair data set, and the pre-trained model is used to generate a new vocabulary according to the preset image-text pair data set; An updating module, used for extracting the network structure of the successfully trained pre-trained model as a new vocabulary network structure, using the new vocabulary network structure to update the original vocabulary network structure of the preset multimodal large model, and determining the updated preset multimodal large model; A fine-tuning module is used to obtain a preset visual question answering dataset and a first information extraction instruction input by a user, use the preset visual question answering dataset as a training dataset of the updated preset multimodal large model, use the first information extraction instruction as a large model instruction of the updated preset multimodal large model, and fine-tune the updated preset multimodal large model to obtain the preset multimodal large model after fine-tuning; An output module is used to receive a target document snapshot and a second information extraction instruction input by a user, input the target document snapshot and the second information extraction instruction into the fine-tuned preset multimodal large model, and obtain the information extraction result output by the fine-tuned preset multimodal large model.

[0015] In addition, to achieve the above-mentioned purpose, the present application also proposes a document snapshot information extraction device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the document snapshot information extraction method described above.

[0016] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the document snapshot information extraction method described above are implemented.

[0017] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the document snapshot information extraction method described above are implemented.

[0018] The present application obtains a preset image-text pair data set, uses the preset image-text pair data set as training data for a pre-trained model, performs autoregressive training on the pre-trained model, and obtains a successfully trained pre-trained model, so that the pre-trained model can generate a new vocabulary based on the input image-text pair data, which helps to expand the vocabulary range of the model and learn richer vocabulary representations, so as to better understand the associations between different types of images and texts; extracts the network structure of the successfully trained pre-trained model as the new vocabulary network structure, uses the new vocabulary network structure to update the original vocabulary network structure of the preset multimodal large model, and determines the updated preset multimodal large model. The updated vocabulary can provide richer and more accurate vocabulary representations, so that the model can process complex visual-language The method can achieve better performance on specific visual question-answering tasks; obtain a preset visual question-answering dataset and a first information extraction instruction input by the user, use the preset visual question-answering dataset as a training dataset for the updated preset multimodal large model, use the first information extraction instruction as the large model instruction of the updated preset multimodal large model, fine-tune the updated preset multimodal large model, and obtain the preset multimodal large model after fine-tuning. By fine-tuning the updated multimodal large model, the model can achieve better performance on specific visual question-answering tasks; receive a target document snapshot and a second information extraction instruction input by the user, input the target document snapshot and the second information extraction instruction into the fine-tuned preset multimodal large model, and obtain the information extraction result output by the fine-tuned preset multimodal large model. This application uses a variety of image-text pair datasets and autoregressive training to enable the model to better understand the relationship between images and texts, improve cross-modal understanding capabilities, and significantly improve the performance of the model on specific tasks through vocabulary updates and fine-tuning, especially in visual question-answering tasks related to documents and charts, so that the model is both efficient and accurate in the information extraction process of complex documents, and can quickly respond to user needs and provide high-quality results. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 A flowchart of the first embodiment of the method for extracting snapshot information from a document of the present application is provided; Figure 2 A new vocabulary network structure diagram provided for the first embodiment of the document snapshot information extraction method of the present application; Figure 3 A model training framework diagram for the first stage of model training provided in Example 1 of the present application; Figure 4 A model training framework diagram for the second stage of model training provided in Example 1 of the present application; Figure 5 This is a schematic diagram of the module structure of the document snapshot information extraction device according to an embodiment of the present application; Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the document snapshot information extraction method in the embodiment of the present application.

[0022] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0023] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0024] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0025] Based on this, the embodiment of the present application provides a document snapshot information extraction method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the document snapshot information extraction method of the present application.

[0026] In this embodiment, the document snapshot information extraction method includes steps S10 to S40: Step S10, obtaining a preset image-text pair dataset, using the preset image-text pair dataset as training data for a pre-trained model, performing autoregressive training on the pre-trained model, and obtaining a successfully trained pre-trained model, wherein the preset image-text pair dataset includes a document image-text pair dataset, a chart image-text pair dataset, and a negative natural image-text pair dataset, and the pre-trained model is used to generate a new vocabulary based on the preset image-text pair dataset.

[0027] It should be noted that the document image text pair dataset contains document images (such as scanned PDF (Portable Document Format) pages, tables, etc.) and their corresponding text descriptions, the chart image text pair dataset contains chart images (such as bar charts, line charts, pie charts, etc.) and their corresponding text descriptions, and the negative natural image text pair dataset contains natural images unrelated to documents and charts and their corresponding text descriptions. The above three types of image-text pair datasets are merged to form a comprehensive dataset, which is used as training data for the pre-trained model. These data are used to enhance the robustness of the model so that it can distinguish between related and unrelated image-text pairs. Autoregressive training is a sequence generation method in which the model predicts elements in the sequence one by one, and the prediction of each element depends on the previously generated elements. The pre-trained model is based on the ViTDet encoder in Meta's open source SAM plus a small decoder (decoder-only transformer). This model will be further fine-tuned or trained on the above-mentioned image-text pair datasets to learn how to generate corresponding text descriptions based on images. In multimodal tasks, autoregressive training can be extended to the joint generation of images and text. Through autoregressive training, the model is able to generate a new vocabulary based on the input image and text data. Specifically, the model learns how to extract features from images and text and generate new vocabulary representations. After training, the model is able to generate a new vocabulary that contains richer vocabulary representations and can better capture the relationship between images and text. The new vocabulary network structure refers to the specific architecture used to generate new visual vocabulary. Once the model training is completed, its internal structure (especially the decoder part) can be used to build a system specifically for generating new visual vocabulary.

[0028] Specifically, the SAM-base model is used in this embodiment to train the network structure of the new vocabulary. This embodiment uses a vocabulary network and a small decoder-only transformer to generate the required new visual vocabulary (i.e., the new vocabulary) through autoregression. The meta open source SAM pre-trained ViTDet image encoder (SAM-base) is used as the main part of the new vocabulary network (new vocabulary network structure).

[0029] Since the input and output resolutions do not match, the input image resolution of SAM-base is (1024×1024) and the output stride is 16, the feature shape of the last layer is (64×64×256), which cannot be aligned with the output of CLIP (Contrastive Language)-L (N×C is 256×1024), so the features need to be adjusted to align with the output of the CLIP-L model.

[0030] The new vocabulary network structure is as follows Figure 2 As shown in Figure 1, two convolutional layers are added to the network structure to solve the problem that the output features of SAM-base do not match the output of CLIP-L. The input image size is 3×1024×1024. It first passes through SAM-imageencoder to produce a feature shape of 64×64×256. Then, a convolutional layer (Conv1) with a kernel size of 3×3 is added to convert the feature shape to 32×32×512. Then, another convolutional layer (Conv2) is added, also using a kernel size of 3×3 to further convert the feature shape to 16×16×1024. Finally, the output features are flattened to 256×1024 to align with the image token shape of CLIP-VIT.

[0031] During the training process, three types of image-text pairs (document image-document text, chart image-json text, landscape-text) are input into the model to train the network structure of the new vocabulary, using autoregressive training. The loss value is calculated using the OPT-125M model, and the gradient is calculated using the backpropagation algorithm, and the embedding linear layer and the new vocabulary network structure are adjusted. Batch_Size is 512, epoch is 3, AdamW optimizer and learning rate of 5e-5 are used as training parameters to train the new vocabulary network structure and update the model parameters.

[0032] It is understandable that by obtaining a diverse dataset of image-text pairs and training them with autoregression, a new vocabulary network structure is generated. This not only expands the vocabulary range of the model, but also enhances its cross-modal understanding ability. The new vocabulary provides richer vocabulary representation, allowing the model to perform better when dealing with complex visual-language tasks. This step lays a solid foundation for subsequent vocabulary updates and fine-tuning, ensuring the efficiency and accuracy of the entire information extraction process.

[0033] like Figure 3 As shown in the first stage, the model receives document data, chart data and natural images, which together have a dimension of 3×1024×1024. The three types of data are sent to the new vocabulary network structure, which converts the data into a 256*1024 vector. The vector is sent to the embedding layer and converted into a 256*4096 vector. The vector is sent to the OPT-125M model to generate a new vocabulary result, and the generated result is returned to the input OPT-125M model in the autoregression.

[0034] Step S20, extracting the network structure of the successfully trained pre-trained model as the new vocabulary network structure, using the new vocabulary network structure to update the original vocabulary network structure of the preset multimodal large model, and determining the updated preset multimodal large model.

[0035] It should be noted that the preset multimodal large model is usually a multimodal model that has been pre-trained on large-scale data, such as CLIP (Contrastive Language–Image Pre-training) based on the Transformer architecture or similar models. These models have been pre-trained on a large amount of image-text pair data and have preliminary cross-modal understanding capabilities.

[0036] The new vocabulary network structure generated by the autoregressive training in step S10 includes richer vocabulary representation and an expanded vocabulary range. After the vocabulary is updated, the vocabulary range of the multimodal large model is expanded, and it can better handle diverse text content.

[0037] Specifically, in order to quickly introduce new visual features, the newly generated visual vocabulary needs to be merged with the original CLIP vocabulary. By processing the two visual vocabularies in parallel and using a separate input embedding layer (a simple linear layer), it is possible to ensure that the concatenated image token channel is aligned with the input of the Large Language Model (LLM). The original CLIP vocabulary is 1024-dimensional, and the linear layer converts the 1024-dimensional input to a 2048-dimensional output. By processing the newly generated visual vocabulary in parallel, the two 2048-dimensional outputs are spliced ​​together to form a 4096-dimensional representation, which is exactly aligned with the input of the LLM (Qwen-7B or Vicuna-7B).

[0038] It can be understood that the updated preset multimodal large model is determined by obtaining the preset multimodal large model and updating its original vocabulary network structure according to the new vocabulary network structure. This step not only expands the vocabulary range of the model, but also provides richer vocabulary representation, so that the model performs better when processing complex visual-language tasks. The updated vocabulary enables the model to better adapt to specific task requirements, thereby improving the accuracy and practicality of the entire information extraction process.

[0039] In a feasible implementation manner, the preset multimodal large model in step S20 includes a multimodal pre-trained model and a large language model, and the original vocabulary network structure of the preset multimodal large model is updated using the new vocabulary network structure, and the step of determining the updated preset multimodal large model includes steps A10 to A50: Step A10, obtaining the original image embedding layer corresponding to the original vocabulary network structure of the multimodal pre-training model.

[0040] It should be noted that we first need to clarify the specific architecture of the multimodal pre-training model currently used. This usually includes the image encoder, the text encoder, and the interaction mechanism between them, and determines the part of the image encoder that is responsible for converting the input image into a high-dimensional vector representation, namely the image embedding layer.

[0041] Among them, the multimodal pre-training model is usually the CLIP model. CLIP is a multimodal pre-training model developed by OpenAI that can learn the alignment relationship between images and text. CLIP-ViT is a variant of the CLIP model, in which ViT (vision transformer) is a visual model based on the Transformer architecture. In CLIP, "vocabulary" usually refers to the word list used to represent parts of text. This word list contains all words or sub-word units that the model can understand and process. CLIP uses a relatively large word list to cover a wide range of language expressions.

[0042] Step A20, creating a new image embedding layer in the multimodal pre-trained model according to the new vocabulary network structure.

[0043] It should be noted that when adding a new image embedding layer to the existing multimodal pre-trained model, the dimension of the newly generated visual vocabulary is 1024, and the output dimension of the new image embedding layer is set to 2048, which is consistent with the original image embedding layer. Create a new linear layer with an input dimension of 1024 and an output dimension of 2048. This new linear layer will serve as a new image embedding layer to process the newly generated visual vocabulary. The linear layer is a commonly used fully connected layer that performs matrix multiplication operations to convert the input tensor into a higher-dimensional output tensor.

[0044] Step A30, splicing the original image embedding layer and the new image embedding layer to determine a parallelized image embedding layer, wherein an output vector of the parallelized image embedding layer is obtained by splicing an output vector of the original image embedding layer and an output vector of the new image embedding layer.

[0045] It is understandable that the output vectors of the original image embedding layer and the new image embedding layer are concatenated to form a higher-dimensional feature representation. This step not only retains the original model's ability to understand image data, but also introduces new visual features, enhancing the model's expressiveness and generalization capabilities. In this way, the model can better understand and process complex multimodal data, providing stronger support for subsequent tasks.

[0046] Step A40, updating the image embedding layer of the multimodal pre-trained model according to the parallelized image embedding layer to obtain an updated multimodal pre-trained model.

[0047] It should be noted that the parallelized image embedding layer is composed of the original image embedding layer and the new image embedding layer in parallel. In fact, the original image embedding layer is not really "deleted", but it is used as part of the parallelization process. The parallelized image embedding layer composed of the original image embedding layer and the new image embedding layer in parallel replaces the original single original image embedding layer.

[0048] It is understandable that the parallelized image embedding layer is integrated into the multimodal pre-trained model to replace the original image embedding layer. In this way, the model is able to utilize both old and new visual features, thereby improving its expressiveness and generalization capabilities. This step not only retains the functionality of the original model, but also enhances the model's ability to understand complex multimodal data. Finally, through testing and verification, it is ensured that the updated model can work properly and achieve the expected results.

[0049] Step A50, determining an updated preset multimodal large model according to the updated multimodal pre-trained model and the large language model.

[0050] This implementation method updates the preset multimodal large model according to the new vocabulary network structure, which not only retains the original multimodal pre-trained model's ability to understand images, but also introduces additional information brought by the newly generated visual vocabulary. The updated preset multimodal large model constructed in this way has stronger expressive power and better generalization performance, and is suitable for more complex multimodal tasks.

[0051] Step S30, obtain a preset visual question and answer dataset and a first information extraction instruction input by the user, use the preset visual question and answer dataset as a training dataset for the updated preset multimodal large model, use the first information extraction instruction as the large model instruction of the updated preset multimodal large model, fine-tune the updated preset multimodal large model, and obtain the preset multimodal large model after fine-tuning.

[0052] It should be noted that visual question answering datasets usually contain images and related questions and answers. For example, VQA (Visual Question Answering) v2.0, VQA-CP v2, etc. The dataset consists of: images, i.e., various types of images, such as document images, chart images, etc., questions, i.e., natural language questions related to images, answers, i.e., answers to corresponding questions, which can be in text form or coordinates pointing to a certain location in the image. The first information extraction instruction is a specific information extraction requirement provided by the user, such as "extracting key data from a table", "identifying trends in a chart", etc.

[0053] During the fine-tuning process, you can choose to freeze the underlying feature extraction layers of the model and only fine-tune the high-level task head and other related layers. After fine-tuning, the model performs better on specific visual question answering tasks and can answer image-related questions more accurately.

[0054] It can be understood that by obtaining the preset visual question-answering dataset and the first information extraction instruction input by the user, the dataset is used as the training data of the updated multimodal large model, and the instruction is used as the guidance information of the model for fine-tuning. This step not only improves the performance of the model on specific tasks, but also enhances the flexibility and accuracy of the model, so that it can better meet the specific needs of users. The fine-tuned model is both efficient and accurate in the information extraction process of complex documents, and can quickly respond to user needs and provide high-quality results.

[0055] In a feasible implementation, in step S30, the preset visual question answering dataset is used as the training dataset of the updated preset multimodal large model, the first information extraction instruction is used as the large model instruction of the updated preset multimodal large model, and the updated preset multimodal large model is fine-tuned to obtain the preset multimodal large model after fine-tuning, including step S31: Step S31, using the preset visual question answering dataset as the model input of the updated multimodal pre-trained model, using the output vector of the updated multimodal pre-trained model as the model input of the large language model, using the first information extraction instruction as the large model instruction of the large language model, and jointly iteratively fine-tuning the updated multimodal pre-trained model and the large language model to obtain the preset multimodal large model after fine-tuning.

[0056] It should be noted that the visual question answering dataset is usually a VQA dataset. The process of joint iterative fine-tuning includes: inputting the image into the updated multimodal pre-trained model to obtain high-dimensional feature representations, passing these feature representations as input to the large language model, calculating the loss based on the output of the large language model and the true label, and updating the parameters of the multimodal pre-trained model and the large language model through the back-propagation algorithm.

[0057] like Figure 4 As shown in the second stage, the multimodal pre-training model receives visual question answering data, and sends the visual question answering data to the original vocabulary network structure and the new vocabulary network structure at the same time, and converts them into 256*1024 vectors respectively. The two image vectors pass through the embedding layer and are converted into 256*2048 vectors respectively. The two image vectors are spliced ​​together to form a 256*2048 vector. The spliced ​​vector is sent to the large language model. At the same time, the first information extraction instruction is sent to the large language model, the marked block (marked with freeze) is frozen, and the part in the unmarked block is trained.

[0058] In a feasible implementation, the step of jointly iteratively fine-tuning the updated multimodal pre-trained model and the large language model in step S31 to obtain a preset multimodal large model after fine-tuning includes steps B10 to B50: Step B10: input the visual question answering data in the preset visual question answering dataset into the updated multimodal pre-trained model to obtain the output vector of the updated multimodal pre-trained model.

[0059] The visual question answering data is input into the original vocabulary network structure and the new vocabulary network structure running in parallel of the updated multimodal pre-trained model to obtain the output vector of the spliced ​​multimodal pre-trained model.

[0060] Specifically, the model uses parallelized image embedding layers (for example, the original image embedding layer and the new image embedding layer), and processes the image through these two embedding layers respectively to generate two feature vectors, which are then concatenated into a higher-dimensional feature vector. The final high-dimensional feature vector contains rich information about the image and text, and can be used as the input of a large language model.

[0061] Step B20, inputting the output vector and the first information extraction instruction into the large language model to obtain the output result of the large language model.

[0062] The output vector and the first information extraction instruction are input into the large language model, and the large language model generates an output result based on these inputs.

[0063] Step B30, obtaining the true result corresponding to the output result from the preset visual question answering dataset, and calculating the loss value according to the difference between the output result and the true result.

[0064] Read the current batch of image-question pairs and their corresponding standard answers from the preset visual question answering dataset, input the model's output results and the true results into the selected loss function, and calculate the loss value.

[0065] In a feasible implementation manner, the step of calculating the loss value according to the difference between the output result and the true result in step B30 includes steps C10 to C30: Step C10, calculating the normalized edit distance and the balanced F score between the output result and the true result.

[0066] It should be noted that the normalized edit distance is a measure of the similarity between two strings, based on the minimum number of single-character editing operations (insertion, deletion, or substitution) required to convert one string into another. The balanced F score is the F1 score, which is used to measure the overlap between the model's predicted answer and the true answer. It is the harmonic mean of precision and recall. Precision is the proportion of correctly predicted positive samples to all predicted positive samples, and recall is the proportion of correctly predicted positive samples to all actual positive samples. Specifically, the edit distance is calculated using a dynamic programming algorithm, and the edit distance is divided by the length of the longer string to obtain the normalized edit distance. The predicted answer and the true answer are segmented, the number of words that appear together is calculated, the precision and recall are calculated according to the formula, and then the F1 score is calculated.

[0067] Step C20, obtaining a preset first weight of the normalized edit distance and a preset second weight of the balanced F-score.

[0068] It should be noted that the preset first weight of the normalized edit distance and the preset second weight of the balanced F-score need to be determined based on experience or experiments.

[0069] Step C30, performing weighted calculation on the normalized edit distance and the F1 score according to the preset first weight and the preset second weight, to obtain a loss value between the output result and the true result.

[0070] The normalized edit distance and F1 score are weighted and summed according to the preset weights to obtain the final loss value. Since the normalized edit distance is as small as possible, and the F1 score is as large as possible, it is necessary to take the complement of the F1 score (i.e. 1-F1 score).

[0071] This implementation calculates the normalized edit distance and F1 score and combines them with preset weights to construct a comprehensive loss function, which can more comprehensively evaluate the performance of the model and guide the optimization of model parameters.

[0072] Step B40, back-propagating the updated multimodal pre-trained model and the model parameters to be fine-tuned of the large language model according to the loss value until the model converges, and then determining the final fine-tuning model parameters.

[0073] Based on the loss value calculated in the forward propagation, the gradient of each learnable parameter is calculated using the automatic derivation mechanism (such as autograd in PyTorch), and the selected optimization algorithm is used to update the model parameters. The weights of the model are updated according to the calculated gradient and learning rate, as well as other possible hyperparameters. The changes in the loss value are checked regularly to determine whether the predetermined convergence conditions have been met.

[0074] In a feasible implementation, the model parameters to be fine-tuned include model parameters of the original image embedding layer, model parameters of the new image embedding layer, and model parameters of the large language model.

[0075] During fine-tuning, most of the parameters of the network structure are frozen, and only the unfrozen parameters are updated to adapt the model to new tasks or data sets. For example, in this application, the two network structures and the concatenated output vectors in the visual encoder of the multimodal pre-trained model are frozen, and only the model parameters of the original image embedding layer, the new image embedding layer, and the large language model are updated. Figure 4 In the process, the marked blocks are partially frozen and the unmarked blocks are partially updated.

[0076] Step B50, determining the preset multimodal large model after fine-tuning according to the final fine-tuning model parameters.

[0077] After completing the update of the multimodal pre-trained model and integrating it with the large language model, this implementation method fine-tunes the entire system through specific task data sets and task instructions. The updated multimodal pre-trained model can extract richer and more representative features, and the large language model can better understand the contextual relationship between images and texts and generate high-quality answers. The joint optimization of the multimodal pre-trained model and the large language model ensures that the two work together to improve the performance of the overall system. The final fine-tuned preset multimodal large model can perform well on specific tasks and provide more accurate and reliable solutions.

[0078] Step S40, receiving the target document snapshot and the second information extraction instruction input by the user, inputting the target document snapshot and the second information extraction instruction into the fine-tuned preset multimodal large model, and obtaining the information extraction result output by the fine-tuned preset multimodal large model.

[0079] It should be noted that the target document snapshot is a document image provided by the user for information extraction, which may be a scanned PDF page, table, chart, etc. The second information extraction instruction is a specific information extraction requirement provided by the user, such as "extract key data from a table", "identify trends in a chart", "extract document titles", etc.

[0080] The preprocessed image is input into the model’s visual encoder, the preprocessed instruction embedding vector is input into the model’s language encoder, and the image features and instruction embedding vector are processed to form a comprehensive input representation.

[0081] This embodiment obtains a preset image-text pair data set, uses the preset image-text pair data set as training data for a pre-trained model, performs autoregressive training on the pre-trained model, and obtains a successfully trained pre-trained model, so that the pre-trained model can generate a new vocabulary based on the input image-text pair data, which helps to expand the vocabulary range of the model and learn richer vocabulary representations, so as to better understand the associations between different types of images and texts; extracts the network structure of the successfully trained pre-trained model as a new vocabulary network structure, uses the new vocabulary network structure to update the original vocabulary network structure of the preset multimodal large model, and determines the updated preset multimodal large model. The updated vocabulary can provide richer and more accurate vocabulary representations, so that the model can process complex visual-language The method can achieve better performance in specific visual question answering tasks; obtain a preset visual question answering data set and a first information extraction instruction input by the user, use the preset visual question answering data set as the training data set of the updated preset multimodal large model, use the first information extraction instruction as the large model instruction of the updated preset multimodal large model, fine-tune the updated preset multimodal large model, and obtain the preset multimodal large model after fine-tuning. By fine-tuning the updated multimodal large model, the model can achieve better performance in specific visual question answering tasks; receive the target document snapshot and the second information extraction instruction input by the user, input the target document snapshot and the second information extraction instruction into the fine-tuned preset multimodal large model, and obtain the information extraction result output by the fine-tuned preset multimodal large model. This embodiment uses a diversified image-text pair data set and autoregressive training to enable the model to better understand the relationship between images and texts, improve cross-modal understanding capabilities, and significantly improve the performance of the model on specific tasks through vocabulary updates and fine-tuning, especially in visual question answering tasks related to documents and charts, so that the model is both efficient and accurate in the information extraction process of complex documents, and can quickly respond to user needs and provide high-quality results.

[0082] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated later. On this basis, before step S10, the document snapshot information extraction method also includes steps S01 to S06: Step S01 , collecting a plurality of original document data including plain text, data formulas and tables from an external data source, and extracting plain text data, data formula data and table data from each original document data.

[0083] Step S02, based on a plurality of predefined templates, the plain text data, the data formula data and the table data are rendered to obtain a rendered first document file, wherein each predefined template corresponds to a different document type.

[0084] Step S03, converting each first document file into a document image, and converting each first document file into a second document file in a preset format, and pairing each document image with the corresponding second document file to generate a document image-text pair data set.

[0085] It should be noted that the external data source is arXiv, which is a preprint server covering academic papers in many fields such as physics, mathematics, and computer science. The original document data is a .tex source file. The preset format is the unified Mathpix Markdown format. The predefined templates can be divided into different types of templates such as academic paper templates, technical report templates, and corporate report templates, and their composition and key content are different. Rendering refers to filling the extracted plain text data into the corresponding position in the template.

[0086] Specifically, we collected a large number of .tex source files from arXiv. We used regular expressions to extract tables, mathematical formulas, and plain text from .tex source files. We collected more than 10 templates to perform batch rendering, and used the pdflatex tool to re-render the extracted content into PDF format based on the template. We converted the text true value of each document page (i.e. the extracted text content) into the Mathpix Markdown style to unify the format. Through this construction process, we obtained 500,000 English pages and 400,000 Chinese pages.

[0087] Step S04, obtaining multiple original charts from an external data source, extracting the chart text true value from each original chart, and converting each chart text true value into JSON (JavaScript Object Notation, JavaScript, object notation) data, and using each JSON data as a chart image text pair data set.

[0088] Specifically, matplotlib and pyecharts were selected as chart rendering tools to build 250,000 matplotlib-style or pyecharts-style charts in both Chinese and English. The true values ​​of the chart text (such as titles, x-axis and y-axis labels) were randomly selected from the natural language processing corpus downloaded from the Internet. The true value of the text of each chart (i.e., the text content in the chart) was converted to JSON format to unify the format of the chart text for subsequent processing and use.

[0089] Step S05, obtaining multiple natural images from an external data source, and randomly assigning predefined description text to each natural image, pairing each natural image with the corresponding predefined description text, and generating a negative natural image-text pair data set.

[0090] Step S06: taking the document image-text pair dataset, the chart image-text pair dataset and the negative natural image-text pair dataset as preset image-text pair datasets.

[0091] Specifically, in order to ensure that the newly introduced visual vocabulary does not generate noise when processing natural images, it is necessary to construct negative natural image-text pairs. The COCO (Common Objects in Context) dataset is a large-scale image recognition, segmentation, and annotation dataset that contains a large number of natural images and corresponding annotation information. This dataset covers a variety of scenes and objects and is suitable for constructing negative sample datasets. 120,000 images were extracted from the COCO dataset, covering different scenes and contents, and a fixed text description was randomly selected for each image. The text part was randomly selected from the following sentences: "This is a natural image"; "This is a natural picture"; "This is a natural photo"; "This is a natural image"; "This is a lens of nature".

[0092] By introducing negative samples, we ensure that the model will not be affected by the newly introduced visual vocabulary when processing natural images, reducing the possibility of misidentification. Negative samples help improve the model's ability to distinguish different types of images and improve its robustness.

[0093] This embodiment generates various types of image-text pair data sets by collecting and processing original document data, chart data, and natural image data from external data sources. These data sets are not only diverse and rich, but also consistent and structured, providing high-quality data support for subsequent model training and application. Through these steps, the system can generate documents and charts that are both professional and practical, and enhance the robustness and generalization ability of the model by introducing negative samples, thereby significantly improving the performance of the model in multimodal information processing tasks.

[0094] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the document snapshot information extraction method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0095] This application also provides a document snapshot information extraction device, please refer to Figure 5 , the document snapshot information extraction device comprises: The training module 10 is used to obtain a preset image-text pair data set, use the preset image-text pair data set as training data for a pre-trained model, perform autoregressive training on the pre-trained model, and obtain the successfully trained pre-trained model, wherein the preset image-text pair data set includes a document image-text pair data set, a chart image-text pair data set, and a negative natural image-text pair data set, and the pre-trained model is used to generate a new vocabulary according to the preset image-text pair data set; An updating module 20, configured to extract the network structure of the successfully trained pre-trained model as a new vocabulary network structure, update the original vocabulary network structure of the preset multimodal large model using the new vocabulary network structure, and determine the updated preset multimodal large model; A fine-tuning module 30 is used to obtain a preset visual question answering dataset and a first information extraction instruction input by a user, use the preset visual question answering dataset as a training dataset of the updated preset multimodal large model, use the first information extraction instruction as a large model instruction of the updated preset multimodal large model, and fine-tune the updated preset multimodal large model to obtain the preset multimodal large model after fine-tuning; The output module 40 is used to receive the target document snapshot and the second information extraction instruction input by the user, input the target document snapshot and the second information extraction instruction into the fine-tuned preset multimodal large model, and obtain the information extraction result output by the fine-tuned preset multimodal large model.

[0096] The document snapshot information extraction device provided by the present application adopts the document snapshot information extraction method in the above embodiment, which can solve the technical problem of how to improve the accuracy of information extraction in complex document snapshots. Compared with the prior art, the beneficial effects of the document snapshot information extraction device provided by the present application are the same as the beneficial effects of the document snapshot information extraction method provided by the above embodiment, and other technical features in the document snapshot information extraction device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0097] The present application provides a document snapshot information extraction device, which includes: at least one processor; and a memory that is communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the document snapshot information extraction method in the above-mentioned embodiment one.

[0098] Reference below Figure 6, which shows a schematic diagram of the structure of a document snapshot information extraction device suitable for implementing the embodiment of the present application. The document snapshot information extraction device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The document snapshot information extraction device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0099] like Figure 6 As shown, the document snapshot information extraction device may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the document snapshot information extraction device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the document snapshot information extraction device to communicate wirelessly or wired with other devices to exchange data. Although the document snapshot information extraction device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0100] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0101] The document snapshot information extraction device provided by the present application adopts the document snapshot information extraction method in the above embodiment, which can solve the technical problem of how to improve the accuracy of information extraction in complex document snapshots. Compared with the prior art, the beneficial effects of the document snapshot information extraction device provided by the present application are the same as the beneficial effects of the document snapshot information extraction method provided by the above embodiment, and other technical features in the document snapshot information extraction device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0102] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0103] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0104] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the document snapshot information extraction method in the above-mentioned embodiment.

[0105] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned document snapshot information extraction method, and can solve the technical problem of how to improve the accuracy of information extraction in complex document snapshots. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the document snapshot information extraction method provided in the above-mentioned embodiment, and will not be repeated here.

[0106] The present application also provides a computer program product, including a computer program, which implements the steps of the document snapshot information extraction method as described above when executed by a processor.

[0107] The computer program product provided by the present application can solve the technical problem of how to improve the accuracy of information extraction in complex document snapshots. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the document snapshot information extraction method provided by the above embodiment, which will not be repeated here.

[0108] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A document snapshot information extraction method, characterized in that: The document snapshot information extraction method comprises: Obtain a preset image-text pair data set, use the preset image-text pair data set as training data for a pre-trained model, perform autoregressive training on the pre-trained model, and obtain the successfully trained pre-trained model, wherein the preset image-text pair data set includes a document image-text pair data set, a chart image-text pair data set, and a negative natural image-text pair data set, and the pre-trained model is used to generate a new vocabulary according to the preset image-text pair data set; Extracting the network structure of the successfully trained pre-trained model as a new vocabulary network structure, using the new vocabulary network structure to update the original vocabulary network structure of the preset multimodal large model, and determining the updated preset multimodal large model; Obtaining a preset visual question answering dataset and a first information extraction instruction input by a user, using the preset visual question answering dataset as a training dataset of the updated preset multimodal large model, using the first information extraction instruction as a large model instruction of the updated preset multimodal large model, and fine-tuning the updated preset multimodal large model to obtain the preset multimodal large model after fine-tuning; Receive a target document snapshot and a second information extraction instruction input by a user, input the target document snapshot and the second information extraction instruction into the fine-tuned preset multimodal large model, and obtain the information extraction result output by the fine-tuned preset multimodal large model.

2. The document snapshot information extraction method according to claim 1, characterized in that: Before the step of obtaining the preset image-text pair data set, the method includes: Collecting a plurality of original document data including plain text, data formulas and tables from an external data source, and extracting plain text data, data formula data and table data from each of the original document data; Based on a plurality of predefined templates, the plain text data, the data formula data and the table data are rendered to obtain a rendered first document file, wherein each of the predefined templates corresponds to a different document type; Converting each of the first document files into a document image, and converting each of the first document files into a second document file in text format, and pairing each of the document images with a corresponding second document file to generate a document image-text pair data set; Acquire multiple original charts from the external data source, extract the chart text true value from each original chart, convert each chart text true value into JSON data, and use each JSON data as a chart image text pair data set; Acquire multiple natural images from the external data source, randomly assign predefined description text to each of the natural images, pair each of the natural images with the corresponding predefined description text, and generate a negative natural image-text pair dataset; The document image-text pair dataset, the chart image-text pair dataset and the negative natural image-text pair dataset are used as the preset image-text pair datasets.

3. The document snapshot information extraction method according to claim 1, characterized in that: The preset multimodal large model includes a multimodal pre-trained model and a large language model, and the step of updating the original vocabulary network structure of the preset multimodal large model by using the new vocabulary network structure to determine the updated preset multimodal large model includes: Obtaining an original image embedding layer corresponding to the original vocabulary network structure of the multimodal pre-trained model; Creating a new image embedding layer in the multimodal pre-trained model according to the new vocabulary network structure; The original image embedding layer and the new image embedding layer are spliced ​​to determine a parallelized image embedding layer, wherein an output vector of the parallelized image embedding layer is obtained by splicing an output vector of the original image embedding layer and an output vector of the new image embedding layer; Updating the image embedding layer of the multimodal pre-trained model according to the parallelized image embedding layer to obtain an updated multimodal pre-trained model; The updated preset multimodal large model is determined according to the updated multimodal pre-trained model and the large language model.

4. The document snapshot information extraction method according to claim 3, characterized in that: The step of using the preset visual question answering dataset as a training dataset of the updated preset multimodal large model, using the first information extraction instruction as a large model instruction of the updated preset multimodal large model, and fine-tuning the updated preset multimodal large model to obtain the preset multimodal large model after fine-tuning includes: The preset visual question answering dataset is used as the model input of the updated multimodal pre-trained model, the output vector of the updated multimodal pre-trained model is used as the model input of the large language model, the first information extraction instruction is used as the large model instruction of the large language model, the updated multimodal pre-trained model and the large language model are jointly iteratively fine-tuned, and the preset multimodal large model after fine-tuning is obtained.

5. The document snapshot information extraction method according to claim 4, characterized in that: The step of jointly iteratively fine-tuning the updated multimodal pre-trained model and the large language model to obtain the preset multimodal large model after fine-tuning includes: Inputting the visual question answering data in the preset visual question answering dataset into the updated multimodal pre-trained model to obtain an output vector of the updated multimodal pre-trained model; Inputting the output vector and the first information extraction instruction into the large language model to obtain an output result of the large language model; Obtaining a true result corresponding to the output result from the preset visual question answering dataset, and calculating a loss value according to a difference between the output result and the true result; Back-propagating the loss value to update the updated multimodal pre-trained model and the model parameters to be fine-tuned of the large language model until the model converges, and then determining the final fine-tuning model parameters; The preset multimodal large model after fine-tuning is determined according to the final fine-tuning model parameters.

6. The document snapshot information extraction method according to claim 5, characterized in that: The step of calculating the loss value according to the difference between the output result and the true result comprises: Calculating a normalized edit distance and a balanced F score between the output result and the true result; Obtaining a preset first weight of the normalized edit distance and a preset second weight of the balanced F-score; The normalized edit distance and the balanced F-score are weightedly calculated according to the preset first weight and the preset second weight to obtain a loss value between the output result and the true result.

7. The document snapshot information extraction method according to claim 5, characterized in that: The model parameters to be fine-tuned include model parameters of the original image embedding layer, model parameters of the new image embedding layer, and model parameters of the large language model.

8. A document snapshot information extraction device, characterized in that: The document snapshot information extraction device comprises: A training module, used to obtain a preset image-text pair data set, use the preset image-text pair data set as training data for a pre-trained model, perform autoregressive training on the pre-trained model, and obtain the successfully trained pre-trained model, wherein the preset image-text pair data set includes a document image-text pair data set, a chart image-text pair data set, and a negative natural image-text pair data set, and the pre-trained model is used to generate a new vocabulary according to the preset image-text pair data set; An updating module, used for extracting the network structure of the successfully trained pre-trained model as a new vocabulary network structure, using the new vocabulary network structure to update the original vocabulary network structure of the preset multimodal large model, and determining the updated preset multimodal large model; A fine-tuning module is used to obtain a preset visual question answering dataset and a first information extraction instruction input by a user, use the preset visual question answering dataset as a training dataset of the updated preset multimodal large model, use the first information extraction instruction as a large model instruction of the updated preset multimodal large model, and fine-tune the updated preset multimodal large model to obtain the preset multimodal large model after fine-tuning; An output module is used to receive a target document snapshot and a second information extraction instruction input by a user, input the target document snapshot and the second information extraction instruction into the fine-tuned preset multimodal large model, and obtain the information extraction result output by the fine-tuned preset multimodal large model.

9. A document snapshot information extraction device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the document snapshot information extraction method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the document snapshot information extraction method according to any one of claims 1 to 7 are implemented.

Images