Multi-modal document information extraction method and device based on large model and intelligent agent

Abstract

The invention provides an information extraction method and device for a multi-modal document based on a large model and an intelligent agent, and relates to the technical fields of information processing, artificial intelligence and the like. According to the specific implementation scheme, a page set of a multi-modal document is obtained, the multi-modal document comprises multiple pages of content, and the page set comprises pages of all the pages of content; based on the page set of the multi-modal document, a pre-trained multi-modal content extraction model is adopted to extract the text content of the multi-modal document, and the multi-modal content extraction model is trained based on a large language model; the multi-modal content extraction model can extract the text content of each modal in the multi-modal document; based on the text content of the multi-modal document, a pre-trained information extraction model is adopted for information extraction, and the information extraction model is also formed by training based on a large language model. According to the technology disclosed by the invention, the information extraction accuracy and the information extraction efficiency can be effectively improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, specifically to technical fields such as information processing and artificial intelligence, and more particularly to a method, device and intelligent agent for extracting information from multimodal documents. Background Art

[0002] Documents are a very common way to record information and may include various formats such as Word and Portable Document Format (PDF).

[0003] Documents can be used in various scenarios. For example, a lawyer's letter can be a PDF document. Analysis reports of various industries can be Word or PDF documents, and so on. Documents can include information in at least two of the various modalities, such as text, pictures, web pages, tables, charts, and handwritten signatures, and the content is very rich. Therefore, the documents here can also be called multimodal documents. Summary of the invention

[0004] The present invention provides a method, device and intelligent agent for extracting information from multimodal documents based on a large model.

[0005] According to one aspect of the present disclosure, a method for extracting information from a multimodal document based on a large model is provided, comprising the following steps:

[0006] Acquire a page set of a multimodal document, the multimodal document including multiple pages of content, the page set including pages of each page of content;

[0007] Based on the page set of the multimodal document, a pre-trained multimodal content extraction model is used to extract text content of the multimodal document, wherein the multimodal content extraction model is trained based on a large language model; the multimodal content extraction model can extract text content of each modality in the multimodal document;

[0008] Based on the text content of the multimodal document, information extraction is performed using a pre-trained information extraction model, which is also trained based on a large language model.

[0009] According to another aspect of the present disclosure, there is provided an information extraction device for a multimodal document based on a large model, comprising:

[0010] A page acquisition module, used to acquire a page set of a multimodal document, wherein the multimodal document includes multiple pages of content, and the page set includes pages of each page of content;

[0011] A document content extraction module, configured to extract text content of the multimodal document based on a page set of the multimodal document using a pre-trained multimodal content extraction model, wherein the multimodal content extraction model is trained based on a large language model; the multimodal content extraction model is capable of extracting text content of each modality in the multimodal document;

[0012] The information extraction module is used to extract information based on the text content of the multimodal document using a pre-trained information extraction model, wherein the information extraction model is also trained based on the large language model.

[0013] According to another aspect of the present disclosure, there is provided an intelligent agent, comprising:

[0014] An input module, used for receiving an input multimodal document;

[0015] A processing module, configured to determine an information extraction task based on the multimodal document received by the input module, determine a multimodal content extraction model and an information extraction model based on the information extraction task, and obtain an information extraction result by calling the multimodal content extraction model and the information extraction model to execute the method of the above aspects and any possible implementation manner;

[0016] An output module is used to output the information extraction result obtained by the processing module.

[0017] According to another aspect of the present disclosure, there is provided an electronic device, including:

[0018] at least one processor; and

[0019] a memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any possible implementation manner and the aspects described above.

[0021] According to yet another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method of the above-mentioned aspects and any possible implementation manner.

[0022] According to yet another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements the above-mentioned aspects and the method of any possible implementation manner.

[0023] According to the technology disclosed in the present invention, the accuracy and efficiency of information extraction in multimodal documents can be effectively improved.

[0024] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0026] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0027] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0028] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;

[0029] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0030] Figure 5 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0031] Figure 6 is a schematic diagram according to a sixth embodiment of the present disclosure;

[0032] Figure 7 is a schematic diagram according to a seventh embodiment of the present disclosure;

[0033] Figure 8 is a schematic diagram according to an eighth embodiment of the present disclosure;

[0034] Fig. 9 is a schematic diagram according to a ninth embodiment of the present disclosure;

[0035] Fig.10 is a schematic diagram according to a tenth embodiment of the present disclosure;

[0036] Fig.11 is a schematic diagram according to an eleventh embodiment of the present disclosure;

[0037] Fig.12 is a schematic diagram according to a twelfth embodiment of the present disclosure;

[0038] Fig.13 The block diagram is a block diagram of an electronic device for implementing the method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0039] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0040] Obviously, the described embodiments are only part of the embodiments of the present disclosure, but not all of them. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present disclosure.

[0041] It should be noted that the terminal devices involved in the embodiments of the present disclosure may include but are not limited to mobile phones, personal digital assistants (PDAs), wireless handheld devices, tablet computers and other smart devices; display devices may include but are not limited to personal computers, televisions and other devices with display functions.

[0042] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0043] Figure 1 is a schematic diagram according to the first embodiment of the present disclosure; Figure 1 As shown, this embodiment provides a method for extracting information from multimodal documents based on a large model, which may specifically include the following steps:

[0044] S101, obtaining a page set of a multimodal document, the multimodal document including multiple pages of content, and the page set including pages of each page of content;

[0045] S102, based on the page set of the multimodal document, using a pre-trained multimodal content extraction model to extract text content of the multimodal document;

[0046] The execution subject of the information extraction method of multimodal documents based on a large model of this embodiment can be a multimodal document information extraction device based on a large model. The device can be an electronic entity, or can also be a software integrated application. When used, the device can directly extract the information to be extracted from the multimodal document.

[0047] The multimodal content extraction model in this embodiment is trained based on a large language model (LLM); the multimodal content extraction model can extract the text content of each modality in a multimodal document;

[0048] LLM is a deep learning model trained using a large amount of text data. It can understand the meaning of language text and generate natural language text. It has very powerful functions.

[0049] S103, extracting information based on the text content of the multimodal document using a pre-trained information extraction model;

[0050] The information extraction model of this embodiment is also trained based on LLM.

[0051] The multimodal document of this embodiment is a long document including multiple pages of content, and the long document may include information of multiple modes. Specifically, each page may include information of at least one mode.

[0052] The multimodal document of this embodiment may be a document in a word format or a document in a PDF format.

[0053] It should be noted that for multimodal documents of different formats, the formats of pages in a page set may be different. For example, for a PDF document, each page may be in a picture format, and a page set may be a picture set. For a document in word format, each page may be in the format of a multimodal document, or each page may be converted into a picture format.

[0054] In this embodiment, the multimodal content extraction model can specifically extract the text content included in each modality of each page in the page set of the multimodal document to obtain the text content of the multimodal document. In other words, if the page includes text information in a non-text modality, it can also be extracted, which can effectively ensure that all text information in text format in the multimodal document can be extracted, and effectively ensure that the text content included in the text content of the multimodal document is very comprehensive and rich.

[0055] In the prior art, when it is necessary to extract information from a multimodal document, a parser is used to parse the information of the text modality in the multimodal document and then extract the information; however, the parser is unable to parse the information of other modalities such as pictures, and thus is unable to extract the text information in the non-text modality, resulting in the extracted information being incomplete and inaccurate.

[0056] The information extraction method for multimodal documents based on a large model in this embodiment uses a multimodal content extraction model to extract the text content of each modality in the multimodal document, and further extracts the required information from the text content of the multimodal document. Compared with the prior art, since the extracted text content of the multimodal document is very rich and comprehensive, it can effectively ensure the comprehensiveness and accuracy of information extraction when using the information extraction model to extract information from the text content of the multimodal document, thereby effectively improving the information extraction efficiency of the multimodal document.

[0057] Figure 2 is a schematic diagram according to the second embodiment of the present disclosure; Figure 2 As shown, the information extraction method of multimodal documents based on a large model in this embodiment is as follows: Figure 1 Based on the technical solutions of the embodiments shown in the figure, the technical solutions of the present disclosure are further described in more detail. Figure 2 As shown, the information extraction method of multimodal documents based on a large model in this embodiment may specifically include the following steps:

[0058] S201, obtaining a page set of a multimodal document, the multimodal document including multiple pages of content, and the page set including pages of each page of content;

[0059] For example, the modalities in the multimodal document of this embodiment may include: at least two modalities of text, picture, web page, table, chart, and handwritten signature. At least one modality may be included in each page.

[0060] S202: for each page in the page set of the multimodal document, extract text information of each modality in the page using a multimodal content extraction model;

[0061] For example, in a specific implementation, for each page in a page set of a multimodal document, the regional blocks of each modality included in the page can be obtained first. That is to say, when a page includes information of more than two modalities, the page should be divided into regions based on the modality of the information to obtain regional blocks of each modality. When the entire page has only one modality, the entire page can be considered as a large regional block. Then, based on the regional blocks of each modality included in the page, a multimodal content extraction model is used to extract text information of each modality in the page. In this way, a multimodal content extraction model can be used to extract text information from the regional blocks of each modality in a targeted manner, thereby effectively ensuring the comprehensiveness and accuracy of information extraction.

[0062] The base model of the multimodal content extraction model of this embodiment can be the large language model Llama3 released by Meta. That is, the large language model Llama3 is trained, and the trained large language model Llama3 is used as the multimodal content extraction model of this embodiment.

[0063] Further optionally, in one embodiment of the present disclosure, for each page in a page set of a multimodal document, obtaining the regional blocks of each modality included in the page may specifically include: for each page in the page set of the multimodal document, using a pre-trained modality recognition model to identify the regional blocks of each modality included in the page.

[0064] The modal recognition model of this embodiment is a pre-trained neural network model. When in use, a page is input into the modal recognition model, and the modal recognition model can identify each modality and corresponding region block included in the page. For example, the region block can be identified by position information, and the position of the region block in the page can be identified. Alternatively, the multimodal content recognition model of this embodiment can also be directly used to identify the region blocks of each modality included in the page.

[0065] Or optionally, for each page in the page set of the multimodal document, the region blocks of each modality included in the page may also be identified based on a pre-configured modality identification rule.

[0066] For example, the pre-configured modal recognition rules can set the text direction, text format, character spacing, and line spacing in the regional blocks of the same modality to be consistent; at the same time, for the picture modality, the pre-configured modal recognition rules also include background color and picture-specific information recognition rules. For the handwritten signature modality, the format of its text does not belong to any preset format and can be considered to belong to the handwritten signature modality. For the table modality, the pre-configured modal recognition rules also include the structural information of the table, and then it can be determined that it is a table. For web pages, charts, etc., the pre-configured modal recognition rules also need to set corresponding recognition rules respectively, and no further examples will be given here.

[0067] The text direction may include a horizontal arrangement direction or a vertical arrangement direction. The text format may include at least one of a font, a size, a color, and whether it is bold, whether it is marked with a line, etc. The character spacing refers to the distance between characters. The line spacing refers to the distance between lines. The consistency of the above conditions in this embodiment may refer to being completely identical, or a ratio threshold may be set based on experience or actual needs. If the similarity rate of the text format in a certain modal area block is greater than the ratio threshold, it is considered to be consistent.

[0068] For example, when dividing the area blocks in the page based on the modal recognition rules, the page can be analyzed from top to bottom and from left to right, one line at a time. Based on the modal recognition rules, if the conditions are met, this line is the same modal area block. If the left and right sides of the same line are inconsistent, they are divided into two different modal area blocks. By analyzing the entire page according to the above modal recognition rules, the page can be clustered according to the modality and the area blocks of each modality in the page can be obtained.

[0069] In this embodiment, the text information of each mode in the extracted page can be in markdown format; correspondingly, the text content of the spliced ​​page and the text content of the multimodal document are both in markdown format.

[0070] For example, for the region blocks of each modality in the page, the region blocks of the modality are input into the multimodal content extraction model, and a prompt word (Prompt) is input at the same time. For example, the Prompt can be "extract the text information in the region block and output it in markdown format". The multimodal content extraction model can extract text information from the region blocks of the modality based on the input region blocks and Prompt of the modality, and output the extracted text information in markdown format. Referring to the description of the above embodiment, the region blocks of each modality can be text, tables, charts, pictures, or handwriting, etc.

[0071] S203, splicing the text information of each mode in the page to obtain the text content of the page;

[0072] S204, splicing text contents corresponding to pages of content in the multimodal document to obtain text contents of the multimodal document;

[0073] Specifically, steps S202-S204 can be completed in a multimodal content extraction model. Steps S203 and S204 can be implemented by setting a splicing unit in the multimodal content extraction model to achieve the splicing of text content of different modes in the same page and the splicing of text content of different pages. For the multimodal content extraction model, it can be an end-to-end model, the input is each page in the page set of the multimodal document, and the output is the text content of the multimodal document.

[0074] Optionally, steps S203-S204 may also be implemented outside the multimodal content extraction model to ensure the implementation of the functions.

[0075] Steps S202-S204 are as above Figure 1 A specific implementation of step S102 of the illustrated embodiment. In this way, the text content of the multimodal document can be fully and accurately acquired.

[0076] S205, configuration information extraction specification;

[0077] S206: Extract information using a pre-trained information extraction model according to the text content of the multimodal document and the configured information extraction specification.

[0078] For example, in this embodiment, the information extraction specification may include an identifier of the field to be extracted, and may also include at least one of the description information of the field and an example of the value of the field; the description information of the field includes explanation information of the field and / or the location of the field.

[0079] The information extraction model can extract the value of the field corresponding to the field identifier from the text content of the multimodal document according to at least one of the description information of the field and the example of the field value in the information extraction specification to realize information extraction.

[0080] For example, lawyer's letters generally exist in the form of PDF, and lawyer's letters generally sue various infringements, such as: corporate reputation rights, personal reputation rights, portrait rights, personal information rights, privacy rights, etc. As a legal staff, you need to extract some core information from this long PDF document, such as: the name, unit, email address and other information related to the defendant and the complainant, as well as the name, phone number, email address and other information related to the agent, express number, work name, infringement link, and complaint content. The content of the lawyer's letter is presented in a variety of forms, including text, pictures, web pages, tables, charts, and handwritten signatures. At the same time, many contents are relatively vague. Using the above method of this embodiment, for each modal information in the lawyer's letter in PDF format, a multimodal content extraction model can be used to extract content, and the text content of each modality can be obtained, which can effectively ensure the comprehensiveness and accuracy of the text content obtained in the lawyer's letter.

[0081] Furthermore, by configuring information extraction specifications based on the core information to be extracted and adopting an information extraction model, the specific content of the core information to be extracted can be accurately and effectively extracted, which can effectively improve the efficiency of information extraction.

[0082] It should be noted that when the multimodal document is long, the text content of the MarkDown format multimodal document obtained after content extraction can be a very long text, such as 10 to 100 pages. Due to the long text, it is necessary to use LLM that supports 128K context length to build an information extraction model that can handle ultra-long text information.

[0083] When extracting specific information, the text content of the multimodal document can be input into the information extraction model, and a prompt can be constructed based on the information extraction specification and input into the information extraction model. For example, in this embodiment, the prompt in a certain scenario can be expressed as: You are a professional complaint information extraction assistant, who can strictly follow the requirements to extract the value of the "specified field" in the information extraction specification from the provided long text, and return the extracted results in JSON format.

[0084] The information extraction method of multimodal documents based on a large model of this embodiment adopts a multimodal content extraction model to comprehensively and accurately extract the text content of the multimodal document; the information extraction model and the configured information extraction specification are adopted to accurately extract the values ​​of the required fields from the text content of the multimodal document. The technical solution of this embodiment can be applied to extract information from multimodal documents of various formats, and can effectively ensure the comprehensiveness and accuracy of the information extraction.

[0085] Since the annotation cost of extracting information from very long texts is extremely high and almost impossible to do manually, in this embodiment, a knowledge distillation framework can be built, which mainly uses a strong model to automatically synthesize training data, and then fine-tunes the small model used as the information extraction model based on the synthesized training data. For example, the strong model in this embodiment can use the GPT4 or EB4 model, and the small model can use the Llama3-8B model.

[0086] Specifically, the training method of the information extraction model in the above embodiment may include the following steps:

[0087] (a1) Constructing a super-long text dataset, information extraction specifications and prompts corresponding to each super-long text;

[0088] The super-long text data set may include multiple super-long texts with long lengths. The information extraction specifications corresponding to each super-long text are used to standardize the fields to be extracted in the Prompt; the Prompt may also include: task definition, for example, it may be "You are a super-long text information extraction tool. Given a super-long text, you need to extract the value of the specified field"; the specified field is the information of the field specified in the information extraction specification. Furthermore, the Prompt may also include some requirements, such as not extracting field values ​​that do not exist in the text; the output must be in JSON format, as well as examples of the output information format, etc. The super-long text in this embodiment refers to a relatively long text.

[0089] (b1) For each super-long text, at least two strong models are used to extract information from the super-long text;

[0090] Specifically, the super-long text and the constructed prompt are input into each strong model to extract the required information from the super-long text. For example, taking the multimodal document as a lawyer's letter, the constructed prompt can be exemplified as follows:

[0091] You are a professional complaint information extraction assistant who can strictly follow the requirements to extract the value of the "specified field" from the provided long text and return the extracted results in JSON format.

[0092] Required fields:

[0093] 1. File name: The file name provided by the complainant, usually displayed in the center of the image, such as: lawyer's letter; complaint letter; notification letter; warning letter; early warning letter; letter; statement; notice; announcement, etc.

[0094] 2. File number: The number of the complaint letter, usually displayed on the right side of the image, for example: XX Lawyer’s Letter

[2023] No. 0411.

[0095] 3. Respondent: generally refers to the company name or its affiliated companies, such as: M Company, N Company, etc.

[0096] 4. Reasons for complaint: general corporate reputation rights, personal reputation rights, portrait rights, personal information rights, privacy rights, trademark rights, copyrights, false information, counterfeit websites, trade secrets, counter-notifications, unfair competition, and others.

[0097] 5. Complaint content: The specific request content of the complainant described in a natural paragraph, for example: suspicious persons registered an account in your company's YY company forum and posted messages that infringed on the famous music circle, etc., and immediately deleted the post, blocked it and disconnected the link.

[0098] 6. Works involved: Names of the works involved, such as Work Name A, Work Name B, etc.

[0099] 7. Infringing link: The infringing link address associated with the complaint case.

[0100] 8. Complainant: Generally an institutional entity or a natural person, such as company C, natural person D, etc.

[0101] 9. Complainant-Contact Person: The contact person of the complainant. If the complainant is the name of a company or organization, this is generally the legal person of the company or organization.

[0102] 10. Complainant – Telephone: Complainant’s telephone or mobile number.

[0103] 11. Complainant-Email: The email account of the complainant.

[0104] 12. Agent: The agent who represents the complainant in the case, usually a lawyer.

[0105] 13.Agent-Unit: The name of the unit where the agent works, usually a law firm.

[0106] 14.Agent-Phone: Agent’s telephone number or mobile phone number.

[0107] 15.Agent-Email: Agent’s email account.

[0108] 16.Agent-Fax: Agent’s fax number.

[0109] 17.Agent-Unit Address: the address of the agent’s unit.

[0110] 18.Agent-Unit Postal Code: Agent’s unit postal code.

[0111] 19. Date of complaint: The time when the complaint occurred, usually located below the law firm’s seal. For example, if the date mentioned is “October 16, 2023”, it must be converted to “2023-10-16”.

[0112] 20. Express delivery number: It is located below the barcode on the express delivery order. If it is courier company 1, it may be a number starting with ABC, for example, it may be ABCXXX...XX, where each X can be any number between 0-9; if it is courier company 2, it may be a number starting with DE, for example, it may be DEYYY...YYY, where each Y can be any number between 0-9.

[0113] Require:

[0114] 1. You only need to give the values ​​of the fields contained in the text. For the fields not mentioned in the text, no results need to be given.

[0115] 2. If there are multiple values ​​for a field, all of them need to be given.

[0116] 3. The result must be output in JSON format. The following is an example answer:

[0117]

[0118] Please use the intelligence of your big model to give the value that can be extracted from the field. There is no need to output any thinking process. The following is the input of the super long text:

[0119] {Contexts}

[0120] The parameter magnitudes and capabilities of the at least two strong models and the information extraction model in this embodiment are basically consistent, but the objective evaluation capabilities are slightly different. The capabilities of at least two strong models are stronger than the information extraction model to be trained in this embodiment, so they are called strong models. For example, in this embodiment, the first strong model of the at least two strong models can adopt models such as GPT4 or EB4, and the second strong model of the at least two strong models is weaker than the first strong model, and models such as EB3.5 or Llama 70B can be adopted. Furthermore, if the at least two strong models also include other strong models, it can be implemented using a model different from the first strong model and the second strong model.

[0121] (c1) for each super-long text, compare whether the answers of at least two strong models for information extraction of the super-long text are consistent; if they are consistent, execute step (d1); otherwise, if they are inconsistent, execute step (e1);

[0122] (d1) Obtain the super-long text, the corresponding prompt and the corresponding extracted answer as training data; execute step (f1);

[0123] (e1) Obtain the manually annotated correct answer; use the super-long text, the corresponding prompt and the manually annotated correct answer as training data; and execute step (f1);

[0124] (f1) Use the training data to train the information extraction model for the SFT task.

[0125] It should be noted that if the answers of at least two strong models to the information extraction of the super-long text are inconsistent, the answers of at least two strong models to the information extraction of the super-long text are displayed to the user. If the user determines that the information extraction answer extracted by one of the strong models is correct, the answer is directly marked as the correct answer. If the user determines that the information extraction answer extracted by each strong model is incorrect, the following step (g1) can be executed at this time;

[0126] (g1) Optimize prompt and / or overlong text; return to step (b1) and continue processing;

[0127] For example, you can optimize the description information of the fields specified in the prompt to more clearly limit the relevant information of the fields to be extracted. Or you can optimize the description of the text in the long text to make it easier to extract the value of the field to be extracted. Or you can optimize both at the same time.

[0128] The technical solution of this embodiment uses at least two strong models for knowledge distillation to automatically generate easy sample training data and difficult sample training data, which can minimize the cost of manual annotation and improve the efficiency of generating training samples. Then, the generated easy sample training data and difficult sample training data are used to train the information extraction model. Since the accuracy of the used training data is very high, the accuracy of the trained information extraction model can be effectively improved.

[0129] Figure 3 is a schematic diagram according to the third embodiment of the present disclosure; Figure 3 As shown, this embodiment provides a training method for a multimodal content extraction model based on a large model, including the following steps:

[0130] S301, collecting pages of each page content of each multimodal document in a plurality of multimodal documents, and generating a page library;

[0131] The execution subject of the training method of the multimodal content extraction model based on a large model in this embodiment can be a training device for the multimodal content extraction model based on a large model, and the device can be an electronic entity or a software integrated application. When in use, the device can Figure 1 or Figure 2 The multimodal content extraction model used in the illustrated embodiment is trained.

[0132] Refer to the above Figure 1 and Figure 2 The embodiment shown in the drawings records that the multiple multimodal documents of this embodiment may include documents in PDF format, documents in Word format, and documents in other formats, which are not limited here. For multimodal documents in any format, each page of each multimodal document is collected separately to form a page library.

[0133] S302: For each page in the page library, use a pre-trained multimodal large model to extract the text content of the page;

[0134] For example, the multimodal large model of the present embodiment is a pre-trained model, which may be a Generative Pre-Trained (GPT) 4O model, where "O" stands for Omni, which means omnipotent.

[0135] In this embodiment, the GPT4O model is used as a multimodal large model. The model is a mature model with very strong performance. Based on this model, text content of various modes in each page in the page library can be extracted.

[0136] S303, constructing first training data suitable for supervised fine tuning (Supervised Fine Tuning; SFT) task training and second training data suitable for direct preference optimization (Direct Preference Optimization; DPO) task training based on the text content of each page;

[0137] S304, performing SFT task training and DPO task training on the multimodal content extraction model based on the first training data and the second training data respectively;

[0138] The multimodal content extraction model of this embodiment has fewer parameters than the multimodal large model. For example, it can be considered that the parameters of the multimodal large model can reach hundreds of billions, and the model is a heavyweight model that occupies a large space. However, the parameters of the multimodal content extraction model only reach several billion, and the model is a lightweight model that occupies a small space.

[0139] The multimodal content extraction model is trained based on LLM. For example, the LLM in this embodiment can adopt the large language model Llama3 released by Meta.

[0140] In this embodiment, the number of the constructed first training data and the second training data is not limited, and can include multiple pieces. The more the number of constructed first training data and the second training data is, the more accurate the trained multimodal content extraction model is.

[0141] The technical solution of this embodiment is used to Figure 1 and Figure 2 The multimodal content extraction model in is trained so that the multimodal content extraction model can accurately and effectively extract text content in the multimodal document when used.

[0142] The training method of the large model-based multimodal content extraction model of this embodiment adopts the above-mentioned technical solution to construct the first training data for the SFT task and the second training data for the DPO task, and then uses the first training data and the second training data to respectively train the multimodal content extraction model for the SFT task and the DPO task. This can accurately and effectively train the multimodal content extraction model, thereby effectively improving the accuracy of the trained multimodal content extraction model.

[0143] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure; Figure 4 As shown, the training method of the multimodal content extraction model based on the large model of this embodiment is as follows: Figure 3 Based on the basic scheme of the embodiment shown, the technical scheme of the present disclosure is further described in more detail. Figure 4As shown, the training method of the multimodal content extraction model based on the large model of this embodiment may include the following steps:

[0144] S401, collecting pages of contents of each page of each multimodal document among a plurality of multimodal documents, and generating a page library;

[0145] S402: For each page in the page library, use a pre-trained multimodal large model to extract the text content of the page;

[0146] Steps S401-S402 may specifically adopt the above Figure 3 The implementation method of steps S301 - S302 of the illustrated embodiment will not be described in detail here.

[0147] S403, performing format check on the text content of each page;

[0148] For example, the format check of this embodiment may include checking of syntax, semantics, structured information, etc. The structured information check may refer to checking of structured information included in a table, etc. For example, a pre-trained format check model may be used to implement the format check.

[0149] S404, in response to the text content of the page having a format check error, obtaining manually annotated correct text content;

[0150] Specifically, for a certain page, if the format check has an error, the text content of the current page is displayed; the staff manually checks the error based on the displayed text content of the current page, and inputs the manually annotated correct text content. At this time, the corresponding multimodal content extraction model training device can obtain the input manually annotated correct text content.

[0151] S405: construct first training data based on the page and the correct text content of the page;

[0152] S406, constructing second training data based on pages with format detection errors, text contents with format errors, and manually annotated correct text contents;

[0153] In this embodiment, the above-mentioned method can effectively ensure the accuracy of the constructed first training data and second training data, thereby providing effective data support for the subsequent training of the multimodal content extraction model.

[0154] S407, using the first training data to train the multimodal content extraction model for the SFT task;

[0155] For example, specific training may include the following steps:

[0156] (a2) for the first training data, obtaining predicted text content of a page of the first training data based on the multimodal content extraction model;

[0157] (b2) Based on the predicted text content of the page of the first training data and the correct text content of the page of the first training data, the parameters of the multimodal content extraction model are adjusted to implement the training of the SFT task.

[0158] Specifically, the first training data only includes the page and the correct text content of the page. These two parts of data can realize the training of the SFT task. For details, you can also refer to the training principle of the SFT task, which will not be repeated here. By training the multimodal content extraction model for the SFT task, the accuracy of the multimodal content extraction model extraction task can be effectively improved.

[0159] Specifically, when step (a2) is implemented, it may include the following steps:

[0160] (a3) for the first training data, obtaining a region block of each modality included in a page of the first training data;

[0161] For example, for the first training data, a pre-trained modality recognition model may be used to identify the region blocks corresponding to each modality included in the page of the first training data.

[0162] Alternatively, in this embodiment, the region blocks of each mode included in the page of the first training data may also be identified based on the pre-configured mode recognition rule. Figure 2 The relevant records of the illustrated embodiment will not be repeated here.

[0163] No matter which of the above methods is used, the area blocks of each mode in the page can be accurately and effectively obtained.

[0164] (b3) extracting text information of each modality in the page of the first training data using a multimodal content extraction model based on the regional blocks of each modality included in the page of the first training data;

[0165] (c3) splicing the text information of each mode in the page of the first training data to obtain the predicted text content of the page of the first training data.

[0166] For details, please refer to the above Figure 2 The relevant contents of the illustrated embodiment will not be repeated here.

[0167] S408, using the second training data to train the multimodal content extraction model for the DPO task;

[0168] For example, when step S408 is specifically implemented, it may include the following steps:

[0169] (a4) for the second training data, obtaining predicted text content of a page of the second training data based on the multimodal content extraction model;

[0170] For example, when step (a4) is specifically implemented, it may include the following steps:

[0171] (a5) for the second training data, obtaining a region block of each modality included in a page of the second training data;

[0172] (b5) extracting text information of each modality in the page of the second training data using a multimodal content extraction model based on the regional blocks of each modality included in the page of the second training data;

[0173] (c5) splicing the text information of each mode in the page of the second training data to obtain the predicted text content of the page of the second training data.

[0174] Similarly, step (c5) can be implemented by setting a splicing unit in the multimodal content extraction model, or can be implemented outside the multimodal content extraction model. Figure 2 The relevant contents of the illustrated embodiment will not be repeated here.

[0175] (b4) Based on the predicted text content of the pages of the second training data, the correct text content of the pages of the second training data, and the malformed text content of the pages of the second training data, the parameters of the multimodal content extraction model are adjusted to implement the training of the DPO task.

[0176] Specifically, the pages in the second training data, the text content with incorrect format, and the manually annotated correct text content can realize the training of the DPO task. For details, please refer to the training principle of the DPO task, which will not be repeated here. By training the multimodal content extraction model for the DPO task, the accuracy of the multimodal content extraction model extraction task can be effectively improved.

[0177] In this embodiment, the example of first training the SFT task and then training the DPO task is taken. In practical applications, the DPO task can be trained first and then the SFT task; or the two can be performed in turn; first, a round of SFT task training is performed using multiple first training data, and then a round of DPO task training is performed using multiple second training data, and the training is repeated in this manner. In short, no matter which training method is used, it is ensured that the multimodal content extraction model is trained on two tasks. The training method of this embodiment allows the multimodal content extraction model to be trained on two tasks, which can effectively improve the accuracy of the multimodal content extraction model.

[0178] S409, using the test set to test the multimodal content extraction model;

[0179] In this embodiment, the test set includes multiple pages and information of multiple modes, and each page includes information of at least one mode.

[0180] Specifically, the test set is used to test whether the information extracted by the multimodal content extraction model is correct, and for each test data in the test set, a corresponding test result can be annotated, that is, the test result can indicate whether the information extracted by the content extraction model for the test data is correct.

[0181] S410, based on the test results, obtaining difficult page samples from the test set;

[0182] The difficult page sample in this embodiment may refer to a test page where errors occur in information extraction by the content extraction model.

[0183] Optionally, in this embodiment, the difficult page samples can be directly used to train the multimodal content extraction model. However, considering that the number of difficult page samples is small, further, in this embodiment, the following method can be used to enrich the number of difficult samples and enhance the training effect.

[0184] S411, based on the difficult page sample, obtaining a reference difficult page from a page library;

[0185] The reference difficult page in this embodiment is a sample in the page library that matches the difficult page sample.

[0186] For example, the specific implementation may include the following steps:

[0187] (a6) based on the difficult page sample, obtaining a corresponding modal layout diagram, where the modal layout diagram includes the modalities of each area block in the page;

[0188] Specifically, please refer to the above Figure 2 The records of the illustrated embodiment can identify the regional blocks of each mode included in the page, and then obtain the modal layout diagram of the page.

[0189] (b6) Based on the modal layout diagram of the difficult page sample, pages with the same layout are obtained from the pre-generated page library as reference difficult pages.

[0190] S412. Based on the reference difficult page, continue to train the multimodal content extraction model.

[0191] Specifically, the method of continuing training may adopt the method of steps S402 to S408 above, which will not be described in detail here.

[0192] Since the multimodal content extraction model does not have enough accuracy in extracting samples of difficult pages, it can be considered that the multimodal content extraction model has difficulty in extracting pages of this type of modal layout diagram. At this time, pages with the same layout can be screened as reference difficult pages. Based on the reference difficult pages, the multimodal content extraction model can be effectively trained in a targeted manner to improve the accuracy of the multimodal content extraction model. Moreover, in this embodiment, the reference difficult pages obtained in this way are very reasonable and accurate.

[0193] In addition, optionally, in this embodiment, if the number of difficult page samples is also small, we can further refer to the modal information included in the difficult page samples, obtain pages including the same modal information from the page library, and use them as reference difficult pages to effectively train the multimodal content extraction model.

[0194] The training method of the multimodal content extraction model based on a large model in this embodiment, by adopting the above-mentioned technical solution and the data distillation technology, accurately and effectively constructs the first training data and the second training data, and then uses the first training data to train the multimodal content extraction model for the SFT task, and uses the second training data to train the multimodal content extraction model for the DPO task, which can effectively improve the accuracy of the multimodal content extraction model. Furthermore, in this embodiment, reference difficult pages are further reasonably, accurately and effectively obtained, and the reference difficult pages are used to train the multimodal content extraction model, which can further and more effectively train the multimodal content extraction model in a more targeted manner, and can further improve the comprehensiveness and accuracy of the pages extracted by the trained multimodal content extraction model.

[0195] Figure 5 is a schematic diagram according to the fifth embodiment of the present disclosure; Figure 5 As shown, this embodiment provides a multimodal content extraction method based on a large model, which may specifically include the following steps:

[0196] S501, obtaining a page set of a multimodal document, the multimodal document including multiple pages of content, and the page set including pages of each page of content;

[0197] S502, based on the page set of the multimodal document, using a pre-trained multimodal content extraction model to extract text content of the multimodal document;

[0198] The multimodal content extraction model of this embodiment is trained based on a large language model; the multimodal content extraction model can extract text content of each modality in a multimodal document.

[0199] The multimodal content extraction method based on a large model in this embodiment describes how to extract the text content of a multimodal document. Figure 1 and Figure 2 The difference of the embodiment shown is that only the text content of the multimodal document is extracted; it is not necessary to use a pre-trained information extraction model based on the text content of the multimodal document to perform further information extraction. Figure 1 The relevant records of the illustrated embodiment will not be repeated here.

[0200] The multimodal content extraction method based on a large model in this embodiment refers to the above Figure 3 or Figure 4 The method for using the multimodal content extraction model trained in the illustrated embodiment.

[0201] The large model-based multimodal content extraction method of this embodiment extracts content from multimodal documents by using a multimodal content extraction model. Since the multimodal content extraction model can extract the text content of each modality in the multimodal document, the technical solution of this embodiment can comprehensively and accurately extract the text content of the multimodal document.

[0202] Further optionally, in the above Figure 5 Based on the technical solution of the embodiment shown, step S502, when implemented specifically, may include the following steps:

[0203] (a7) for each page in the page set of the multimodal document, using a multimodal content extraction model to extract text information of each modality in the page;

[0204] (b7) splicing the text information of each mode in the page to obtain the text content of the page;

[0205] (c7) The text contents corresponding to the pages of the multimodal document are concatenated to obtain the text contents of the multimodal document.

[0206] Specifically, steps (a7)-(c7) can be completed in a multimodal content extraction model. Steps (b7) and (c7) can be implemented by setting a splicing unit in the multimodal content extraction model to achieve the splicing of text content of different modes in the same page and the splicing of text content of different pages. For the multimodal content extraction model, it can be an end-to-end model, the input is each page in the page set of the multimodal document, and the output is the text content of the multimodal document.

[0207] Optionally, step (b7) and step (c7) may also be implemented outside the multimodal content extraction model to ensure the realization of the functions.

[0208] Steps (a7) to (c7) are a specific implementation of step S502 of the embodiment shown in above 5. In this way, the text content of the multimodal document can be fully and accurately acquired.

[0209] Further optionally, when the above step (a7) is implemented specifically, it may include the following steps:

[0210] (a8) for each page in the page set of the multimodal document, obtaining a region block of each modality included in the page;

[0211] For example, the specific implementation of step (a8) may include the following two methods:

[0212] Method 1: For each page in the page set of the multimodal document, a pre-trained modality recognition model is used to identify the regional blocks of each modality included in the page.

[0213] The modality recognition model is a pre-trained neural network model. When in use, a page is input into the modality recognition model, and the modality recognition model can identify each modality and corresponding region block included in the page. For example, the region block can be identified by position information, and the position of the region block in the page can be identified. Alternatively, the multimodal content recognition model of this embodiment can also be directly used to identify the region blocks of each modality included in the page.

[0214] Method 2: for each page in the page set of the multimodal document, based on the pre-configured modality identification rules, the regional blocks of each modality included in the page are identified.

[0215] The pre-set modal recognition rules can refer to the above Figure 2 The relevant records of the illustrated embodiment will not be repeated here.

[0216] (b8) Based on the regional blocks of each modality included in the page, a multimodal content extraction model is used to extract text information of each modality in the page.

[0217] In this embodiment, the text information of each modality is in markdown format; correspondingly, the text content of the spliced ​​page and the text content of the multimodal document are both in markdown format.

[0218] The modalities in the multimodal document in this embodiment include: at least two modalities of text, picture, web page, table, chart, and handwritten signature.

[0219] The multimodal content extraction method based on a large model in this embodiment can be specifically implemented by referring to the above Figure 1 or Figure 2The relevant records of content extraction based on the multimodal content extraction model in the illustrated embodiment will not be repeated in detail here.

[0220] The multimodal content extraction method based on a large model of this embodiment uses a multimodal content extraction model to extract content, which can fully and accurately obtain the text content of each modality, and effectively ensure the comprehensiveness and accuracy of the text content of the extracted multimodal document.

[0221] Figure 6 is a schematic diagram according to the sixth embodiment of the present disclosure; Figure 6 As shown, this embodiment provides an information extraction device 600 for multimodal documents based on a large model, comprising:

[0222] The page acquisition module 601 is used to acquire a page set of a multimodal document, wherein the multimodal document includes multiple pages of content, and the page set includes pages of each page of content;

[0223] A document content extraction module 602 is used to extract text content of the multimodal document based on the page set of the multimodal document using a pre-trained multimodal content extraction model, wherein the multimodal content extraction model is trained based on a large language model; the multimodal content extraction model can extract text content of each modality in the multimodal document;

[0224] The information extraction module 603 is used to extract information based on the text content of the multimodal document using a pre-trained information extraction model, where the information extraction model is also trained based on a large language model.

[0225] The information extraction device 600 for multimodal documents based on a large model in this embodiment implements the implementation principle of information extraction for multimodal documents based on a large model by adopting the above-mentioned modules, which is the same as the implementation of the above-mentioned related method embodiments. For details, please refer to the records of the above-mentioned related method embodiments, which will not be repeated here.

[0226] Figure 7 is a schematic diagram according to the seventh embodiment of the present disclosure; Figure 7 As shown, this embodiment provides an information extraction device 700 for multimodal documents based on a large model, comprising the above Figure 6 The modules with the same name and function are shown as follows: page acquisition module 601 , document content extraction module 602 , and information extraction module 603 .

[0227] like Figure 7 As shown, in this embodiment, the document content extraction module 602 includes:

[0228] The text extraction unit 7021 is used to extract text information of each mode in each page of the page set of the multimodal document using the multimodal content extraction model;

[0229] A splicing unit 7022, used for splicing the text information of each mode in the page to obtain the text content of the page;

[0230] The splicing unit 7022 is further used to splice the text contents corresponding to the pages of each page content in the multimodal document to obtain the text contents of the multimodal document.

[0231] Further optionally, in one embodiment of the present disclosure, the text extraction unit 6021 is used to:

[0232] For each page in the page set of the multimodal document, obtaining a region block of each modality included in the page;

[0233] Based on the regional blocks of each modality included in the page, the multimodal content extraction model is used to extract text information of each modality in the page.

[0234] Further optionally, in one embodiment of the present disclosure, the text extraction unit 7021 is used to:

[0235] For each page in the page set of the multimodal document, a pre-trained modality recognition model is used to identify the region blocks of each modality included in the page.

[0236] Further optionally, in one embodiment of the present disclosure, the text extraction unit 7021 is used to:

[0237] For each page in the page set of the multimodal document, based on a pre-configured modality identification rule, a region block of each modality included in the page is identified.

[0238] Further optionally, in one embodiment of the present disclosure, the text information of each modality is in markdown format;

[0239] The modalities in the multimodal document include: at least two modalities of text, picture, web page, table, chart, and handwritten signature.

[0240] Further optionally, if Figure 7 As shown, in one embodiment of the present disclosure, the information extraction module 703 includes:

[0241] Configuration unit 7031, used for configuring information extraction specifications;

[0242] The information extraction unit 7032 is used to extract information using the information extraction model according to the text content of the multimodal document and the configured information extraction specification.

[0243] Further optionally, in one embodiment of the present disclosure, the information extraction specification includes an identification of the field to be extracted, and also includes at least one of the description information of the field and an example of the value of the field; the description information of the field includes explanation information of the field and / or the location of the field.

[0244] The information extraction device 700 for multimodal documents based on a large model in this embodiment implements the information extraction principle of multimodal documents based on a large model by adopting the above-mentioned modules, which is the same as the implementation of the above-mentioned related method embodiments. For details, please refer to the records of the above-mentioned related method embodiments, which will not be repeated here.

[0245] Figure 8 is a schematic diagram according to an eighth embodiment of the present disclosure; Figure 8 As shown, this embodiment provides a training device 800 for a multimodal content extraction model based on a large model, comprising:

[0246] The acquisition module 801 is used to acquire the pages of the contents of each page of each multimodal document in a plurality of multimodal documents and generate a page library;

[0247] A content extraction module 802 is used to extract text content of each page in the page library using a pre-trained multimodal macro model;

[0248] A construction module 803 is used to construct first training data suitable for supervised fine-tuning task training and second training data suitable for direct preference optimization task training based on the text content of each of the pages;

[0249] The training module 804 is used to perform supervised fine-tuning task training and direct preference optimization task training on the multimodal content extraction model based on the first training data and the second training data, respectively; the multimodal content extraction model has fewer parameters than the multimodal large model; the multimodal content extraction model is trained based on the large language model.

[0250] The training device 800 of the multimodal content extraction model based on a large model of this embodiment implements the implementation principle of the training of the multimodal content extraction model based on a large model by adopting the above-mentioned modules, which is the same as the implementation of the above-mentioned related method embodiments. For details, please refer to the records of the above-mentioned related method embodiments, which will not be repeated here.

[0251] Fig. 9 is a schematic diagram according to the ninth embodiment of the present disclosure; Fig. 9As shown, this embodiment provides a training device 900 for a multimodal content extraction model based on a large model, including the above Figure 7 The modules with the same name and function are shown as: acquisition module 901 , content extraction module 902 , construction module 903 and training module 904 .

[0252] like Fig. 9 As shown, in this embodiment, the construction module 903 includes:

[0253] The checking unit 9031 is used to check the format of the text content of each of the pages;

[0254] An acquisition unit 9032 is used for acquiring manually annotated correct text content in response to a format check error existing in the text content of the page;

[0255] A construction unit 9033, configured to construct first training data based on the page and the correct text content of the page;

[0256] The construction unit 9033 is further used to construct second training data based on the pages with format detection errors, the text contents with format errors, and the manually annotated correct text contents.

[0257] Further optionally, if Fig. 9 As shown, in one embodiment of the present disclosure, the training module 904 includes:

[0258] A first training unit 9041 is used to train the multimodal content extraction model for a supervised fine-tuning task using the first training data;

[0259] The second training unit 9042 is used to use the second training data to train the multimodal content extraction model for a direct preference optimization task.

[0260] Further optionally, in one embodiment of the present disclosure, the first training unit 9041 is used to:

[0261] For the first training data, obtaining predicted text content of a page of the first training data based on a multimodal content extraction model;

[0262] Based on the predicted text content of the page of the first training data and the correct text content of the page of the first training data, the parameters of the multimodal content extraction model are adjusted to achieve the training of the supervised fine-tuning task.

[0263] Further optionally, in one embodiment of the present disclosure, the first training unit 9041 is used to:

[0264] For the first training data, obtaining a region block of each mode included in a page of the first training data;

[0265] Based on the regional blocks of each modality included in the page of the first training data, the multimodal content extraction model is used to extract text information of each modality in the page of the first training data;

[0266] The text information of each mode in the page of the first training data is concatenated to obtain the predicted text content of the page of the first training data.

[0267] Further optionally, in one embodiment of the present disclosure, the first training unit 9041 is used to:

[0268] For the first training data, a pre-trained modality recognition model is used to identify each modality and a corresponding region block included in a page of the first training data;

[0269] Based on the identified modes and corresponding regions included in the page of the first training data, the region blocks of the modes included in the page of the first training data are obtained.

[0270] Further optionally, in one embodiment of the present disclosure, the first training unit 9041 is used to:

[0271] Based on preconfigured modality identification rules, regional blocks of each modality included in the page of the first training data are identified.

[0272] Further optionally, in one embodiment of the present disclosure, the second training unit 9042 is used to:

[0273] For the second training data, obtaining predicted text content of a page of the second training data based on a multimodal content extraction model;

[0274] Based on the predicted text content of the pages of the second training data, the correct text content of the pages of the second training data, and the malformed text content of the pages of the second training data, the parameters of the multimodal content extraction model are adjusted to achieve training of the direct preference optimization task.

[0275] Further optionally, in one embodiment of the present disclosure, the second training unit 9042 is used to:

[0276] For the second training data, obtaining a region block of each mode included in a page of the second training data;

[0277] Based on the regional blocks of each modality included in the page of the second training data, using the multimodal content extraction model, extracting text information of each modality in the page of the second training data;

[0278] The text information of each mode in the page of the second training data is concatenated to obtain the predicted text content of the page of the second training data.

[0279] Further optionally, if Fig. 9 As shown, in one embodiment of the present disclosure, the training device 900 for the multimodal content extraction model further includes:

[0280] A testing module 905 is used to test the multimodal content extraction model using a test set; the test set includes multiple pages and information of multiple modes, and each page includes information of at least one mode;

[0281] An acquisition module 906, configured to acquire a difficult page sample from the test set based on the test result;

[0282] The acquisition module 906 is further configured to acquire a matching difficult page from a pre-created page set based on the difficult page sample;

[0283] The training module 904 is used to continue training the multimodal content extraction model based on the difficult-to-match page.

[0284] Further optionally, in one embodiment of the present disclosure, the acquisition module 906 is used to:

[0285] Based on the difficult page sample, a corresponding modal layout diagram is obtained, wherein the modal layout diagram includes a mode of each area block in the page;

[0286] Based on the modal layout diagram of the difficult page sample, a page with the same layout is obtained from a pre-created training page library as a matching difficult page.

[0287] The training device 900 of the multimodal content extraction model based on a large model of this embodiment implements the implementation principle of the training of the multimodal content extraction model based on a large model by adopting the above-mentioned modules, which is the same as the implementation of the above-mentioned related method embodiments. For details, please refer to the records of the above-mentioned related method embodiments, which will not be repeated here.

[0288] Fig.10 is a schematic diagram according to the tenth embodiment of the present disclosure; Fig.10 As shown, this embodiment provides a multimodal content extraction device 1000 based on a large model, including:

[0289] The page acquisition module 1001 is used to acquire a page set of a multimodal document, wherein the multimodal document includes multiple pages of content, and the page set includes pages of each page of content;

[0290] The document content extraction module 1002 is used to extract the text content of the multimodal document based on the page set of the multimodal document using a pre-trained multimodal content extraction model, wherein the multimodal content extraction model is trained based on a large language model; the multimodal content extraction model can extract the text content of each modality in the multimodal document.

[0291] The large model-based multimodal content extraction device 1000 of this embodiment implements the implementation principle of large model-based multimodal content extraction by adopting the above-mentioned modules, which is the same as the implementation principle of the above-mentioned related method embodiments. For details, please refer to the records of the above-mentioned related method embodiments, which will not be repeated here.

[0292] Further optionally, in one embodiment of the present disclosure, the above Fig.10 In the multimodal content extraction device 1000 based on a large model of the illustrated embodiment, the document content extraction module 1002 is used to:

[0293] For each page in the page set of the multimodal document, using the multimodal content extraction model to extract text information of each modality in the page;

[0294] splicing the text information of each mode in the page to obtain the text content of the page;

[0295] The text contents corresponding to the pages of the content of each page in the multimodal document are concatenated to obtain the text contents of the multimodal document.

[0296] Further optionally, in one embodiment of the present disclosure, the document content extraction module 1002 is used to:

[0297] For each page in the page set of the multimodal document, obtaining a region block of each modality included in the page;

[0298] Based on the regional blocks of each modality included in the page, the multimodal content extraction model is used to extract text information of each modality in the page.

[0299] Further optionally, in one embodiment of the present disclosure, the document content extraction module 1002 is used to:

[0300] For each page in the page set of the multimodal document, a pre-trained modality recognition model is used to identify the region blocks of each modality included in the page.

[0301] Further optionally, in one embodiment of the present disclosure, the document content extraction module 1002 is used to:

[0302] For each page in the page set of the multimodal document, based on a pre-configured modality identification rule, a region block of each modality included in the page is identified.

[0303] Further optionally, in one embodiment of the present disclosure, the text information of each modality is in markdown format;

[0304] The modalities in the multimodal document include: at least two modalities of text, picture, web page, table, chart, and handwritten signature.

[0305] The large model-based multimodal content extraction device 1000 of the above embodiment implements the implementation principle of large model-based multimodal content extraction by adopting the above module, which is the same as the implementation of the above related method embodiments. For details, please refer to the records of the above related method embodiments, which will not be repeated here.

[0306] Fig.11 is a schematic diagram according to the eleventh embodiment of the present disclosure; Fig.11 As shown, this embodiment provides an intelligent agent 1100, including:

[0307] An input module 1101 is used to receive an input multimodal document;

[0308] The processing module 1102 is used to determine the information extraction task based on the multimodal document received by the input module, determine the multimodal content extraction model and the information extraction model based on the information extraction task, and perform the above by calling the multimodal content extraction model and the information extraction model. Figure 1 or Figure 2 The information extraction method of multimodal documents based on large models obtains information extraction results;

[0309] The output module 1103 is used to output the information extraction result obtained by the processing module.

[0310] The multimodal content extraction model of this embodiment is as follows: Figure 8 or Fig. 9 The training device of the multimodal content extraction model based on the large model adopts the above Figure 3 or Figure 4 The training method and the multimodal content extraction model obtained by training; the training of the information extraction model can also refer to the records of the above-mentioned related embodiments, which will not be repeated here.

[0311] The agent 1100 of this embodiment realizes the realization principle and technical effect of information extraction, which can be referred to above Figure 1 or Figure 2 The description of the illustrated embodiment will not be repeated here.

[0312] Fig.12 is a schematic diagram according to the twelfth embodiment of the present disclosure; Fig.12 As shown, this embodiment provides an intelligent agent 1200, including:

[0313] An input module 1201, used for receiving an input multimodal document;

[0314] The processing module 1202 is used to determine a content extraction task based on the multimodal document received by the input module, determine a multimodal content extraction model based on the content extraction task, and execute the multimodal content extraction model by calling the multimodal content extraction model. Figure 5 The multimodal content extraction method based on the large model of the embodiment shown obtains the content extraction result;

[0315] The output module 1203 is used to output the content extraction result obtained by the processing module.

[0316] The multimodal content extraction model of this embodiment is as follows: Figure 8 or Fig. 9 The training device of the multimodal content extraction model based on the large model adopts the above Figure 3 or Figure 4 The training method described above obtains a multimodal content extraction model through training.

[0317] The intelligent agent 1200 of this embodiment realizes the realization principle and technical effect of content extraction, which can be referred to above Figure 5 The description of the illustrated embodiment will not be repeated here.

[0318] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0319] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0320] Fig.13 A schematic block diagram of an example electronic device 1300 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0321] like Fig.13 As shown, the device 1300 includes a computing unit 1301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1302 or a computer program loaded from a storage unit 1308 into a random access memory (RAM) 1303. In the RAM 1303, various programs and data required for the operation of the device 1300 can also be stored. The computing unit 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0322] A number of components in the device 1300 are connected to the I / O interface 1305, including: an input unit 1306, such as a keyboard, a mouse, etc.; an output unit 1307, such as various types of displays, speakers, etc.; a storage unit 1308, such as a disk, an optical disk, etc.; and a communication unit 1309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1309 allows the device 1300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0323] The computing unit 1301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1301 performs the various methods and processes described above, such as the above-mentioned methods of the present disclosure. For example, in some embodiments, the above-mentioned methods of the present disclosure may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1308. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1300 via the ROM 1302 and / or the communication unit 13013. When the computer program is loaded into the RAM 1303 and executed by the computing unit 1301, one or more steps of the above-mentioned methods of the present disclosure described above may be performed. Alternatively, in other embodiments, the computing unit 1301 may be configured to execute the above method of the present disclosure in any other appropriate manner (for example, by means of firmware).

[0324] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0325] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0326] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0327] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0328] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0329] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0330] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0331] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for extracting information from multimodal documents based on a large model, comprising the following steps: Acquire a page set of a multimodal document, the multimodal document including multiple pages of content, the page set including pages of each page of content; Based on the page set of the multimodal document, a pre-trained multimodal content extraction model is used to extract text content of the multimodal document, wherein the multimodal content extraction model is trained based on a large language model; the multimodal content extraction model can extract text content of each modality in the multimodal document; Based on the text content of the multimodal document, information extraction is performed using a pre-trained information extraction model, which is also trained based on a large language model.

2. The method according to claim 1, wherein: Based on the page set of the multimodal document, a pre-trained multimodal content extraction model is used to extract text content of the multimodal document, including: For each page in the page set of the multimodal document, using the multimodal content extraction model to extract text information of each modality in the page; splicing the text information of each mode in the page to obtain the text content of the page; The text contents corresponding to the pages of the content of each page in the multimodal document are concatenated to obtain the text contents of the multimodal document.

3. The method according to claim 2, wherein: For each page in the page set of the multimodal document, the multimodal content extraction model is used to extract text information of each modality in the page, including: For each page in the page set of the multimodal document, obtaining a region block of each modality included in the page; Based on the regional blocks of each modality included in the page, the multimodal content extraction model is used to extract text information of each modality in the page.

4. The method according to claim 3, wherein: For each page in the page set of the multimodal document, obtaining a region block of each modality included in the page includes: For each page in the page set of the multimodal document, a pre-trained modality recognition model is used to identify the region blocks of each modality included in the page.

5. The method according to claim 3, wherein: For each page in the page set of the multimodal document, obtaining a region block of each modality included in the page includes: For each page in the page set of the multimodal document, based on a pre-configured modality identification rule, a region block of each modality included in the page is identified.

6. The method according to claim 3, wherein: The text information of each mode is in markdown format; The modalities in the multimodal document include: at least two modalities of text, picture, web page, table, chart, and handwritten signature.

7. The method according to claim 1, wherein: Based on the text content of the multimodal document, a pre-trained information extraction model is used to extract information, including: Configuration information extraction specifications; According to the text content of the multimodal document and the configured information extraction specification, the information extraction model is adopted to perform information extraction.

8. The method according to claim 7, wherein: The information extraction specification includes the identification of the field to be extracted, and also includes at least one of the description information of the field and an example of the value of the field; the description information of the field includes the explanation information of the field and / or the location of the field.

9. An information extraction device for multimodal documents based on a large model, comprising: A page acquisition module, used to acquire a page set of a multimodal document, wherein the multimodal document includes multiple pages of content, and the page set includes pages of each page of content; A document content extraction module, configured to extract text content of the multimodal document based on a page set of the multimodal document using a pre-trained multimodal content extraction model, wherein the multimodal content extraction model is trained based on a large language model; the multimodal content extraction model is capable of extracting text content of each modality in the multimodal document; The information extraction module is used to extract information based on the text content of the multimodal document using a pre-trained information extraction model, wherein the information extraction model is also trained based on the large language model.

10. The device according to claim 9, wherein: The document content extraction module comprises: A text extraction unit, configured to extract text information of each mode in each page of the page set of the multimodal document by using the multimodal content extraction model; A splicing unit, used for splicing the text information of each mode in the page to obtain the text content of the page; The splicing unit is further used to splice the text contents corresponding to the pages of each page content in the multimodal document to obtain the text contents of the multimodal document.

11. The device according to claim 10, wherein: The text extraction unit is used to: For each page in the page set of the multimodal document, obtaining a region block of each modality included in the page; Based on the regional blocks of each modality included in the page, the multimodal content extraction model is used to extract text information of each modality in the page.

12. The device according to claim 11, wherein: The text extraction unit is used to: For each page in the page set of the multimodal document, a pre-trained modality recognition model is used to identify the region blocks of each modality included in the page.

13. The device according to claim 11, wherein: The text extraction unit is used to: For each page in the page set of the multimodal document, based on a pre-configured modality identification rule, a region block of each modality included in the page is identified.

14. The device according to claim 11, wherein: The text information of each mode is in markdown format; The modalities in the multimodal document include: at least two modalities of text, picture, web page, table, chart, and handwritten signature.

15. The device according to claim 9, wherein: The information extraction module comprises: Configuration unit, used to configure information extraction specifications; The information extraction unit is used to extract information using the information extraction model according to the text content of the multimodal document and the configured information extraction specification.

16. The device according to claim 15, wherein: The information extraction specification includes the identification of the field to be extracted, and also includes at least one of the description information of the field and an example of the value of the field; the description information of the field includes the explanation information of the field and / or the location of the field.

17. An intelligent agent, comprising: An input module, used for receiving an input multimodal document; A processing module, configured to determine an information extraction task based on the multimodal document received by the input module, determine a multimodal content extraction model and an information extraction model based on the information extraction task, and obtain an information extraction result by executing the method according to any one of claims 1 to 8 above by calling the multimodal content extraction model and the information extraction model; An output module is used to output the information extraction result obtained by the processing module.

18. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

19. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-8.

20. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images