A method for obtaining text based on a hierarchical memory network multi-modal large model
By constructing a multimodal large model with a hierarchical memory network, the problems of 'page silos' and 'dialogue amnesia' in long document analysis are solved, cross-page information association and continuous dialogue memory are realized, and the interaction depth and efficiency of the multimodal large model are improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 北京中科闻歌科技股份有限公司
- Filing Date
- 2026-03-06
- Publication Date
- 2026-06-05
AI Technical Summary
Existing multimodal large models suffer from 'page silos' and 'dialogue amnesia' when processing long documents. They cannot effectively understand the global structure of the document and the cross-page information association, and lack persistent dialogue memory, which limits the depth and fluency of interaction.
A multimodal large model based on a hierarchical memory network is constructed. By acquiring target summary text and continuously incorporating key historical dialogue information across pages, a memory text library is established to provide a structured global context, thereby realizing stateful dialogue memory and cross-page information association.
It solves the problem of context fragmentation caused by fragmented processing in existing technologies, realizes smooth and in-depth multi-turn continuous dialogue, improves the intelligence and efficiency of long document analysis and interaction, and reduces the user burden.
Smart Images
Figure CN122152997A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of document analysis technology, and in particular to a method for acquiring text based on a multimodal large model of hierarchical memory networks. Background Technology
[0002] In recent years, large-scale multimodal language models, as a key frontier of general artificial intelligence, have been dedicated to achieving deep understanding and natural interaction with complex documents interwoven with elements such as text, images, and tables. As large language models have demonstrated powerful capabilities in pure text tasks, extending their capabilities to the multimodal domain, particularly the analysis of long, multi-page real-world documents, has become a core focus for both academia and industry.
[0003] Existing technologies primarily follow two paradigms of evolution. The first is collaborative systems, whose core idea is to utilize a large language model as a "brain" or controller to schedule and invoke visual models specialized for specific tasks. For example, VisualChatGPT uses ChatGPT as a dialogue interface to coordinate a series of basic visual models to complete editing tasks; MM-REACT guides ChatGPT in multimodal reasoning and action planning through designed prompts; and HuggingGPT allows ChatGPT to act as a system controller, managing and integrating numerous AI models on the Hugging Face platform. While these methods are flexible, they suffer from issues such as insufficient alignment, efficiency, and error accumulation between different modal models.
[0004] To overcome the aforementioned limitations, a second paradigm—a unified end-to-end model architecture—emerged. This type of approach aims to train a single model that can directly understand multimodal inputs. Representative works include: BLIP-2 innovatively introduces a lightweight Q-Former module to establish an efficient bridge between the frozen visual encoder and the large language model, achieving tight alignment between vision and language; LLaVA proposes a "visual instruction fine-tuning" paradigm, utilizing visual features guided by the language model for high-quality data synthesis and training; MiniGPT-4 demonstrates that connecting the visual encoder and language model through a simple linear projection layer can stimulate powerful multimodal understanding and generation capabilities; and the Flamingo model focuses on improving the model's multimodal reasoning ability in low-sample scenarios. These unified models significantly improve overall performance and unlock enormous potential.
[0005] However, when faced with long real-world documents such as reports, contracts, and papers, existing technologies reveal fundamental bottlenecks. Firstly, there's the "page silo" problem: whether collaborative or end-to-end models, each page of a document is typically treated as an isolated unit, or only a limited sliding window is used, lacking a coherent grasp of the document's global structure, cross-page information connections, and overall narrative logic. Secondly, there's the "dialogue amnesia" problem: existing models generally employ a stateless dialogue mode, treating each question and answer as an independent task, failing to effectively remember, track, and utilize previous interaction history and context. This causes the model to be unable to understand follow-up questions that rely on historical context, such as "reviewing the third point mentioned earlier" or "comparing the chart on the previous page with the current data," severely impairing the depth and fluency of the interaction. These two major obstacles—"page silos" and "dialogue amnesia"—limit the breakthrough application of multimodal large-scale models in deep analysis of long documents and intelligent discussions. Summary of the Invention
[0006] To address the aforementioned technical problems, this invention provides a method for acquiring text using a multimodal large model based on a hierarchical memory network. By constructing a target summary text and continuously incorporating key historical dialogue information across pages (first text, second text, etc.), the model can provide a structured global context when processing subsequent questions. This enables the model to transcend the limitations of single pages or sliding windows, understanding and associating information scattered across multiple parts of a document, fundamentally solving the context fragmentation problem caused by fragmented processing in existing technologies.
[0007] According to a first aspect of the present invention, a method for acquiring text based on a multimodal large model of a hierarchical memory network is provided, the method comprising the following steps:
[0008] S100, Obtain the first question statement and first text provided by the target user;
[0009] S200, input the first question statement and the first text into the trained large model to obtain the first answer statement corresponding to the first question statement;
[0010] S300, obtain the second question statement and second text provided by the target user;
[0011] S400, input the second question statement, the first text style, and the second text into the trained large model to obtain the second answer statement corresponding to the second question statement;
[0012] S500, obtain the set of third-party question statements provided by the target user;
[0013] S600, input any one of the third question statements in the set of third question statements, the first text, the second text and the third text corresponding to the third question statement into the trained large model to obtain the third answer statement corresponding to the third question statement;
[0014] S700, Generate the target dialogue text based on all the third question statements and the third answer statement corresponding to each of the third question statements.
[0015] Specifically, the first question statement is the first question statement provided by the target user.
[0016] Specifically, step S100 also includes the following step to obtain the first text:
[0017] S101, Obtain the target document;
[0018] S101, Based on the target document, obtain the target summary text corresponding to the target document and use the target summary text as the first text.
[0019] Specifically, the target document is a document whose number of pages exceeds a preset page number threshold.
[0020] Specifically, step S101 also includes the following steps to obtain the target summary text:
[0021] S1011, Obtain the target document text set A={A1, ..., A2} of the target document. i , ..., A m}, A i It is the text of the i-th target document, where the value of i ranges from 1 to m, and m is the number of target document texts in the target document;
[0022] S1012, each A i The input is fed into the first encoder to obtain A. i The corresponding first document feature vector B i Among them, B i ={B i1 B i2 B i3 B i4}, B i1 It is A i The corresponding text content information, B i2 It is A i Corresponding visual layout information, B i3 It is A i The corresponding chart content information, B i4 It is A i Corresponding image semantic information;
[0023] S1013, Obtain the metadata C of the target document; wherein, the metadata of the target document includes the table of contents of the target document, the chapter titles of the target document, and the page numbers of the target document;
[0024] S1014, each B i Both combine with C to generate B. i The corresponding second document feature vector B 0 i ={B i1 B i2 B i3 B i4 ,C};
[0025] S1015, each B 0 i The text is input into the second encoder to obtain the summary text corresponding to the target document and to use the summary text corresponding to the target document as the target summary text.
[0026] Specifically, the second encoder is an encoder that adopts the Transformer architecture.
[0027] Specifically, the second question statement is the second question statement provided by the target user after the first question statement.
[0028] Specifically, in step S300, the second text is the text constructed from the first question statement and the first answer statement corresponding to the first question statement, and the second text is stored in the memory text library.
[0029] According to a second aspect of the present invention, a non-transitory computer-readable storage medium is provided, wherein at least one instruction or at least one program is stored therein, the at least one instruction or the at least one program being loaded and executed by a processor to implement the above-described method for acquiring text from a multimodal large model based on a hierarchical memory network.
[0030] According to a third aspect of the present invention, an electronic device is provided, including a processor and the aforementioned non-transitory computer-readable storage medium.
[0031] The present invention has at least the following beneficial effects:
[0032] 1. This invention constructs a target summary text and continuously incorporates key historical dialogue information (first text, second text, etc.) across pages, providing a structured global context when the model processes subsequent questions. This enables the model to transcend the limitations of single pages or sliding windows, understanding and associating information scattered across multiple parts of a document, fundamentally solving the context fragmentation problem caused by fragmented processing in existing technologies.
[0033] 2. By establishing and dynamically maintaining a memory text library containing historical question-and-answer pairs (such as the second text), and using relevant memories as contextual input, this invention endows the model with stateful and persistent dialogue memory capabilities. This enables the model to accurately understand complex follow-up questions that rely on historical interactions, such as reference, comparison, and progression, achieving truly fluent and in-depth multi-turn continuous dialogue, and enhancing the intelligence and depth of the interaction.
[0034] 3. Users no longer need to manually break down complex problems or repeatedly provide historical background. Through automated context building and memory management, the model can directly handle high-level tasks involving global analysis of long documents and multi-round comparisons, such as cross-chapter content sorting and before-and-after data comparison and demonstration, which greatly reduces the user's burden and improves the efficiency of analysis and discussion.
[0035] 4. This method is based on a general large-scale model architecture, and its core lies in the construction and management strategy of the input context, rather than being bound to a specific domain model. Therefore, it can be widely applied to various scenarios that require deep understanding of multimodal long documents and intelligent dialogue, such as financial report analysis, legal contract review, academic paper reading, and processing of long consulting documents, and has high scalability and practical value. Attached Figure Description
[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0037] Figure 1 This is a flowchart illustrating a method for acquiring text from a multimodal large model based on a hierarchical memory network, as provided in an embodiment of the present invention. Detailed Implementation
[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0039] Example 1 like Figure 1 As shown in the figure, this embodiment provides a method for obtaining text from a multimodal large model based on a hierarchical memory network. The method includes the following steps:
[0040] S100, Obtain the first question statement and first text provided by the target user.
[0041] Specifically, the first question statement is the first question statement provided by the target user.
[0042] Specifically, step S100 also includes the following step to obtain the first text:
[0043] S101, Obtain the target document;
[0044] S101, Based on the target document, obtain the target summary text corresponding to the target document and use the target summary text as the first text.
[0045] Specifically, the target document is a document whose page count exceeds a preset page count threshold; for example, the target document is a contract, report, etc. Those skilled in the art can set the preset page count threshold according to actual needs, which will not be elaborated here.
[0046] Specifically, step S101 also includes the following steps to obtain the target summary text:
[0047] S1011, Obtain the target document text set A={A1, ..., A2} of the target document. i , ..., A m}, A i It is the text of the i-th target document, where i ranges from 1 to m, and m is the number of target document texts in the target document; further, it can be understood as: the target document text is the text of each page in the target document.
[0048] S1012, each A i The input is fed into the first encoder to obtain A. i The corresponding first document feature vector B i Among them, B i ={B i1 B i2 B i3 B i4}, B i1 It is A i The corresponding text content information, B i2 It is A i Corresponding visual layout information, B i3 It is A i The corresponding chart content information, B i4 It is A i The corresponding semantic information of the image.
[0049] S1013, Obtain the metadata C of the target document; wherein, the metadata of the target document includes the table of contents of the target document, the chapter titles of the target document, and the page numbers of the target document; those skilled in the art are familiar with any method of obtaining the metadata of a target document in the prior art, and will not elaborate further here.
[0050] S1014, each B i Both combine with C to generate B. i The corresponding second document feature vector B 0 i ={B i1 B i2 B i3 B i4 ,C};
[0051] S1015, each B 0 i The text is input into the second encoder to obtain the summary text corresponding to the target document and to use the summary text corresponding to the target document as the target summary text.
[0052] Specifically, the second encoder is an encoder that adopts the Transformer architecture.
[0053] S200, input the first question statement and the first text into the trained large model to obtain the first answer statement corresponding to the first question statement.
[0054] Specifically, the trained large model includes an attention mechanism.
[0055] Furthermore, the attention mechanism weights W1 of the large model trained in step S200, wherein W1 satisfies the following condition:
[0056] Where Q1 is the vector corresponding to the first question statement, K1 is the vector corresponding to the keywords in the first text, and V1 is the vector corresponding to the first text.
[0057] S300, retrieve the second question statement and second text provided by the target user.
[0058] Specifically, the second question statement is the second question statement provided by the target user after the first question statement.
[0059] Specifically, in step S300, the second text is the text constructed from the first question statement and the first answer statement corresponding to the first question statement, and the second text is stored in the memory text library.
[0060] S400, input the second question statement, the first text style, and the second text into the trained large model to obtain the second answer statement corresponding to the second question statement.
[0061] Furthermore, the attention mechanism weights W2 of the large model trained in step S400, wherein W2 satisfies the following condition:
[0062] Where Q2 is the vector corresponding to the second question statement, K2 is the vector concatenated with the vector corresponding to the keywords in the first text and the vector corresponding to the keywords in the second text, and V2 is the vector concatenated with the vector corresponding to the first text and the vector corresponding to the second text.
[0063] S500, Obtain the set of third question statements D provided by the target user, where D = {D1, ..., D2} j , ..., D n}, D j It is the j-th third question statement, where j ranges from 1 to n, and n is the number of third question statements.
[0064] Specifically, the third question statement is a question statement provided by the target user after the second question statement.
[0065] Furthermore, D is sorted according to the order in which the questions are provided by the target users.
[0066] S600, input any one of the third question statements, the first text, the second text, and the third text corresponding to the third question statement in D into the trained large model to obtain the third answer statement corresponding to the third question statement.
[0067] Furthermore, the attention mechanism weights W3 of the large model trained in step S600, wherein W3 satisfies the following condition:
[0068] Where Q3 is the vector corresponding to the third question statement, K2 is the vector concatenated from the vectors corresponding to the keywords in the first text, the second text, and the third text, and V2 is the vector concatenated from the vectors corresponding to the first text, the second text, and the third text.
[0069] Furthermore, the third answer statement corresponding to the third question statement is inserted into the second text in S300 according to the sorting to update the second text; furthermore, the second text in S600 is the updated second text and the updated second text is stored in the memory text library.
[0070] Furthermore, the third text is obtained from the memory text library and is associated with the third question statement. This can be further understood as follows: the degree of association between the keywords in any text segment in the memory text library and the third question statement is used as the target degree of association, and the text segment content with the target degree of association being greater than the preset degree of association is written into the third text.
[0071] S700, generate target dialogue text based on all the statements of the third question and the third answer statement corresponding to each of the third questions; further understood as: write all the statements of the third question and the third answer statement corresponding to each of the third questions into the second text in sequence until no third statement and the third answer statement corresponding to the third question are written, and the generated second text is used as the target dialogue text.
[0072] The above embodiment provides a method for obtaining text using a multimodal large model based on a hierarchical memory network. The method includes: S100, obtaining a first question statement and a first text provided by a target user; S200, inputting the first question statement and the first text into a trained large model to obtain a first answer statement corresponding to the first question statement; S300, obtaining a second question statement and a second text provided by the target user; S400, inputting the second question statement, the first text, and the second text into the trained large model to obtain a second answer statement corresponding to the second question statement; S500, obtaining a set of third question statements provided by the target user; S600, inputting any one of the third question statements, the first text, the second text, and the third text corresponding to the third question statement into the trained large model to obtain a third answer statement corresponding to the third question statement; S700, generating target dialogue text based on all the third question statements and the third answer statement corresponding to each of the third question statements. It can be seen that the following technical effects are achieved:
[0073] 1. This invention constructs a target summary text and continuously incorporates key historical dialogue information (first text, second text, etc.) across pages, providing a structured global context when the model processes subsequent questions. This enables the model to transcend the limitations of single pages or sliding windows, understanding and associating information scattered across multiple parts of a document, fundamentally solving the context fragmentation problem caused by fragmented processing in existing technologies.
[0074] 2. By establishing and dynamically maintaining a memory text library containing historical question-and-answer pairs (such as the second text), and using relevant memories as contextual input, this invention endows the model with stateful and persistent dialogue memory capabilities. This enables the model to accurately understand complex follow-up questions that rely on historical interactions, such as reference, comparison, and progression, achieving truly fluent and in-depth multi-turn continuous dialogue, and enhancing the intelligence and depth of the interaction.
[0075] 3. Users no longer need to manually break down complex problems or repeatedly provide historical background. Through automated context building and memory management, the model can directly handle high-level tasks involving global analysis of long documents and multi-round comparisons, such as cross-chapter content sorting and before-and-after data comparison and demonstration, which greatly reduces the user's burden and improves the efficiency of analysis and discussion.
[0076] 4. This method is based on a general large-scale model architecture, and its core lies in the construction and management strategy of the input context, rather than being bound to a specific domain model. Therefore, it can be widely applied to various scenarios that require deep understanding of multimodal long documents and intelligent dialogue, such as financial report analysis, legal contract review, academic paper reading, and processing of long consulting documents, and has high scalability and practical value.
[0077] While specific embodiments of the invention have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of the invention. It should also be understood that various modifications can be made to the embodiments without departing from the scope and spirit of the invention. The scope of the invention is defined by the appended claims.
Claims
1. A method for acquiring text from a multimodal large model based on a hierarchical memory network, characterized in that, The method includes the following steps: S100, Obtain the first question statement and first text provided by the target user; S200, input the first question statement and the first text into the trained large model to obtain the first answer statement corresponding to the first question statement; S300, obtain the second question statement and second text provided by the target user; S400, input the second question statement, the first text style, and the second text into the trained large model to obtain the second answer statement corresponding to the second question statement; S500, obtain the set of third-party question statements provided by the target user; S600, input any one of the third question statements in the set of third question statements, the first text, the second text and the third text corresponding to the third question statement into the trained large model to obtain the third answer statement corresponding to the third question statement; S700, Generate the target dialogue text based on all the third question statements and the third answer statement corresponding to each of the third question statements.
2. The method for acquiring text based on a multimodal large model using a hierarchical memory network according to claim 1, characterized in that, The first question statement is the first question statement provided by the target user.
3. The method for acquiring text based on a multimodal large model using a hierarchical memory network according to claim 1, characterized in that, Step S100 also includes the following step to obtain the first text: S101, Obtain the target document; S101, Based on the target document, obtain the target summary text corresponding to the target document and use the target summary text as the first text.
4. The method for acquiring text based on a multimodal large model using a hierarchical memory network according to claim 3, characterized in that, The target document is a document whose number of pages exceeds a preset page number threshold.
5. The method for acquiring text based on a multimodal large model using a hierarchical memory network according to claim 3, characterized in that, Step S101 also includes the following step to obtain the target summary text: S1011, Obtain the target document text set A={A1, ..., A2} of the target document. i , ..., A m }, A i It is the text of the i-th target document, where the value of i ranges from 1 to m, and m is the number of target document texts in the target document; S1012, each A i The input is fed into the first encoder to obtain A. i The corresponding first document feature vector B i Among them, B i ={B i1 B i2 B i3 B i4 }, B i1 It is A i The corresponding text content information, B i2 It is A i Corresponding visual layout information, B i3 It is A i The corresponding chart content information, B i4 It is A i Corresponding image semantic information; S1013, Obtain the metadata C of the target document; wherein, the metadata of the target document includes the table of contents of the target document, the chapter titles of the target document, and the page numbers of the target document; S1014, each B i Both combine with C to generate B. i The corresponding second document feature vector B 0 i ={B i1 B i2 B i3 B i4 ,C}; S1015, each B 0 i The text is input into the second encoder to obtain the summary text corresponding to the target document and to use the summary text corresponding to the target document as the target summary text.
6. The method for acquiring text based on a multimodal large model using a hierarchical memory network according to claim 5, characterized in that, The second encoder is an encoder that uses the Transformer architecture.
7. The method for acquiring text based on a multimodal large model using a hierarchical memory network according to claim 1, characterized in that, The second question statement is the second question statement provided by the target user after the first question statement.
8. The method for acquiring text based on a multimodal large model using a hierarchical memory network according to claim 1, characterized in that, In step S300, the second text is the text constructed from the first question statement and the first answer statement corresponding to the first question statement, and the second text is stored in the memory text library.