Intelligent agent context optimization method and device based on multi-modal understanding model

Abstract

The invention provides an agent context optimization method and device based on a multi-modal understanding model, and the method comprises the steps: converting plain text context information inputted into an agent into a visual document image through a visual typesetting method according to the structured content; wherein the structured content is a logic level of plain text context information; analyzing the visual document image through a visual language model to obtain identification information of the visual document image; according to the identification information, reconstructing the visual document image into a structured context representation to obtain an optimized context; outputting the optimized context to an intelligent agent; compared with the prior art which is limited by linear text representation and cannot break through the Token window limitation of the model and semantic detail loss, the method has the advantages that the Token length limitation of a plain text model is broken through by converting the text into the image, and two-dimensional high-density compression of information is realized; through the analysis of the visual language model, the deep understanding of the context structure and semantics is realized.

Description

Technical Field

[0001] This invention relates to the field of intelligent agents, and more particularly to an intelligent agent context optimization method and apparatus based on a multimodal understanding model. Background Technology

[0002] When performing a task, an intelligent agent needs to continuously remember past inputs, actions, results, environmental states, and global goals. This information collectively constitutes the context upon which the model relies for reasoning. However, as the task progresses and the number of interaction rounds accumulates, a series of challenges arise.

[0003] First, there is a limitation to the length of the context. Although current large models can support inputs of hundreds of thousands or even millions of tokens, they may still exceed this limit when handling complex tasks. This leads to the forgetting of historical information outside the window, resulting in problems such as loss of task constraints, interruption of inference, logical errors, or repeated execution. Second, even if all information remains within the context window, the model's attention to earlier information gradually decreases, causing uneven distribution of attention weights and breaking of previously relied-upon logical chains, i.e., context degradation. In addition, long contexts often contain a large amount of irrelevant or outdated information, making it difficult for the model to effectively distinguish between key content and minor details. This makes the model susceptible to information noise, affecting the accuracy and consistency of inference. Finally, as the size of the context increases in each round of input, memory and bandwidth consumption increases linearly, significantly increasing computational and storage costs. This problem is particularly prominent in multi-agent collaborative scenarios.

[0004] Context engineering is a systematic approach and strategy designed to help agents efficiently organize and utilize information within a limited context window, thereby improving the coherence and accuracy of reasoning and decision-making in multi-turn tasks. Its core concept is to optimize the structure, representation, and selection mechanisms of the context, enabling the model to focus on key information and ignore irrelevant content. In the agent's workflow, key techniques include context compression, retrieval enhancement, dynamic context management, external memory mechanisms, and context planning, autonomously scheduling the necessary information through meta-inference. These methods work together to effectively mitigate the forgetting, interference, and efficiency problems caused by long contexts, significantly enhancing the consistency, accuracy, and computational efficiency of agents when performing complex tasks.

[0005] While context engineering is quite mature in text processing, it has some structural limitations. These limitations are particularly evident when dealing with extremely long contexts, complex formats, or heterogeneous information from multiple sources. First, it is limited by linear text representations and struggles to effectively convey the inherent spatial and hierarchical relationships in tables, charts, and other similar content. Second, even after compression, it cannot overcome the model's own token window limitations and may suffer performance degradation due to attention degradation. Furthermore, its core context compression technique inevitably leads to the loss of semantic details, especially when processing technical or structural content. In addition, traditional large language models are insensitive to non-textual structural information such as formatting and layout within the context. Summary of the Invention

[0006] This invention provides an agent context optimization method and apparatus based on a multimodal understanding model, which addresses the shortcomings of existing technologies that are limited by linear text representation, cannot overcome the model's own token window limitation, and suffer from the loss of semantic details. It achieves two-dimensional high-density compression of information and a deep understanding of context structure and semantics.

[0007] This invention provides an agent context optimization method based on a multimodal understanding model, comprising the following steps.

[0008] Using a visual typography method, the plain text context information input to the agent is converted into a visual document image based on structured content; where structured content refers to the logical hierarchy of the plain text context information. Visual language models are used to parse visual document images to obtain their recognition information, which includes the text content, structural information, and semantic relationships of the visual document images. Based on the recognition information, the visual document image is reconstructed into a structured context representation, resulting in an optimized context. The optimized context is output to the agent.

[0009] The agent context optimization method based on a multimodal understanding model provided by the present invention further includes: The visual language model is optimized based on task-related datasets.

[0010] According to the present invention, an agent context optimization method based on a multimodal understanding model optimizes a visual language model based on a task-related dataset, including: By fusing optical character recognition and summarization tasks, a multi-task loss function is constructed; the visual language model is then optimized across modalities using the multi-task loss function to obtain the first optimized model. For the preset task of the intelligent agent, a task-related dataset is collected and constructed; based on the task-related dataset, the first optimized model is optimized to obtain the optimized model; wherein, the task-related dataset is a dataset labeled with key information or summary information of the preset task.

[0011] According to the present invention, an agent context optimization method based on a multimodal understanding model converts plain text context information input to the agent into a visual document image using a visual typography method, based on structured content. The method includes: The plain text context information input to the agent is combined with task instructions and environmental state, and then formatted and organized according to logical hierarchy to obtain structured content. Visual typography methods are used to render structured content to obtain visual document images; these methods include at least one of the following visual techniques: layout, color, indentation, columns, or borders.

[0012] According to the present invention, an agent context optimization method based on a multimodal understanding model is provided, which identifies information, including document content and layout information of visual document images; and parses the visual document images using a visual language model to obtain the identification information of the visual document images, including: The optical character recognition method using a visual language model is used to identify text and positional information in visual document images, thereby obtaining document content and positional information. The optical character recognition method based on visual language models identifies at least one semantic structure among titles, chapters, tables, and code blocks in a visual document image based on document content and location information, thereby obtaining layout information.

[0013] The present invention also provides an agent context optimization device based on a multimodal understanding model, comprising the following modules: The visualization rendering module is used to convert the plain text context information input to the agent into a visual document image based on the structured content using visual typography methods; where the structured content is the logical hierarchy of the plain text context information. The parsing module uses a visual language model to parse visual document images and obtain their recognition information, which includes the text content, structural information, and semantic relationships of the visual document images. The reconstruction module is used to reconstruct the visual document image into a structured context representation based on the recognition information, thus obtaining the optimized context. The output module is used to output the optimized context to the agent.

[0014] According to the present invention, an agent context optimization device based on a multimodal understanding model further includes: The optimization module is used to optimize the visual language model based on task-related datasets.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the agent context optimization method based on the multimodal understanding model as described above.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the agent context optimization method based on a multimodal understanding model as described above.

[0017] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the agent context optimization method based on a multimodal understanding model as described above.

[0018] This invention provides an agent context optimization method, apparatus, device, and storage medium based on a multimodal understanding model. Through a visual typography method, it converts plain text context information input to the agent into a visual document image based on structured content; where the structured content is the logical hierarchy of the plain text context information; through a visual language model, it parses the visual document image to obtain recognition information; where the recognition information includes the text content, structural information, and semantic relationships of the visual document image; based on the recognition information, it reconstructs the visual document image into a structured context representation to obtain the optimized context; and the optimized context is output to the agent. Compared with existing technologies that are limited by linear text representation, unable to overcome the model's own token window limitations and semantic detail loss, this invention, by converting text into an image, overcomes the token length limitation of the plain text model, achieving two-dimensional high-density compression of information; through the parsing of the visual language model, it achieves a deep understanding of the context structure and semantics, overcoming the shortcomings of linear text representation and attention degradation. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the agent context optimization method based on a multimodal understanding model provided by the present invention.

[0021] Figure 2 This is a schematic diagram of the overall process of the agent context optimization method based on a multimodal understanding model, provided in a specific example of the present invention. Figure 3 This is a schematic diagram of the intelligent agent context optimization device based on a multimodal understanding model provided by the present invention.

[0022] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] The following is combined Figures 1-4 This invention is described.

[0025] Figure 1 This is a flowchart illustrating the agent context optimization method based on a multimodal understanding model provided by the present invention, as shown below. Figure 1 As shown, the method includes the following: Step 101: Using a visual layout method, convert the plain text context information input to the agent into a visual document image based on the structured content; where the structured content is the logical hierarchy of the plain text context information.

[0026] In step 101 above, the agent context optimization method based on a multimodal understanding model proposed in this paper forms a new context optimization paradigm through three steps: visualization rendering, OCR (Optical Character Recognition) structure parsing, and VLM (Visual Language Model) fine-tuning. It significantly outperforms traditional pure text context engineering methods in terms of information compression rate, structured representation capability, and model learnability, providing a new scalable path for solving long context problems.

[0027] Optionally, step 101 includes steps A1 to A2: Step A1: The plain text context information input to the agent is combined with task instructions and environmental state, and then formatted and organized according to logical hierarchy to obtain structured content.

[0028] Step A2: Render the structured content using visual typography methods to obtain a visual document image; wherein, visual typography methods include at least one of the following visual means: layout, color, indentation, columns, or borders.

[0029] In steps A1 to A2 above, context visualization rendering transforms the contextual information, originally existing in plain text form, into a structured visual representation, enabling efficient encoding and compression in two-dimensional space. The main processes include: information formatting and layout rendering.

[0030] Information formatting organizes contextual information such as the agent's dialogue history, task instructions, environment status, and API call results into a visual document structure according to logical hierarchy.

[0031] Layout rendering transforms structured text, tables, code snippets, process descriptions, and other content into formatted document images with clear hierarchical relationships. Rendering can employ techniques such as typography, color, indentation, columns, and borders to visually represent the logical relationships within the document.

[0032] Steps A1 and A2 refine the specific methods of visualization rendering, clarifying the source of input information and specific rendering techniques. The beneficial effect is that it ensures the converted visual document accurately and efficiently carries the logical relationships of the original context, laying the foundation for subsequent precise analysis.

[0033] Step 102: Analyze the visual document image using a visual language model to obtain the recognition information of the visual document image; wherein, the recognition information includes the text content, structural information and semantic relationships of the visual document image.

[0034] In step 102 above, the rendered document image is understood and parsed using a Visual Language Model (VLM). The VLM not only has a visual encoder for image feature extraction but also incorporates a language decoder or cross-modal attention structure to achieve text-level and semantic-level understanding of the image content.

[0035] Optionally, step 102 includes steps B1 to B2: Step B1: Using the optical character recognition method of visual language model, identify the text and position information in the visual document image to obtain the document content and position information.

[0036] Step B2: Using the optical character recognition method of visual language model, based on the document content and location information, identify at least one semantic structure among the title, chapter, table, and code block in the visual document image to obtain layout information.

[0037] In steps B1 to B2 above, VLM automatically recognizes the text content in the document, achieving optical character recognition while preserving the text's positional information. The visual language model identifies semantic levels from the document layout, such as structures like headings, chapters, tables, comments, and code blocks.

[0038] Steps B1 and B2 clarify the specific tasks of visual language model parsing, namely OCR and structural understanding. Its beneficial effect lies in the fact that by combining character recognition and structural analysis, it can accurately capture contextual relationships that are difficult to express in traditional text sequences, achieving true understanding of the context, rather than simply reading it.

[0039] Step 103: Based on the recognition information, reconstruct the visual document image into a structured context representation to obtain the optimized context.

[0040] In step 103 above, the image information is reconstructed into a structured contextual representation based on the identified content and positional relationship.

[0041] Step 104: Output the optimized context to the agent.

[0042] In step 104 above, the plain text context information includes historical information generated by the agent during multiple rounds of interaction. The machine-readable structured context representation or the refined key information / summary is input into the agent model to support its subsequent reasoning, planning, or decision-making.

[0043] This invention clarifies the application of the method's inputs and outputs in practical intelligent agent systems. Its beneficial effect lies in clearly defining the role and position of this technical solution within the entire artificial intelligence system, emphasizing that its ultimate goal is to improve the performance of the intelligent agent's core tasks: reasoning, planning, and decision-making.

[0044] Optionally, the agent context optimization method based on a multimodal understanding model provided by the present invention further includes step C1: Step C1: Optimize the visual language model based on the task-related dataset.

[0045] In step C1 above, the VLM learns how to understand and summarize context through learning, elevating the model from a perceptual system to an agent-level component with reasoning and extraction capabilities. The visual language model is fine-tuned based on task-related datasets to optimize its ability to extract key information or generate summaries from visual document images. This fine-tuning mechanism endows the model with adaptive, task-oriented context extraction capabilities, rather than relying on fixed rules.

[0046] Optionally, step C1 includes steps C11 to C12: Step C11: Construct a multi-task loss function by fusing the optical character recognition task and the summary generation task; optimize the visual language model across modalities using the multi-task loss function to obtain the first optimized model.

[0047] Step C12: For the agent's preset task, collect and construct a task-related dataset; based on the task-related dataset, optimize the first optimized model to obtain the optimized model; wherein, the task-related dataset is a dataset labeled with key information or summary information of the preset task.

[0048] In steps C11 to C12 above, in the visual and linguistic feature spaces, multi-task loss, mainly including OCR and summarization generation, is used to enhance the model's cross-modal semantic consistency. OCR is responsible for maintaining the VLM's accuracy in recognizing rendered content. Summarization generation is responsible for fine-tuning the VLM using labeled document image-context summaries or document image-key information extraction data, enabling the model to learn to extract core content from complex visual contexts.

[0049] For specific agent tasks, such as dialogue continuation, tool call planning, and knowledge generation, the model is further optimized so that its output can be directly used in subsequent reasoning stages.

[0050] The visual language model is fine-tuned using a multi-task loss function that includes optical character recognition and summary generation tasks to enhance the semantic consistency between its visual and linguistic features.

[0051] Multi-task loss ensures that the model does not lose its fundamental recognition capabilities during learning and refinement, guaranteeing the robustness of the technology. Task-oriented reinforcement enables the technical solution to flexibly adapt to different intelligent agent application scenarios, such as dialogue, planning, and knowledge generation, improving the practicality and generalization ability of the solution.

[0052] This technical solution proposes a multimodal understanding model to optimize context processing during agent execution. By representing the spatial density of images, a large amount of contextual information can be condensed into one or a few images, achieving a compression rate far exceeding that of plain text tokens. Through joint modeling of OCR and visual semantics, the context is no longer limited to a one-dimensional token sequence but is presented as a structured spatial relationship. The model not only sees the text but also understands its semantic hierarchy and logical relationships. The model can automatically extract key semantics from the rendered context, generate structured summaries, or provide controllable contextual representations for the agent, thereby achieving adaptive long context management and optimization.

[0053] The present invention provides the following specific examples to further explain the invention.

[0054] like Figure 2As shown, this invention comprises three main stages: a context visualization rendering stage, a visual semantic parsing and structured reconstruction stage, and a context extraction and fine-tuning optimization stage. The first stage, context visualization rendering, transforms the contextual information of plain text into a structured visual representation. The second stage, visual semantic parsing and structured reconstruction, involves a visual language model performing in-depth analysis of the document image generated in the previous stage. The third stage is context extraction and fine-tuning optimization. The overall method overcomes the length and structural limitations of plain text models by rendering the text context into a structured visual document and using a visual language model for parsing and extraction. Finally, targeted fine-tuning enables the model to possess efficient task-oriented context understanding and summarization capabilities.

[0055] The context visualization rendering stage includes: firstly, organizing the agent's dialogue history, task instructions, environmental status, and other information in a logical hierarchy; then, using visual techniques such as layout, color, indentation, and columns, rendering it into a well-formatted and clearly hierarchical document image, thereby achieving efficient encoding and compression of information in a two-dimensional space.

[0056] The visual semantic parsing and structured reconstruction stage includes: First, the model accurately identifies all text and its location in the image using OCR technology, and then understands the semantic structure of the document, such as titles, chapters, tables, etc. Finally, the model integrates content and layout information to reconstruct the visual document into a machine-readable structured contextual representation.

[0057] The context extraction and fine-tuning optimization phase includes the following: The goal is to enable the visual language model to learn how to proactively extract the essence of context. Through cross-modal alignment optimization, the model learns to generate high-quality summaries or extract key information from complex visual contexts while ensuring accurate OCR recognition. Furthermore, through task-oriented reinforcement, the model is further fine-tuned so that its output can directly support the agent's subsequent advanced tasks such as reasoning, planning, and decision-making, thus becoming a core component of an agent with greater understanding and extraction capabilities.

[0058] This invention has the following significant advantages over current agent context methods.

[0059] Transforming text-to-visual context representation: This involves rendering raw text context, such as conversation history, logs, documents, and code, into structured, formatted document images, achieving high-density information expression through visual typography and spatial layout. This process leverages the two-dimensional characteristics of visual space, enabling context to be represented with a higher compression rate, thus overcoming the limitations of text input length. It converts traditional text sequence input into visual document input, achieving semantically dense representation through spatial layout and formatting information. This two-dimensional representation naturally contains hierarchical and relational information, significantly improving the structure and compression rate of contextual expression.

[0060] This system integrates visual structure understanding and text recognition. Leveraging the OCR and visual perception capabilities of the Visual Language Model (VLM), it simultaneously processes text and layout information within documents, achieving structured reconstruction of context and capture of semantic relationships. This enables the model to understand the context, not just read it. The rendered images are processed using a visual language model, utilizing the model's OCR and visual understanding capabilities to reconstruct the image content into structured semantic information. By recognizing document structure, paragraph hierarchy, and table layout, the model can capture contextual relationships that are difficult to express with traditional text sequences, thus overcoming the limitations of linear text representation.

[0061] A fine-tunable learning mechanism for contextual understanding. By fine-tuning the Visual Learning Model (VLM), it acquires the ability to extract and summarize context for specific tasks, achieving a shift from general visual understanding to task-oriented context optimization, thereby continuously improving context processing performance. Task-oriented fine-tuning based on the VLM enables the model not only to recognize visual context but also to learn how to extract key content, generate semantic summaries, or contextual representations required for the task. This step gives the system the learnable ability to "understand and compress context," rather than relying solely on fixed rules or prompt strategies.

[0062] This invention provides an agent context optimization method based on a multimodal understanding model. Through a visual typography method, the plain text context information input to the agent is converted into a visual document image based on structured content. The structured content refers to the logical hierarchy of the plain text context information. A visual language model is used to parse the visual document image, obtaining its recognition information, which includes the text content, structural information, and semantic relationships of the visual document image. The optimized context is then output to the agent. Based on the recognition information, the visual document image is reconstructed into a structured context representation, resulting in the optimized context. Compared to existing technologies that are limited by linear text representation, unable to overcome the model's own token window limitations and suffer from semantic detail loss, this method, by converting text into an image, overcomes the token length limitation of the plain text model, achieving two-dimensional high-density compression of information. Through parsing using a visual language model, a deep understanding of the context structure and semantics is achieved, overcoming the shortcomings of linear text representation and attention degradation.

[0063] The following describes the agent context optimization device based on a multimodal understanding model provided by the present invention. The agent context optimization device based on a multimodal understanding model described below and the agent context optimization method based on a multimodal understanding model described above can be referred to in correspondence.

[0064] Figure 3 This is a flowchart illustrating the agent context optimization device based on a multimodal understanding model provided by the present invention, as shown below. Figure 3 As shown, the device includes the following: The visualization rendering module 301 is used to convert the plain text context information input to the agent into a visual document image based on the structured content using a visual typography method; wherein, the structured content is the logical hierarchy of the plain text context information.

[0065] The parsing module 302 is used to parse the visual document image through a visual language model to obtain the recognition information of the visual document image; wherein, the recognition information includes the text content, structural information and semantic relationships of the visual document image.

[0066] The reconstruction module 303 is used to reconstruct the visual document image into a structured context representation based on the recognition information, thereby obtaining the optimized context.

[0067] Output module 304 is used to output the optimized context to the agent.

[0068] Optionally, the agent context optimization device based on a multimodal understanding model provided by the present invention further includes: The optimization module is used to optimize the visual language model based on task-related datasets.

[0069] This invention provides an agent context optimization device based on a multimodal understanding model. Using a visual typography method, it converts plain text context information input to the agent into a visual document image based on structured content. The structured content represents the logical hierarchy of the plain text context information. A visual language model is used to parse the visual document image, obtaining its recognition information, which includes the text content, structural information, and semantic relationships of the visual document image. The optimized context is then output to the agent. Based on the recognition information, the visual document image is reconstructed into a structured context representation, resulting in the optimized context. Compared to existing technologies limited by linear text representation, which cannot overcome the model's own token window limitations and suffer from semantic detail loss, this invention overcomes the token length limitation of plain text models by converting text into images, achieving two-dimensional high-density compression of information. Through the parsing of the visual language model, a deep understanding of the context structure and semantics is achieved, overcoming the shortcomings of linear text representation and attention degradation.

[0070] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840. The processor 810, communications interface 820, and memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions from the memory 830 to execute an agent context optimization method based on a multimodal understanding model.

[0071] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0072] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the agent context optimization method based on the multimodal understanding model provided by the above methods.

[0073] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the agent context optimization method based on the multimodal understanding model provided by the above methods.

[0074] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0075] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An agent context optimization method based on a multi-modal understanding model, characterized in that, include: Using a visual typography method, the plain text context information input to the agent is converted into a visual document image based on structured content; wherein, the structured content is the logical hierarchy of the plain text context information; The visual document image is parsed using a visual language model to obtain the recognition information of the visual document image, wherein the recognition information includes the text content, structural information, and semantic relationships of the visual document image. Based on the recognition information, the visual document image is reconstructed into a structured context representation to obtain the optimized context; The optimized context is output to the agent.

2. The multi-modal understanding model based agent context optimization method according to claim 1, characterized in that, The method further includes: The visual language model is optimized based on task-related datasets.

3. The multi-modal understanding model based agent context optimization method according to claim 2, characterized in that, The optimization of the visual language model based on the task-related dataset includes: By fusing optical character recognition and summarization tasks, a multi-task loss function is constructed; the visual language model is then optimized for cross-modal alignment using the multi-task loss function to obtain a first optimized model. For a preset task of the intelligent agent, a task-related dataset is collected and constructed; based on the task-related dataset, the first optimized model is optimized to obtain an optimized model; wherein, the task-related dataset is a dataset labeled with key information or summary information of the preset task.

4. The agent context optimization method based on a multimodal understanding model according to claim 1, characterized in that, The method of converting plain text context information input to the agent into a visual document image based on structured content through visual typography includes: The plain text context information input to the agent is combined with task instructions and environmental state, and then formatted and organized according to logical hierarchy to obtain structured content. The structured content is rendered using a visual layout method to obtain a visual document image; wherein the visual layout method includes at least one visual means selected from layout, color, indentation, columns, or borders.

5. The agent context optimization method based on a multimodal understanding model according to claim 1, characterized in that, The identification information includes document content and layout information of the visual document image; the step of parsing the visual document image using a visual language model to obtain the identification information of the visual document image includes: The optical character recognition method using a visual language model is used to identify the text and position information in the visual document image, thereby obtaining the document content and position information; Using an optical character recognition method based on a visual language model, layout information is obtained by recognizing at least one semantic structure among titles, chapters, tables, and code blocks in the visual document image based on the document content and location information.

6. An agent context optimization device based on a multimodal understanding model, characterized in that, include: The visualization rendering module is used to convert the plain text context information input to the agent into a visual document image based on structured content using a visual typography method; wherein, the structured content is the logical hierarchy of the plain text context information; The parsing module parses the visual document image using a visual language model to obtain the recognition information of the visual document image; wherein, the recognition information includes the text content, structural information, and semantic relationships of the visual document image; The reconstruction module is used to reconstruct the visual document image into a structured context representation based on the recognition information, thereby obtaining an optimized context. The output module is used to output the optimized context to the agent.

7. The agent context optimization device based on a multimodal understanding model according to claim 6, characterized in that, The device further includes: The optimization module is used to optimize the visual language model based on the task-related dataset.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the agent context optimization method based on a multimodal understanding model as described in any one of claims 1 to 5.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the agent context optimization method based on the multimodal understanding model as described in any one of claims 1 to 5.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the agent context optimization method based on the multimodal understanding model as described in any one of claims 1 to 5.

Images