Multi-agent collaborative multimodal reasoning methods, devices, electronic devices and products

Abstract

This application relates to the field of artificial intelligence technology, proposing a multi-agent collaborative multimodal reasoning method, device, electronic device, and computer program product. The method establishes three agents: a text reasoning agent, a visual understanding agent, and a judgment agent. The text reasoning agent leads the overall textual logical reasoning, the visual understanding agent provides visual semantic information, and the judgment agent evaluates whether the multimodal reasoning process requires additional visual semantic information. If so, it triggers a visual information supplementation process, guiding the visual understanding agent to extract the necessary supplementary visual semantic information. By utilizing the collaborative work of these three agents, a closed-loop mechanism of multimodal reasoning, supplementary judgment, and information supplementation is formed. Without the need for complex preprocessing, it enables deep semantic understanding and logical reasoning of multimodal documents, thereby achieving accurate and complete document understanding and response text generation.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a multi-agent collaborative multimodal reasoning method, apparatus, electronic device, and computer program product. Background Technology

[0002] Multimodal documents, as one of the most common forms of knowledge carriers in daily life, integrate at least two of various information formats, including text, images, tables, formulas, audio, and video, and are widely found in practical scenarios such as manuals, reports, and papers. With the rise of document-based knowledge base products, users' demands for structured understanding and intelligent question answering of multimodal documents are increasing. Current technologies for processing multimodal documents typically require complex preprocessing steps such as OCR recognition and format conversion. These processes are not only cumbersome and error-prone, but also prone to semantic loss or information misalignment, affecting the semantic understanding of multimodal documents and ultimately leading to a decrease in the accuracy and completeness of the response text. Summary of the Invention

[0003] In view of this, embodiments of this application provide a multi-agent collaborative multimodal reasoning method, apparatus, electronic device, and computer program product. By utilizing the collaborative work of multiple agents, deep semantic understanding and textual logical reasoning of multimodal documents can be performed without the need for complex preprocessing procedures, thereby achieving accurate and complete document understanding and response text generation.

[0004] The first aspect of this application provides a multi-agent collaborative multimodal reasoning method, including:

[0005] Retrieve user query text related to multimodal documents;

[0006] The text reasoning agent performs multimodal reasoning based on the visual semantic information of the user's query text and the generated multimodal documents, and generates reasoning text.

[0007] The judgment agent determines whether the multimodal reasoning process lacks supporting information based on the reasoning text. If the multimodal reasoning process lacks supporting information, it generates visual information supplementation instructions and sends the visual information supplementation instructions to the visual understanding agent.

[0008] The visual understanding agent performs visual semantic understanding on the target page image of the multimodal document under the guidance of visual information supplementation instructions, updates the visual semantic information, and then returns to execute the text reasoning agent, which performs multimodal reasoning based on the user query text and the visual semantic information of the generated multimodal document to generate thought reasoning text.

[0009] The technical solution of this application embodiment sets up a text reasoning agent, a visual understanding agent, and a judgment agent. The text reasoning agent leads the overall text logic reasoning, the visual understanding agent is responsible for providing visual semantic information, and the judgment agent is responsible for evaluating whether the multimodal reasoning process needs to be supplemented with additional visual semantic information. If supplementation is required, the visual information supplementation process is triggered to guide the visual understanding agent to extract the required supplementary visual semantic information. After acquiring the user query text related to the multimodal document, the text reasoning agent first performs multimodal reasoning based on the user query text and the visual semantic information of the generated multimodal document, generating a thought-based reasoning text. Then, the judgment agent determines whether the multimodal reasoning process lacks supporting information based on the thought-based reasoning text. If so, it generates a visual information supplementation instruction and sends it to the visual understanding agent. Guided by the visual information supplementation instruction, the visual understanding agent performs visual semantic understanding on the target page image of the multimodal document, supplementing and updating the visual semantic information. The text reasoning agent then continues multimodal reasoning based on the updated visual semantic information, thus forming a closed-loop mechanism of multimodal reasoning, supplementation judgment, and information supplementation. It can be seen that as the number of information supplementation attempts increases, the visual semantic information required for multimodal reasoning by the text reasoning agent becomes richer and more complete, thereby continuously improving the overall reasoning effect and ultimately obtaining an accurate and complete response text. The above process utilizes the collaborative work of multiple intelligent agents to perform deep semantic understanding and logical reasoning on multimodal documents without the need for complex preprocessing procedures, thereby achieving accurate and complete document understanding and response text generation.

[0010] In one implementation of this application, the target page image is determined in the following way:

[0011] Obtain the original page images of each page in the multimodal document;

[0012] From each original page image, select the page image containing the required visual information as the target page image; wherein, the required visual information is determined according to the visual information supplementation instruction.

[0013] In one implementation of this application, selecting a page image containing the required visual information from various original page images as the target page image includes:

[0014] The user query text and each original page image are embedded using a document encoding model, and the similarity score between each original page image and the user query text is calculated.

[0015] Based on the similarity score between each original page image and the user's query text, relevant page images are selected from each original page image.

[0016] Using a visual language model, the relevance score between each relevant page image and the user's query text is calculated.

[0017] From each relevant page image, select the page images whose relevance scores are higher than the first threshold as candidate page images;

[0018] From the candidate page images, select the page image containing the required visual information as the target page image.

[0019] In one implementation of this application, relevant page images are selected from the original page images based on the similarity score between each original page image and the user query text, including:

[0020] The similarity scores of each original page image and the user query text are arranged from high to low to obtain a descending similarity sequence, and the descending similarity sequence is fitted to a similarity decrease curve;

[0021] Construct a target straight line connecting the two endpoints of the similarity decrease curve;

[0022] Based on the perpendicular distance between each point in the similarity decline curve and the target line, the turning point where the similarity score drops sharply can be determined from the similarity decline curve.

[0023] From each original page image, select the page image whose similarity score is higher than the similarity score corresponding to the inflection point, and use them as the relevant page images.

[0024] In one implementation of this application, each candidate page image is added to a candidate page list; from each candidate page image, a page image containing the required visual information is selected as the target page image, including:

[0025] If the first page image containing the required visual information is found in the cached page list, then the first page image is determined as the target page image;

[0026] If the first page image cannot be found in the cached page list, but a second page image containing the required visual information is found in the candidate page list, then the second page image is determined as the target page image and added to the cached page list.

[0027] In one implementation of this application, the method further includes:

[0028] If a second page image cannot be found from the candidate page list, the similarity score and relevance score corresponding to each original page image are recalculated to update the candidate page list, provided that the user query text is replaced with a visual information supplementation instruction.

[0029] In one implementation of this application, determining whether the multimodal reasoning process lacks supporting information based on the reasoning text includes:

[0030] The semantic completeness of the reasoning text is scored according to the built-in scoring rules of the judgment agent.

[0031] If the score is below the second threshold, it is determined that the multimodal reasoning process lacks supporting information.

[0032] A second aspect of this application provides a multi-agent collaborative multimodal reasoning device, comprising:

[0033] The query text retrieval module is used to retrieve user query text related to multimodal documents;

[0034] The multimodal reasoning module is used to perform multimodal reasoning based on the visual semantic information of the user query text and the generated multimodal documents through a text reasoning agent, and generate thought reasoning text.

[0035] The visual supplementation guidance module is used by a judgment-type intelligent agent to determine whether the multimodal reasoning process lacks supporting information based on the thinking and reasoning text. If the multimodal reasoning process lacks supporting information, it generates a visual information supplementation instruction and sends the visual information supplementation instruction to the visual understanding intelligent agent.

[0036] The visual semantic understanding module is used to perform visual semantic understanding on the target page image of a multimodal document under the guidance of visual information supplementation instructions through a visual understanding agent, update the visual semantic information, and then return to execute the steps of the text reasoning agent to perform multimodal reasoning based on the user query text and the visual semantic information of the generated multimodal document, and generate thinking reasoning text.

[0037] A third aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the multi-agent collaborative multimodal reasoning method provided in the first aspect of this application.

[0038] A fourth aspect of this application provides a computer program product that, when run on an electronic device, causes the electronic device to execute the multi-agent collaborative multimodal reasoning method provided in the first aspect of this application.

[0039] A fifth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multi-agent collaborative multimodal reasoning method provided in the first aspect of this application.

[0040] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0041] Figure 1 This is a flowchart of a multi-agent collaborative multimodal reasoning method provided in an embodiment of this application;

[0042] Figure 2 This is a schematic diagram illustrating the operational principle of the multi-agent collaborative multimodal reasoning method provided in this application embodiment in a practical application scenario;

[0043] Figure 3 This is a schematic diagram of the structure of a multi-agent collaborative multimodal reasoning device provided in an embodiment of this application;

[0044] Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0045] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail. Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0046] With the rise of document-based knowledge base products, users' demands for structured understanding and intelligent question answering of multimodal documents are increasing. Existing technologies for processing multimodal documents typically employ a method combining document preprocessing with modal understanding. This method treats multimodal documents as a combination of multiple modalities of data. First, preprocessing processes such as image-text parsing and format conversion break the document down into data sets of independent modalities such as text and images. Then, each modal information is modeled and understood separately. However, the above preprocessing process is cumbersome, complex, and error-prone. Furthermore, the overall semantic structure of the document is easily lost during modal segmentation, such as neglecting visual contextual information like layout and formatting, which affects the semantic understanding of multimodal documents. In addition, in intelligent question answering scenarios, the large language models used have context length limitations. In practical applications, it is usually necessary to segment the document content into multiple fragments and retrieve the most relevant fragments during question answering. This process relies on pure text retrieval strategies, making it difficult to effectively integrate visual semantic information, ultimately leading to a decrease in the accuracy and completeness of the response text.

[0047] To address the aforementioned issues, this application proposes a multi-agent collaborative multimodal reasoning method, apparatus, electronic device, and computer program product. By leveraging the collaborative work of multiple agents, it enables deep semantic understanding and textual logical reasoning of multimodal documents without requiring complex preprocessing procedures, thereby achieving accurate and complete document understanding and response text generation. For more specific technical implementation details of this application's embodiments, please refer to the various method embodiments described below.

[0048] It should be understood that the implementing entity of the various method embodiments proposed in this application can be various types of electronic devices, such as mobile phones, tablets, desktop computers, wearable devices, medical devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), large-screen TVs, etc. The embodiments of this application do not impose any restrictions on the specific type of electronic device.

[0049] Please see Figure 1 This application illustrates a multi-agent collaborative multimodal reasoning method provided by an embodiment of the present application, comprising:

[0050] 101. Retrieve user query text related to multimodal documents;

[0051] The technical solution of this application primarily addresses intelligent question-answering scenarios for multimodal documents. By inputting multimodal documents and user query text into an intelligent agent or large language model for analysis, it can return corresponding response text. Here, an intelligent agent refers to a computer program with autonomy, learning ability, and reasoning ability; it is an intelligent entity capable of perceiving environmental information, making decisions, and executing actions. A multimodal document can be any type and topic containing at least two modalities of information, such as a PDF document of a research paper, a PPT document of a survey report, etc. The user query text is a question description text input by the user related to the multimodal document. For example, assuming the multimodal document is a paper on improving the marine environment, the user query text could be "Based on this paper, analyze how to improve the marine environment" or "Based on this paper, determine which factors are related to the marine environment," etc.

[0052] 102. Through a text-based reasoning agent, multimodal reasoning is performed based on the visual semantic information of the user's query text and the generated multimodal documents to generate reasoning text.

[0053] The technical solution of this application embodiment sets up three intelligent agents: a visual understanding agent, a text reasoning agent, and a judgment agent. The visual understanding agent is responsible for providing visual semantic information of the multimodal document; the text reasoning agent leads the overall text logic reasoning; and the judgment agent is responsible for evaluating whether the multimodal reasoning process needs additional visual semantic information. If so, it triggers a visual information supplementation process, guiding the visual understanding agent to extract the required supplementary visual semantic information. In summary, the text reasoning agent leads the overall multimodal reasoning process, the judgment agent evaluates the completeness of the reasoning, and the visual understanding agent provides key visual information support; the three agents work together to complete complex intelligent question-answering tasks. The specific technical implementation details of the collaborative work of the three agents are described below.

[0054] The text reasoning agent is responsible for constructing and executing the overall text reasoning chain. It needs strong text logic reasoning capabilities and typically employs high-performance reasoning models such as Deepseek-R1 and QWQ, which offer advantages in stability, reasoning depth, and controllability. Upon receiving the user's query text, the text reasoning agent begins constructing a text reasoning path. It performs multimodal reasoning based on the user's query text and the visual semantic information of the currently generated multimodal document, generating reasoned text. Multimodal reasoning refers to the process of combining visual semantic information with text reasoning. The text reasoning agent can employ a stepped distribution of reasoning steps, executing one logically divided reasoning step at a time, gradually building the complete text reasoning path. Upon completion of each reasoning step, the text reasoning agent generates corresponding reasoned text, which is the intermediate text generated by the agent during each step of analysis and thinking. For example, when a text reasoning agent completes reasoning step 1, it generates the corresponding thought reasoning text. When it completes reasoning step 2, it generates the corresponding thought reasoning text, and so on, until all reasoning steps are completed, resulting in the final thought reasoning text. Finally, based on this final thought reasoning text, it generates and outputs the response text corresponding to the user's query. Furthermore, during multimodal reasoning, if the text reasoning agent detects missing key supporting information or encounters reasoning obstacles, it pauses the current reasoning process and activates the judgment agent. The judgment agent guides the visual understanding agent to re-parse the relevant content in the multimodal document's page images, supplementing the necessary visual semantic information. Then, the text reasoning agent continues with the next step of multimodal reasoning based on the supplemented visual semantic information. This cycle continues until the entire multimodal reasoning process is completed.

[0055] The visual semantic information of a multimodal document is extracted by a visual understanding agent from the page images of the multimodal document. In the initial stage of multimodal reasoning, the generated multimodal document's visual semantic information may be empty. At this point, the text reasoning agent only performs reasoning based on the user's query text, easily encountering the problem of missing key supporting information. However, through multiple cycles of multimodal reasoning, supplementary judgments, and information supplementation, the generated multimodal document's visual semantic information becomes increasingly rich and complete, and the multimodal reasoning process becomes smoother. Alternatively, before proceeding with the multimodal reasoning process, a visual understanding agent can extract some key visual semantic information from the page images of the multimodal document. This allows the text reasoning agent to combine the user's query text and key visual semantic information for multimodal reasoning in the initial stage, thereby reducing the number of interruptions in the multimodal reasoning process and improving reasoning efficiency.

[0056] 103. Through the judgment-type intelligent agent, determine whether the multimodal reasoning process lacks supporting information based on the thinking and reasoning text, and generate visual information supplementation instructions when the multimodal reasoning process lacks supporting information, and send the visual information supplementation instructions to the visual understanding intelligent agent.

[0057] The judgment agent acts as a coordination hub, monitoring the multimodal reasoning process of the text reasoning agent in real time. When it detects missing supporting information or reasoning obstacles, it triggers a visual information supplementation process, guiding the visual understanding agent to extract the necessary supplementary visual semantic information. Specifically, the judgment agent can determine whether the next reasoning step lacks supporting information and whether additional visual content needs to be supplemented from the page images of the multimodal document, based on the thought-based reasoning text generated by the text reasoning agent in the current reasoning step. If it is determined that the next reasoning step lacks supporting information, the multimodal reasoning process is interrupted, and the visual information supplementation process is triggered. After the visual semantic information is supplemented, the text reasoning agent resumes reasoning from the point of interruption, i.e., continues executing the next reasoning step. Conversely, if it is determined that the next reasoning step does not lack supporting information, the visual information supplementation process does not need to be triggered, and the text reasoning agent can directly execute the next reasoning step.

[0058] In one implementation of this application, determining whether the multimodal reasoning process lacks supporting information based on the reasoning text includes:

[0059] (1) The semantic completeness of the reasoning text is scored according to the built-in scoring rules of the judgment agent;

[0060] (2) If the score is lower than the second threshold, it is determined that the multimodal reasoning process lacks supporting information.

[0061] The judgment agent incorporates a scoring rule to evaluate the semantic completeness of the reasoning text output by the text reasoning agent. If the score is below a set second threshold, the semantic completeness of the reasoning text is deemed insufficient, indicating a lack of supporting information in the multimodal reasoning process. In this case, the conditions for continuing multimodal reasoning are not met, and the judgment agent triggers a visual information supplementation process. Conversely, if the score is above the second threshold, the semantic completeness of the reasoning text is deemed sufficient, indicating that the multimodal reasoning process is not lacking supporting information. Therefore, the conditions for continuing multimodal reasoning are met, and a visual information supplementation process is not necessary. In practice, the judgment agent can utilize Vision Language Models (VLMs) to score the semantic completeness of the reasoning text. By setting the maximum number of lexical units in the model's output, it can be guided to output a score between 0 and 9. By introducing a scoring mechanism for reasoning text, the judgment agent can more accurately identify whether the multimodal reasoning process lacks supporting information, thus triggering the visual information supplementation process more precisely.

[0062] Once the visual information supplementation process begins, the judgment agent can generate a descriptive instruction for the required visual information, denoted as a visual information supplementation instruction, based on the current reasoning path. This instruction is then sent to the visual understanding agent to guide it in locating and extracting relevant visual content from the page images of the multimodal document. For example, if the text reasoning process involves "table data citations" or "formula interpretations," the judgment agent determines that the required visual information is table area information or numerical symbol information. It then generates a visual information supplementation instruction, "Please pay attention to the table area data or numerical symbol data present on the document page," and sends it to the visual understanding agent. This guides the agent to focus on the table area or specific mathematical symbol area in the document page image and extract the corresponding visual features or local image information as supplementary visual semantic information for use by the text reasoning agent.

[0063] 104. Through visual understanding, the intelligent agent performs visual semantic understanding on the target page image of the multimodal document under the guidance of visual information supplementation instructions, and updates the visual semantic information.

[0064] The visual understanding agent is used to parse the document image content, providing necessary visual semantic information support for the multimodal reasoning process, ensuring that the response text is accurate and verifiable. In the visual information supplementation process, the visual understanding agent receives a visual information supplementation instruction from the judgment agent. Guided by this instruction, it performs visual semantic understanding on the target page image of the multimodal document, extracts the required visual content, and supplements and updates the generated visual semantic information. Then, it returns to step 102, where the text reasoning agent continues multimodal reasoning based on the updated visual semantic information. This forms a closed-loop mechanism of multimodal reasoning, supplementary judgment, and information supplementation until the entire multimodal reasoning process is completed, obtaining the final thought-reasoning text. Finally, the corresponding response text is generated and output based on this final thought-reasoning text. Clearly, as the number of information supplementation iterations increases, the visual semantic information required for multimodal reasoning by the text reasoning agent becomes richer and more complete, thereby continuously improving the overall reasoning effect and ultimately obtaining an accurate and complete response text.

[0065] The target page image can be one or more page images contained in a multimodal document. Since it's difficult to determine a priori which document pages are truly helpful in answering the question during actual multimodal inference, one approach is to parse all or most of the page images from the multimodal document as target page images to improve recall. However, since multimodal documents may contain hundreds of page images, many of which may be irrelevant to the user's query text, this approach obviously introduces a large amount of irrelevant visual information, affecting the efficiency and accuracy of multimodal inference.

[0066] In view of this, this application proposes a multi-granularity visual retrieval strategy that integrates two complementary page image retrieval mechanisms. This strategy can quickly locate and filter the page images related to the user's query text from a large number of document page images. These page images are more likely to contain the visual information required in the visual information supplementation process. Therefore, using these page images as target page images can reduce the parsing of irrelevant visual information and effectively improve the efficiency and accuracy of multimodal reasoning.

[0067] In one implementation of this application, the target page image is determined in the following way:

[0068] (1) Obtain the original page images of each page of the multimodal document;

[0069] (2) Select the page image containing the required visual information from each original page image as the target page image; wherein the required visual information is determined according to the visual information supplementation instruction.

[0070] In the visual information supplementation process, based on the visual information supplementation instructions sent by the judgment agent, the required visual information can be determined. Then, from all the original page images of the multimodal document, the page image containing the required visual information is selected as the target page image. Since there are a large number of original page images, using a page-by-page parsing method may consume a significant amount of time to locate the valid page, and may also introduce a large amount of irrelevant visual information that interferes with the multimodal reasoning process. To avoid this, the two page image retrieval mechanisms provided in this application embodiment can be used to perform two-stage filtering on all the original page images of the multimodal document. The final filtered page images have a higher relevance to the user's query text. For specific implementation details of these two page image retrieval mechanisms, please refer to the following text.

[0071] In one implementation of this application, selecting a page image containing the required visual information from various original page images as the target page image includes:

[0072] (1) The user query text and each original page image are embedded and represented by the document encoding model, and the similarity score between each original page image and the user query text is calculated respectively;

[0073] (2) Select relevant page images from each original page image based on the similarity score between each original page image and the user query text;

[0074] (3) Using a visual language model, calculate the relevance score between each relevant page image and the user's query text;

[0075] (4) Select the page images with a relevance score higher than the first threshold from each relevant page image as each candidate page image;

[0076] (5) Select the page image containing the required visual information from each candidate page image as the target page image.

[0077] The first page image retrieval mechanism uses a clustering algorithm to quickly match a batch of page images semantically similar to the user's query text from the original page images of a multimodal document, achieving coarse-grained visual information filtering. Specifically, efficient document encoding models such as ColPali can be used to embed the user's query text and each original page image, calculating the similarity score between each original page image and the user's query text. Then, based on the similarity score between each original page image and the user's query text, page images that are relatively similar to the user's query text are selected from the original page images. These page images usually have a high similarity score with the user's query text and can be recorded as relevant page images. These relevant page images, as the output of the first page image retrieval mechanism, can be added to the relevant page list.

[0078] The second page image retrieval mechanism utilizes the logits distribution output by a large model to evaluate the semantic relevance between relevant page images and the user's query text, achieving fine-grained visual information filtering and ultimately outputting several page images with the highest potential relevance to the user's query text. Specifically, an adaptive fine-grained filtering mechanism based on a visual language model can be introduced. While vector similarity matching can effectively measure semantic relevance, it is difficult to determine whether a particular page image truly helps answer the user's actual question. Therefore, a visual language model can be used to score each relevant page image, that is, to calculate the relevance score between each relevant page image and the user's query text. Then, from among the relevant page images, page images with relevance scores higher than a set first threshold are selected as candidate page images. For example, the maximum number of tokens output by the visual language model can be set to 1, guiding it to output an integer score between 0 and 9 to represent the relevance score between the page image and the user's query text. The scoring formula is as follows:

[0079]

[0080] in, This represents the relevance score, where t represents the corresponding word in the score, i.e., a rating scale of 0 to 9. This represents the probability distribution corresponding to word t. Since only a single word needs to be generated, this scoring process does not require complex decoding and is highly efficient.

[0081] Each candidate page image, as the output of the second page image retrieval mechanism, can be added to the candidate page list. Then, the page image containing the required visual information can be selected from these candidate images as the target page image. Because each candidate page image is more relevant to the user's query text and is more likely to contain the required visual information, this process ensures more accurate image parsing, thereby improving the efficiency and accuracy of multimodal reasoning.

[0082] In one implementation of this application, relevant page images are selected from the original page images based on the similarity score between each original page image and the user query text, including:

[0083] (1) Arrange the similarity scores of each original page image and the user query text from high to low to obtain a descending similarity sequence, and fit the descending similarity sequence into a similarity decrease curve;

[0084] (2) Construct a target straight line connecting the two endpoints of the similarity decrease curve;

[0085] (3) Based on the vertical distance between each point in the similarity decline curve and the target line, determine the turning point where the similarity score drops sharply from the similarity decline curve;

[0086] (4) Select the page images with a similarity score higher than that of the inflection point from each original page image as each relevant page image.

[0087] In the execution of the first page image retrieval mechanism, an inflection point detection mechanism can be introduced to adaptively identify highly similar pages. Specifically, the similarity scores of each original page image and the user's query text can be sorted from high to low to obtain a descending similarity sequence, which is then fitted to a continuous curve, called the similarity decline curve. The horizontal axis represents the position point of the curve, and the vertical axis represents the corresponding similarity score. Then, a target straight line connecting the two endpoints of the similarity decline curve is constructed. For example, assuming the two endpoints of the similarity decline curve are... and Then, a target line AB is constructed. Based on the perpendicular distance between each point in the similarity decline curve and the target line, the inflection point where the similarity score drops sharply can be determined from the similarity decline curve. Finally, page images with similarity scores higher than the similarity scores corresponding to the inflection points can be selected from each original page image as the relevant page images. For example, for the target line AB, the first inflection point where the similarity score drops sharply can be represented as... ,in, Indicates the first curve From point to line The vertical distance. Assuming the turning point. The corresponding similarity score is Then, select the images with similarity scores higher than those from each original page image. The images of those pages are used as the relevant page images obtained through retrieval. This processing method does not require a preset fixed Top-k value, has good adaptability and robustness, and is more flexible than the traditional KNN-based fixed threshold retrieval method, while also leaving more room for subsequent fine-grained screening.

[0088] The technical solution of this application embodiment can introduce a multi-level visual information caching management mechanism, which divides all document pages of a multimodal document into four categories and adds them to four different lists. Specifically, original page images can be added to the original page list, relevant page images obtained through the first page image retrieval mechanism can be added to the relevant page list, candidate page images obtained through the second page image retrieval mechanism can be added to the candidate page list, and page images selected for parsing and currently participating in the text reasoning process can be added to the cached page list. For specific implementation details of this multi-level visual information caching management mechanism, please refer to the following text.

[0089] In one implementation of this application, each candidate page image is added to a candidate page list; from each candidate page image, a page image containing the required visual information is selected as the target page image, including:

[0090] (1) If a first page image containing the required visual information is found in the cached page list, then the first page image is determined as the target page image;

[0091] (2) If the first page image cannot be found in the cached page list, and the second page image containing the required visual information is found in the candidate page list, then the second page image is determined as the target page image and added to the cached page list.

[0092] In the visual information supplementation process, the visual understanding agent first searches the cached page list for a page image containing the required visual information, denoted as the first page image. The agent parses each page image in the cached page list; if it doesn't contain the required visual information, it continues parsing other page images until it finds one containing the required visual information or traverses and parses all page images in the cached page list. If the first page image containing the required visual information is found in the cached page list, it is designated as the target page image, and the required visual semantic information is extracted. If the first page image containing the required visual information is not found in the cached page list, the agent sequentially visits the candidate page list, searching for a page image containing the required visual information in the same way, denoted as the second page image. Similarly, the agent parses each page image in the candidate page list; if it doesn't contain the required visual information, it continues parsing other page images until it finds one containing the required visual information or traverses and parses all page images in the candidate page list. When a page image containing the required visual information is found in the candidate page list, it is called a cache hit. At this time, the page image is determined as the target page image, the required visual semantic information is extracted, and the page image is added to the cache page list.

[0093] Furthermore, if no page image containing the required visual information is found in the candidate page list, one approach is to continue searching downwards, that is, to continue searching the relevant page list and the original page list in the same way. Another approach is for a decision-making agent to trigger a new round of visual retrieval process to update the candidate page list and the relevant page list.

[0094] In one implementation of this application, the method further includes:

[0095] If a second page image cannot be found from the candidate page list, the similarity score and relevance score corresponding to each original page image are recalculated to update the candidate page list, provided that the user query text is replaced with a visual information supplementation instruction.

[0096] Regarding the handling method that triggers a new round of visual retrieval, when replacing the user's query text with visual information supplementation instructions, the similarity score and relevance score corresponding to each original page image are recalculated as described above. Since the similarity score and relevance score between the page image and the visual information supplementation instructions are calculated at this time, different relevant page images and candidate page images can be obtained, thereby updating the candidate page list and the relevant page list. After the list update is completed, the visual understanding agent continues to search for page images containing the required visual information from each list in turn.

[0097] By introducing the aforementioned multi-level visual information caching management mechanism, the visual understanding agent can prioritize parsing those page images that are more likely to contain the required visual information, thereby improving its ability to focus on key information and integrate scattered evidence in multi-source retrieval results.

[0098] As an example, Figure 2 This is a schematic diagram illustrating the operational principle of the multi-agent collaborative multimodal reasoning method provided in this application embodiment in a practical application scenario. Figure 2 The technical framework shown does not require complex preprocessing processes such as OCR and layout recognition for multimodal documents. It directly uses the original document image as input, which can effectively improve the system's robustness and deployment efficiency. Figure 2 The adaptive visual information retrieval and extraction module shown on the left uses the two page image retrieval mechanisms described above to dynamically extract several page images most relevant to the user's query text from long document images, and finally outputs each relevant page image and each candidate page image, providing candidate input for subsequent region-level visual understanding. Figure 2 The visual-reasoning collaborative understanding module shown on the right utilizes a multi-agent collaborative mechanism. Based on the extracted candidate key visual content, it combines the capabilities of text reasoning agents and visual understanding agents to perform deep semantic understanding and logical reasoning. Furthermore, it guides the information supplementation process through a judgment-type agent with judgment and feedback mechanisms, thereby achieving more accurate and complete document understanding and response text generation.

[0099] For example, in Figure 2In the text reasoning agent, when performing multimodal reasoning, it discovers a lack of supporting information, "profit in 2020." At this point, the judgment agent generates a visual information supplementation instruction to "extract the profit data for 2019 and 2020 from the provided document" and sends this instruction to the visual understanding agent. The visual understanding agent first searches for page images containing the required visual information in the cached page list, parses them, extracts and supplements the corresponding visual semantic information, "net profit in 2019 was $240 million, and net profit in 2020 was $250 million." After that, the text reasoning agent can continue to the next step of multimodal reasoning to calculate the company's profit growth rate.

[0100] The technical solution of this application embodiment sets up a text reasoning agent, a visual understanding agent, and a judgment agent. The text reasoning agent leads the overall text logic reasoning, the visual understanding agent is responsible for providing visual semantic information, and the judgment agent is responsible for evaluating whether the multimodal reasoning process needs to be supplemented with additional visual semantic information. If supplementation is required, the visual information supplementation process is triggered to guide the visual understanding agent to extract the required supplementary visual semantic information. After acquiring the user query text related to the multimodal document, the text reasoning agent first performs multimodal reasoning based on the user query text and the visual semantic information of the generated multimodal document, generating a thought-based reasoning text. Then, the judgment agent determines whether the multimodal reasoning process lacks supporting information based on the thought-based reasoning text. If so, it generates a visual information supplementation instruction and sends it to the visual understanding agent. Guided by the visual information supplementation instruction, the visual understanding agent performs visual semantic understanding on the target page image of the multimodal document, supplementing and updating the visual semantic information. The text reasoning agent then continues multimodal reasoning based on the updated visual semantic information, thus forming a closed-loop mechanism of multimodal reasoning, supplementation judgment, and information supplementation. It can be seen that as the number of information supplementation attempts increases, the visual semantic information required for multimodal reasoning by the text reasoning agent becomes richer and more complete, thereby continuously improving the overall reasoning effect and ultimately obtaining an accurate and complete response text. The above process utilizes the collaborative work of multiple intelligent agents to perform deep semantic understanding and logical reasoning on multimodal documents without the need for complex preprocessing procedures, thereby achieving accurate and complete document understanding and response text generation.

[0101] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0102] The above mainly describes a multi-agent collaborative multimodal reasoning method. The following will describe a multi-agent collaborative multimodal reasoning device.

[0103] Please see Figure 3 This application illustrates a multi-agent collaborative multimodal reasoning device according to an embodiment of the present application, comprising:

[0104] The query text acquisition module 301 is used to acquire user query text related to multimodal documents;

[0105] The multimodal reasoning module 302 is used to perform multimodal reasoning based on the visual semantic information of the user query text and the generated multimodal document through a text reasoning agent, and generate thinking reasoning text.

[0106] The visual supplementation guidance module 303 is used to determine whether the multimodal reasoning process lacks supporting information based on the thinking and reasoning text by a judgment-type intelligent agent, and generate a visual information supplementation instruction when the multimodal reasoning process lacks supporting information, and send the visual information supplementation instruction to the visual understanding intelligent agent.

[0107] The visual semantic understanding module 304 is used to perform visual semantic understanding on the target page image of the multimodal document under the guidance of visual information supplementation instructions through the visual understanding agent, update the visual semantic information, and then return to execute the steps of the text reasoning agent to perform multimodal reasoning based on the user query text and the visual semantic information of the generated multimodal document to generate thinking reasoning text.

[0108] In one implementation of this application, the multi-agent collaborative multimodal reasoning device further includes:

[0109] The page image acquisition module is used to acquire the original page images of each page in a multimodal document;

[0110] The page image selection module is used to select a page image containing the required visual information from each original page image as the target page image; wherein the required visual information is determined according to the visual information supplementation instruction.

[0111] In one implementation of this application, the page image selection module includes:

[0112] The similarity calculation unit is used to embed the user query text and each original page image into a document encoding model, and calculate the similarity score between each original page image and the user query text.

[0113] The relevant page image selection unit is used to select relevant page images from each original page image based on the similarity score between each original page image and the user query text.

[0114] The relevance calculation unit is used to calculate the relevance score between each relevant page image and the user query text using a visual language model.

[0115] The candidate page image selection unit is used to select page images with a relevance score higher than the first threshold from each relevant page image as candidate page images.

[0116] The target page image selection unit is used to select a page image containing the required visual information from each candidate page image as the target page image.

[0117] In one implementation of this application, the relevant page image selection unit includes:

[0118] The similarity ranking subunit is used to sort each original page image and the user query text from high to low similarity score to obtain a descending similarity sequence, and then fit the descending similarity sequence into a similarity decrease curve;

[0119] The straight line construction sub-unit is used to construct the target straight line connecting the two endpoints of the similarity decrease curve;

[0120] The inflection point determination subunit is used to determine the inflection point where the similarity score drops sharply from the similarity decline curve based on the vertical distance between each point in the similarity decline curve and the target line.

[0121] The relevant page image selection sub-unit is used to select page images from each original page image whose similarity score is higher than the similarity score corresponding to the inflection point, and use them as the relevant page images.

[0122] In one implementation of this application, each candidate page image is added to a candidate page list; the target page image selection unit includes:

[0123] The first page image lookup subunit is used to determine the first page image as the target page image if a first page image containing the required visual information is found in the cached page list.

[0124] The second page image search subunit is used to determine the second page image as the target page image and add it to the cached page list if the first page image cannot be found in the cached page list and a second page image containing the required visual information is found in the candidate page list.

[0125] In one implementation of this application, the page image selection module further includes:

[0126] The candidate page list update unit is used to recalculate the similarity score and relevance score corresponding to each original page image in order to update the candidate page list if a second page image cannot be found from the candidate page list, and the user query text is replaced with a visual information supplementation instruction.

[0127] In one implementation of this application, the visual supplementary guidance module includes:

[0128] The semantic scoring unit is used to score the semantic completeness of reasoning text according to the built-in scoring rules of the judgment agent;

[0129] The supporting information missing determination unit is used to determine that the multimodal reasoning process lacks supporting information if the score value is lower than a second threshold.

[0130] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multi-agent collaborative multimodal reasoning method described in any of the above embodiments.

[0131] This application also provides a computer program product that, when run on an electronic device, causes the electronic device to execute the multi-agent collaborative multimodal reasoning method as described in any of the above embodiments.

[0132] Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 4 in this embodiment includes: a processor 40, a memory 41, and a computer program 42 stored in the memory 41 and executable on the processor 40. When the processor 40 executes the computer program 42, it implements the steps in the embodiments of the various multi-agent collaborative multimodal reasoning methods described above, for example... Figure 1 Steps 101-104 are shown. Alternatively, when the processor 40 executes the computer program 42, it implements the functions of each module / unit in the above-described device embodiments, for example, implementing... Figure 3 The functions of modules 301-304 of the device shown.

[0133] The computer program 42 can be divided into one or more modules / units, which are stored in the memory 41 and executed by the processor 40 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 42 in the electronic device 4.

[0134] The processor 40 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0135] The memory 41 can be an internal storage unit of the electronic device 4, such as a hard disk or memory. The memory 41 can also be an external storage device of the electronic device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Furthermore, the memory 41 can include both internal and external storage units of the electronic device 4. The memory 41 is used to store the computer program and other programs and data required by the electronic device. The memory 41 can also be used to temporarily store data that has been output or will be output.

[0136] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0137] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0138] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0139] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0140] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0141] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0142] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0143] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0144] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A multi-agent collaborative multi-modal reasoning method, characterized in that, The method comprises: obtaining user query text related to a multi-modal document; generating thinking reasoning text by performing multi-modal reasoning on the user query text and visual semantic information of the multi-modal document generated by a text reasoning agent; determining whether the process of multi-modal reasoning is lacking supporting information according to the thinking reasoning text by a judging agent, and generating a visual information supplement instruction in the case that the process of multi-modal reasoning is lacking supporting information, and sending the visual information supplement instruction to a visual understanding agent; updating the visual semantic information by performing visual semantic understanding on a target page image of the multi-modal document under the guidance of the visual information supplement instruction by the visual understanding agent, and then returning to perform the step of generating thinking reasoning text by performing multi-modal reasoning on the user query text and visual semantic information of the multi-modal document generated by the text reasoning agent; wherein the target page image is determined by: obtaining each original page image of the multi-modal document; embedding representation of the user query text and each original page image by a document encoding model, and calculating the similarity score of each original page image and the user query text respectively; arranging the similarity scores of each original page image and the user query text in descending order to obtain a descending similarity sequence, and fitting the descending similarity sequence into a similarity decline curve; constructing a target straight line connecting two endpoints of the similarity decline curve; determining a turning point of sharp decline of the similarity score from the similarity decline curve according to the perpendicular distance of each point in the similarity decline curve to the target straight line; selecting, from the original page images, a page image corresponding to a similarity score higher than the similarity score corresponding to the turning point as each relevant page image; determining the target page image from the relevant page images.

2. The method of claim 1, wherein, The method of determining the target page image from the relevant page images comprises: selecting, from the relevant page images, a page image containing required visual information as the target page image; wherein the required visual information is determined according to the visual information supplement instruction.

3. The method of claim 2, wherein, The method of selecting, from the relevant page images, a page image containing required visual information as the target page image comprises: calculating the relevance score of each relevant page image and the user query text by a visual language model respectively; selecting, from the relevant page images, a page image corresponding to a relevance score higher than a first threshold value as each candidate page image; selecting, from the candidate page images, a page image containing the required visual information as the target page image.

4. The method of claim 3, wherein, The candidate page images are added to a candidate page list. The method of selecting, from the candidate page images, a page image containing the required visual information as the target page image comprises: If a first page image containing the required visual information is found from the cache page list, the first page image is determined as the target page image; If the first page image is not found from the cache page list, and a second page image containing the required visual information is found from the candidate page list, the second page image is determined as the target page image, and the second page image is added to the cache page list.

5. The method of claim 4, wherein, Further comprising: If the second page image is not found from the candidate page list, the similarity score and the relevance score corresponding to each original page image are recalculated to update the candidate page list in the case of replacing the user query text with the visual information supplement instruction.

6. The method according to any one of claims 1 to 5, characterized in that, The process of determining whether the multi-modal reasoning process lacks supporting information according to the thinking reasoning text includes: According to the built-in scoring rules of the judging agent, the semantic completeness of the thinking reasoning text is scored; If the score value is lower than a second threshold value, it is determined that the multi-modal reasoning process lacks supporting information.

7. A multi-agent collaborative multi-modal reasoning apparatus, characterized by, Comprise: A query text acquisition module for acquiring user query text related to a multi-modal document; A multi-modal reasoning module for generating a thinking reasoning text by a text reasoning agent according to the user query text and the visual semantic information of the generated multi-modal document; A visual supplement guidance module for determining whether the multi-modal reasoning process lacks supporting information according to the thinking reasoning text by a judging agent, and generating a visual information supplement instruction in the case of lacking supporting information in the multi-modal reasoning process, and sending the visual information supplement instruction to a visual understanding agent; A visual semantic understanding module for performing visual semantic understanding on a target page image of the multi-modal document under the guidance of the visual information supplement instruction by the visual understanding agent, updating the visual semantic information, and then returning to the step of generating a thinking reasoning text by a text reasoning agent according to the user query text and the generated visual semantic information of the multi-modal document; A page image acquisition module for acquiring each original page image of the multi-modal document; A similarity calculation unit for embedding representation of the user query text and each original page image by a document encoding model, and calculating the similarity score of each original page image and the user query text; A similarity sorting subunit for arranging the similarity score of each original page image and the user query text from high to low to obtain a descending similarity sequence, and fitting the descending similarity sequence into a similarity decline curve; A straight line construction subunit for constructing a target straight line connecting two endpoints of the similarity decline curve; A turning point determination subunit for determining a turning point of sharp decline in similarity score from the similarity decline curve according to the perpendicular distance of each point in the similarity decline curve to the target straight line. A relevant page image selection subunit is configured to select, from the original page images, a page image corresponding to a similarity score higher than the similarity score corresponding to the turning point as a relevant page image. A target page image determination unit is configured to determine the target page image from the relevant page images.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the multi-agent collaborative multi-modal reasoning method of any one of claims 1 to 6.

9. A computer program product, characterised in that, When the computer program product runs on the electronic device, the electronic device executes the multi-agent collaborative multi-modal reasoning method of any one of claims 1 to 6.

Images