Multi-agent collaborative multi-modal reasoning method and device, electronic equipment and product

Abstract

The invention relates to the technical field of artificial intelligence, and provides a multi-agent collaborative multi-modal reasoning method and device, electronic equipment and a computer program product. According to the method, three agents including a text reasoning agent, a visual understanding agent and a judgment type agent are arranged, the text reasoning agent dominates overall text logical reasoning, and the visual understanding agent is responsible for providing visual semantic information; the judgment type intelligent agent is responsible for evaluating whether additional visual semantic information needs to be supplemented in the multi-mode reasoning process or not, triggering a visual information supplementing process under the condition that the additional visual semantic information needs to be supplemented, and guiding the visual understanding intelligent agent to extract the visual semantic information needing to be supplemented. According to the method, a closed-loop mechanism of multi-modal reasoning, supplementary judgment and information supplementation is formed by utilizing cooperative work of the three agents, and deep semantic understanding and logical reasoning can be performed on a multi-modal document on the basis of not executing a complex preprocessing process, so that accurate and complete document understanding and reply text generation are realized.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a multi-modal reasoning method, device, electronic device and computer program product for multi-agent collaboration. Background Art

[0002] Multimodal documents, as the most common form of everyday knowledge carrier, integrate at least two of various information forms such as text, images, tables, formulas, audio, and video. They are widely present in practical scenarios such as manuals, reports, and papers. With the rise of document-based knowledge base products, users' demand for structured understanding and intelligent question-and-answer capabilities of multimodal documents is increasing. When processing multimodal documents, existing technologies usually require complex preprocessing processes such as OCR recognition and format conversion. Not only are these processes cumbersome and error-prone, they are also prone to semantic loss or information misalignment, affecting the semantic understanding of multimodal documents and ultimately reducing the accuracy and completeness of the response text. Summary of the Invention

[0003] In view of this, the embodiments of the present application provide a multi-modal reasoning method, device, electronic device and computer program product for multi-agent collaboration. By utilizing the collaborative work of multiple agents, it is possible to perform deep semantic understanding and textual logical reasoning on multimodal documents without the need to perform complex preprocessing processes, thereby achieving accurate and complete document understanding and response text generation.

[0004] A first aspect of the embodiments of the present application provides a multi-agent collaborative multimodal reasoning method, comprising: Obtaining user query text related to multimodal documents; Through the text reasoning agent, multimodal reasoning is performed based on the visual semantic information of the user query text and the generated multimodal documents to generate thinking and reasoning text; Through the judgment agent, whether the multimodal reasoning process lacks supporting information is determined based on the thinking and reasoning text, and if the multimodal reasoning process lacks supporting information, visual information supplementary instructions are generated and sent to the visual understanding agent; Through the visual understanding agent, under the guidance of the visual information supplementation instruction, visual semantic understanding is performed on the target page image of the multimodal document, the visual semantic information is updated, and then the text reasoning agent is returned to perform multimodal reasoning based on the user query text and the visual semantic information of the generated multimodal document to generate the steps of thinking and reasoning text.

[0005] The technical solution of the embodiment of the present application sets up a text reasoning agent, a visual understanding agent and a judgment agent, among which the text reasoning agent leads the overall text logical reasoning, the visual understanding agent is responsible for providing visual semantic information, and the judgment agent is responsible for evaluating whether the multimodal reasoning process needs to be supplemented with additional visual semantic information, and triggers the visual information supplement process if supplementation is required, guiding the visual understanding agent to extract the required visual semantic information. After obtaining a user query text related to a multimodal document, a text reasoning agent first performs multimodal reasoning based on the user query text and the visual semantic information of the generated multimodal document, generating a thought reasoning text. Then, a judgment agent determines whether the multimodal reasoning process lacks supporting information based on the thought reasoning text. If so, it generates visual information supplementation instructions and sends these instructions to the visual understanding agent. Guided by the visual information supplementation instructions, the visual understanding agent performs visual semantic understanding on the target page image of the multimodal document, supplementing and updating the visual semantic information. The text reasoning agent then continues multimodal reasoning based on the updated visual semantic information, forming a closed-loop mechanism of multimodal reasoning, supplementary judgment, and information supplementation. It can be seen that with the increase in the number of information supplementations, the visual semantic information required by the text reasoning agent for multimodal reasoning becomes richer and more complete, thereby continuously improving the overall reasoning effect and ultimately obtaining an accurate and complete response text. The above process utilizes the collaborative work of multiple intelligent agents. Without the need to perform complex preprocessing procedures, it can perform deep semantic understanding and logical reasoning on multimodal documents, thereby achieving accurate and complete document understanding and response text generation.

[0006] In one implementation of the embodiment of the present application, the target page image is determined in the following manner: Obtaining original page images of the multimodal document; From each original page image, a page image containing required visual information is selected as a target page image; wherein the required visual information is determined according to the visual information supplement instruction.

[0007] In one implementation of the embodiment of the present application, selecting a page image containing required visual information as a target page image from each original page image includes: The user query text and each original page image are embedded through the document encoding model, and the similarity score between each original page image and the user query text is calculated respectively; Selecting relevant page images from the original page images based on the similarity score between each original page image and the user query text; Calculate the relevance score between each relevant page image and the user query text through the visual language model; Selecting, from the relevant page images, page images whose corresponding relevance scores are higher than a first threshold as candidate page images; From each candidate page image, a page image containing required visual information is selected as the target page image.

[0008] In one implementation of the embodiment of the present application, selecting relevant page images from the original page images based on a similarity score between each original page image and the user query text includes: Arrange the similarity scores between each original page image and the user query text from high to low to obtain a descending similarity sequence, and fit the descending similarity sequence into a similarity descending curve; Construct a target straight line connecting the two endpoints of the similarity decrease curve; According to the vertical distance between each point in the similarity decrease curve and the target straight line, the turning point where the similarity score drops sharply is determined from the similarity decrease curve; From the original page images, page images corresponding to the images with higher similarity scores than the similarity scores corresponding to the turning points are selected as the relevant page images.

[0009] In one implementation of the embodiment of the present application, each candidate page image is added to a candidate page list; and a page image containing required visual information is selected from each candidate page image as a target page image, including: If a first page image containing the required visual information is found in the cache page list, the first page image is determined as the target page image; If the first page image cannot be found in the cache page list, and a second page image containing the required visual information is found in the candidate page list, the second page image is determined as the target page image and added to the cache page list.

[0010] In one implementation of the embodiment of the present application, the method further includes: If the second page image cannot be found in the candidate page list, the similarity score and relevance score corresponding to each original page image are recalculated by replacing the user query text with the visual information supplement instruction to update the candidate page list.

[0011] In one implementation of the embodiment of the present application, determining whether the multimodal reasoning process lacks supporting information based on the thinking and reasoning text includes: Scoring the semantic completeness of the thinking and reasoning text according to the built-in scoring rules of the judgment-type intelligent agent; If the value of the score is lower than the second threshold, it is determined that the multimodal reasoning process lacks supporting information.

[0012] A second aspect of the embodiments of the present application provides a multi-agent collaborative multimodal reasoning device, comprising: A query text acquisition module is used to obtain user query text related to the multimodal document; The multimodal reasoning module is used to generate thinking and reasoning text based on the visual semantic information of the user query text and the generated multimodal documents through the text reasoning agent; The visual supplement guidance module is used to determine whether the multimodal reasoning process lacks supporting information based on the thinking and reasoning text through the judgment-type intelligent agent, and to generate visual information supplement instructions if the multimodal reasoning process lacks supporting information, and send the visual information supplement instructions to the visual understanding intelligent agent; The visual semantic understanding module is used to perform visual semantic understanding on the target page image of the multimodal document through a visual understanding agent under the guidance of visual information supplementation instructions, update the visual semantic information, and then return to execute the steps of generating thinking and reasoning text by performing multimodal reasoning based on the user query text and the visual semantic information of the generated multimodal document through a text reasoning agent.

[0013] The third aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the multi-modal reasoning method for multi-agent collaboration as provided in the first aspect of an embodiment of the present application.

[0014] The fourth aspect of the embodiments of the present application provides a computer program product, which, when running on an electronic device, enables the electronic device to execute the multi-agent collaborative multimodal reasoning method provided by the first aspect of the embodiments of the present application.

[0015] The fifth aspect of the embodiments of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the multi-modal reasoning method for multi-agent collaboration provided by the first aspect of the embodiments of the present application.

[0016] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flowchart of a multi-agent collaborative multimodal reasoning method provided in an embodiment of the present application; Figure 2This is a schematic diagram of the operating principle of the multi-agent collaborative multimodal reasoning method provided in an embodiment of the present application in an actual application scenario; Figure 3 This is a schematic diagram of the structure of a multi-agent collaborative multimodal reasoning device provided in an embodiment of the present application; Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system structures and technologies are provided to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present application with unnecessary details. In addition, in the description of the present application specification and the appended claims, the terms "first," "second," "third," etc. are only used to distinguish descriptions and are not to be understood as indicating or implying relative importance.

[0019] With the rise of document-based knowledge base products, users are increasingly demanding structured understanding of multimodal documents and intelligent question-answering. Existing technologies typically use a method that combines document preprocessing with sub-modal understanding when processing multimodal documents. This method treats multimodal documents as a combination of multiple modal data. First, through preprocessing processes such as image-text parsing and format conversion, the document is split into data sets of independent modalities such as text and images. Each modal information is then modeled and understood separately. However, the above preprocessing process is cumbersome, complex, and error-prone. Furthermore, the overall semantic structure of the document is easily lost during the modal segmentation process. For example, visual contextual information such as typesetting and layout is ignored, which affects the semantic understanding of multimodal documents. Furthermore, in intelligent question-answering scenarios, the large language models used have context length limitations. In practical applications, it is usually necessary to segment the document content into multiple segments and retrieve the most relevant segments for processing during question-answering. This process relies on a pure text retrieval strategy, making it difficult to effectively integrate visual semantic information, ultimately resulting in a decrease in the accuracy and completeness of the response text.

[0020] To address the above issues, the embodiments of this application propose a multi-agent collaborative multimodal reasoning method, device, electronic device, and computer program product. By utilizing the collaborative work of multiple agents, without the need for complex preprocessing procedures, this method can perform deep semantic understanding and textual logical reasoning on multimodal documents, thereby achieving accurate and complete document understanding and response text generation. For more specific technical implementation details of the embodiments of this application, please refer to the various method embodiments described below.

[0021] It should be understood that the execution subjects of the various method embodiments proposed in the present application can be various types of electronic devices, such as mobile phones, tablet computers, desktop computers, wearable devices, medical devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), large-screen TVs, etc. The embodiments of the present application do not impose any restrictions on the specific types of the electronic devices.

[0022] See also Figure 1 , shows a multi-agent collaborative multimodal reasoning method provided by an embodiment of the present application, including: 101. Obtain user query text related to the multimodal document; The technical solution of the embodiment of the present application is mainly aimed at the intelligent question-answering scenario of multimodal documents. By inputting the multimodal document and the user query text into the intelligent agent or large language model for analysis, the corresponding reply text can be returned. Among them, the intelligent agent refers to a computer program with autonomy, learning ability and reasoning ability. It is an intelligent entity that can perceive environmental information, make decisions and perform actions. Multimodal documents can be documents of any type and subject containing at least two modal information such as text and images, such as a PDF document of a certain paper, a PPT document of a certain investigation report, and so on. The user query text is a text describing the problem related to the multimodal document input by the user. For example, assuming that the multimodal document is a paper on the theme of improving the marine environment, the user query text can be "Please analyze how to improve the marine environment based on this paper" or "Please determine what factors are related to the marine environment based on this paper", and so on.

[0023] 102. Through the text reasoning agent, multimodal reasoning is performed based on the visual semantic information of the user query text and the generated multimodal documents to generate thinking and reasoning text; The technical solution of the embodiment of the present application sets up three agents, namely a visual understanding agent, a text reasoning agent and a judgment agent. Among them, the visual understanding agent is responsible for providing visual semantic information of multimodal documents, the text reasoning agent leads the overall text logical reasoning, and the judgment agent is responsible for evaluating whether the multimodal reasoning process needs to be supplemented with additional visual semantic information. If supplementation is required, the visual information supplement process is triggered to guide the visual understanding agent to extract the required supplementary visual semantic information. In general, the text reasoning agent is used to lead the overall process of multimodal reasoning, the judgment agent evaluates the reasoning integrity, and the visual understanding agent provides key visual information support. The three work together to complete complex intelligent question-answering tasks. Next, the specific technical implementation details of the collaborative work of the three agents are described.

[0024] The text reasoning agent is responsible for building and executing the entire text reasoning chain and requires strong textual logical reasoning capabilities. High-performance reasoning models such as deepseek-r1 and qwq are typically used, as these models offer advantages in terms of stability, reasoning depth, and controllability. After receiving the user's query text, the text reasoning agent begins constructing a text reasoning path. It performs multimodal reasoning on the visual semantics of the query text and the currently generated multimodal documents, generating a thought reasoning text. Multimodal reasoning refers to the process of combining visual semantic information to perform text reasoning. The text reasoning agent can employ a step-by-step distributed reasoning approach, executing one logically divided reasoning step at a time and gradually constructing a complete text reasoning path. After completing each reasoning step, the text reasoning agent generates the corresponding thought reasoning text, which is the intermediate text generated by the agent during each analytical and thinking step. For example, when the text reasoning agent completes reasoning step 1, it generates the corresponding thinking and reasoning text. When the text reasoning agent completes reasoning step 2, it generates the corresponding thinking and reasoning text… and so on, until all reasoning steps are completed and the final thinking and reasoning text is obtained. Finally, based on this final thinking and reasoning text, the agent generates and outputs the response text corresponding to the user's query text. Furthermore, during the multimodal reasoning process, if the text reasoning agent detects that key supporting information is missing or reasoning is blocked, it pauses the current reasoning process and activates the judgment agent. The judgment agent guides the visual understanding agent to reparse the relevant content in the page image of the multimodal document and supplement the required visual semantic information. The text reasoning agent then proceeds to the next step of multimodal reasoning based on the supplemented visual semantic information. This cycle continues until the entire multimodal reasoning process is complete.

[0025] The visual semantic information of a multimodal document is extracted from the page images of the multimodal document by a visual understanding agent. In the initial stage of multimodal reasoning, the visual semantic information of the generated multimodal document may be empty. At this time, the text reasoning agent only performs reasoning based on the user's query text, which can easily lead to the problem of missing key supporting information. However, after multiple cycles of multimodal reasoning, supplementary judgment, and information supplementation, the visual semantic information of the generated multimodal document will become increasingly rich and complete, and the multimodal reasoning process will become smoother and smoother. Another implementation method is to use a visual understanding agent to extract a portion of key visual semantic information from the page images of the multimodal document before conducting the multimodal reasoning process. In this way, in the initial stage of multimodal reasoning, the text reasoning agent can combine the user query text and key visual semantic information to perform multimodal reasoning, thereby reducing the number of interruptions in the multimodal reasoning process and improving reasoning efficiency.

[0026] 103. Using a judgment agent, determine whether the multimodal reasoning process lacks supporting information based on the thinking and reasoning text, and if the multimodal reasoning process lacks supporting information, generate visual information supplementary instructions and send the visual information supplementary instructions to the visual understanding agent; The judgment agent serves as a coordination hub, monitoring the multimodal reasoning process of the text reasoning agent in real time. When it detects that supporting information is missing or reasoning is blocked, it triggers the visual information supplementation process, guiding the visual understanding agent to extract the required supplementary visual semantic information. Specifically, the judgment agent can judge whether the next reasoning step lacks supporting information and whether it needs to supplement additional visual content from the page image of the multimodal document based on the thinking and reasoning text generated by the text reasoning agent in the current reasoning step. If it is determined that the next reasoning step lacks supporting information, the multimodal reasoning process is interrupted and the visual information supplementation process is triggered. After the visual semantic information is supplemented, the text reasoning agent continues reasoning from the last interruption, that is, continues to execute the next reasoning step. Conversely, if it is determined that the next reasoning step does not lack supporting information, there is no need to trigger the visual information supplementation process, and the text reasoning agent can directly execute the next reasoning step.

[0027] In one implementation of the embodiment of the present application, determining whether the multimodal reasoning process lacks supporting information based on the thinking and reasoning text includes: (1) Score the semantic completeness of the thinking and reasoning text according to the built-in scoring rules of the judgment agent; (2) If the score value is lower than the second threshold, it is determined that the multimodal reasoning process lacks supporting information.

[0028] The judgment agent has built-in scoring rules that can be used to score the semantic completeness of the thought and reasoning text output by the text reasoning agent. If the score value is below a set second threshold, it can be determined that the semantic completeness of the thought and reasoning text is not up to standard, and the multimodal reasoning process lacks supporting information. In other words, the conditions for continuing multimodal reasoning are not met. At this time, the judgment agent triggers the visual information supplementation process. Conversely, if the score value is above the second threshold, it can be determined that the semantic completeness of the thought and reasoning text is up to standard, and the multimodal reasoning process is not lacking supporting information. In other words, the conditions for continuing multimodal reasoning are met, and there is no need to trigger the visual information supplementation process. In practice, the judgment agent can use vision language models (VLMs) to score the semantic completeness of the thought and reasoning text, guiding it to output a score between 0 and 9 by setting the maximum number of tokens output by the model. By introducing a scoring mechanism for thought and reasoning text, the judgment agent can more accurately identify whether the multimodal reasoning process lacks supporting information and thus more accurately trigger the visual information supplementation process.

[0029] When the visual information supplementation process begins, the judgment agent can generate descriptive instructions for the required visual information based on the current reasoning path, recorded as visual information supplementation instructions, and send the visual information supplementation instructions to the visual understanding agent to guide the visual understanding agent to locate and extract relevant visual content from the page image of the multimodal document. For example, if the text reasoning process involves "table data reference" or "formula meaning explanation", the judgment agent determines that the required visual information is table area information or numerical symbol information. At this time, the visual information supplementation instruction "Please pay attention to the table area data or numerical symbol data in the document page" is generated and sent to the visual understanding agent, thereby guiding the visual understanding agent to focus on the table area or specific mathematical symbol area in the document page image, and extract the corresponding visual features or local image information as supplementary visual semantic information for use by the text reasoning agent.

[0030] 104. Through the visual understanding agent, under the guidance of the visual information supplement instruction, the target page image of the multimodal document is subjected to visual semantic understanding and the visual semantic information is updated.

[0031] The visual understanding agent is used to parse the document image content, providing the necessary visual semantic information support for the multimodal reasoning process, ensuring that the response text is accurate and well-founded. During the visual information supplementation process, the visual understanding agent receives the visual information supplementation instruction sent by the judgment agent. Guided by this visual information supplementation instruction, it performs visual semantic understanding on the target page image of the multimodal document, extracts the required visual content, and supplements and updates the generated visual semantic information. The process then returns to step 102, where the text reasoning agent continues multimodal reasoning based on the updated visual semantic information. This forms a closed-loop mechanism of multimodal reasoning, supplementary judgment, and information supplementation, until the entire multimodal reasoning process concludes, resulting in the final thought and reasoning text. Finally, the corresponding response text is generated and output based on this final thought and reasoning text. Clearly, as the number of information supplementations increases, the visual semantic information required by the text reasoning agent for multimodal reasoning becomes richer and more complete, thereby continuously improving the overall reasoning effect and ultimately achieving an accurate and complete response text.

[0032] The target page image can be one or more page images contained in a multimodal document. Since it is difficult to a priori determine which document pages are truly helpful in answering questions during actual multimodal reasoning, one approach is to obtain all or most of the page images in the multimodal document as target page images for parsing to improve recall. However, since multimodal documents may contain up to hundreds of page images, many of which may be unrelated to the user's query text, this approach will obviously introduce a large amount of irrelevant visual information, affecting the efficiency and accuracy of multimodal reasoning.

[0033] In view of this, an embodiment of the present application proposes a multi-granularity visual retrieval strategy, which integrates two complementary page image retrieval mechanisms, and can quickly locate and filter out the part of page images related to the user query text from a large number of document page images. This part of the page image is more likely to contain the visual information required in the visual information supplement process. Therefore, using this part of the page image as the target page image can reduce the analysis of irrelevant visual information and effectively improve the efficiency and accuracy of multimodal reasoning.

[0034] In one implementation of the embodiment of the present application, the target page image is determined in the following manner: (1) Obtaining the original page images of the multimodal document; (2) Selecting a page image containing the required visual information from each original page image as the target page image; wherein the required visual information is determined according to the visual information supplement instruction.

[0035] In the visual information supplement process, according to the visual information supplement instruction sent by the judgment-type intelligent agent, the visual information that needs to be supplemented can be determined, and then the page image containing the required visual information is selected from all the original page images of the multimodal document as the target page image. Due to the large number of original page images, if a page-by-page parsing method is adopted, it may take a lot of time to locate the valid page, and a large amount of irrelevant visual information may be introduced to interfere with the multimodal reasoning process. In order to avoid this situation, the two page image retrieval mechanisms provided in the embodiment of the present application can be first adopted to perform two-stage screening of all the original page images of the multimodal document, and the page images finally screened out have a higher degree of relevance to the user query text. For the specific implementation of the two page image retrieval mechanisms, please refer to the following.

[0036] In one implementation of the embodiment of the present application, selecting a page image containing required visual information as a target page image from each original page image includes: (1) The user query text and each original page image are embedded through the document encoding model, and the similarity score between each original page image and the user query text is calculated respectively; (2) Select relevant page images from each original page image based on the similarity score between each original page image and the user query text; (3) Using the visual language model, calculate the relevance score between each relevant page image and the user query text; (4) Selecting, from the relevant page images, page images whose corresponding relevance scores are higher than a first threshold as candidate page images; (5) From each candidate page image, select the page image containing the required visual information as the target page image.

[0037] The first page image retrieval mechanism uses a clustering algorithm to quickly match a batch of page images that are semantically similar to the user's query text among the original page images of a multimodal document, thereby achieving coarse-grained visual information screening. Specifically, an efficient document encoding model such as ColPali can be used to embed the user query text and each original page image, and the similarity score between each original page image and the user query text is calculated separately. Then, based on the similarity score between each original page image and the user query text, the page images that are more similar to the user query text are selected from the original page images. These page images generally have a high similarity score with the user query text and can be recorded as relevant page images. Each relevant page image, as the output result of the first page image retrieval mechanism, can be added to the relevant page list.

[0038] The second page image retrieval mechanism uses the logits distribution output by the large model to evaluate the degree of semantic relevance between each relevant page image and the user's query text, achieves fine-grained visual information screening, and ultimately outputs several page images with the greatest potential relevance to the user's query text. Specifically, an adaptive fine-grained screening mechanism based on a visual language model can be introduced. Although vector similarity matching can effectively measure semantic relevance, it is difficult to determine whether a certain page image is truly helpful in answering the user's actual question. To this end, the visual language model can be used to score each relevant page image, that is, the relevance score of each relevant page image to the user's query text is calculated separately, and then from each relevant page image, the page image with the corresponding relevance score higher than the set first threshold is selected as each candidate page image. For example, the maximum number of word units output by the visual language model can be set to 1, and it can be guided to output an integer score between 0 and 9 to represent the relevance score of the page image to the user's query text. The scoring formula is as follows:

[0039] in, Represents the relevance score, t represents the word element with the corresponding score, that is, the rating level from 0 to 9, represents the probability distribution corresponding to word t. Since only a single word needs to be generated, the scoring process does not require complex decoding and has high processing efficiency.

[0040] Each candidate page image, as the output of the second page image retrieval mechanism, can be added to the candidate page list. Subsequently, the page image containing the desired visual information can be selected from the candidate page images as the target page image. Because each candidate page image is more relevant to the user's query text and more likely to contain the desired visual information, this process ensures more accurate image parsing, thereby improving the efficiency and accuracy of multimodal reasoning.

[0041] In one implementation of the embodiment of the present application, selecting relevant page images from the original page images based on a similarity score between each original page image and the user query text includes: (1) Arrange the similarity scores between each original page image and the user query text from high to low to obtain a descending similarity sequence, and fit the descending similarity sequence into a similarity descending curve; (2) Construct a target straight line connecting the two endpoints of the similarity reduction curve; (3) According to the vertical distance between each point in the similarity decrease curve and the target line, determine the turning point where the similarity score drops sharply from the similarity decrease curve; (4) From each original page image, select the page image whose corresponding similarity score is higher than the similarity score corresponding to the turning point as each related page image.

[0042] In the process of executing the first page image retrieval mechanism, in order to achieve adaptive recognition of high-similarity pages, an inflection point detection mechanism can be introduced. Specifically, the similarity scores between each original page image and the user query text can be arranged from high to low to obtain a descending similarity sequence, and then fitted into a continuous curve, called a similarity descent curve, where the horizontal axis represents the position of the curve and the vertical axis represents the corresponding similarity score. Then, a target straight line connecting the two endpoints of the similarity descent curve is constructed. For example, assuming that the two endpoints of the similarity descent curve are and , then construct the target straight line AB. Based on the vertical distance between each point in the similarity drop curve and the target straight line, the turning point where the similarity score drops sharply can be determined from the similarity drop curve. Finally, from each original page image, the page image with the corresponding similarity score higher than the similarity score corresponding to the turning point can be selected as each related page image. For example, for the target straight line AB, the turning point where the first similarity score drops sharply can be expressed as ,in, Indicates the curve Point to Line Assuming the turning point The corresponding similarity score is , then select the corresponding similarity score higher than The page images that are found in the search are used as the relevant page images obtained by the search. This processing method does not require a fixed Top-k value, has good adaptability and robustness, and is more flexible than the traditional KNN-based fixed threshold search method. It also leaves more room for subsequent fine-grained screening.

[0043] The technical solution of the embodiment of the present application can introduce a multi-level visual information cache management mechanism, which divides all document pages of a multimodal document into four categories and adds them to four different lists. Among them, each original page image can be added to the original page list, each related page image obtained through the first page image retrieval mechanism can be added to the related page list, each candidate page image obtained through the second page image retrieval mechanism can be added to the candidate page list, and the page image that has been selected for parsing and is currently participating in the text reasoning process can be added to the cached page list. For the specific implementation details of the multi-level visual information cache management mechanism, please refer to the following.

[0044] In one implementation of the embodiment of the present application, each candidate page image is added to a candidate page list; and a page image containing required visual information is selected from each candidate page image as a target page image, including: (1) If a first page image containing the required visual information is found in the cache page list, the first page image is determined as the target page image; (2) If the first page image cannot be found in the cache page list, and the second page image containing the required visual information is found in the candidate page list, the second page image is determined as the target page image and is added to the cache page list.

[0045] During the visual information supplementation process, the visual understanding agent prioritizes searching the cached page list for a page image containing the required visual information, which is referred to as the first page image. Each time the visual understanding agent parses a page image in the cached page list, if the parsing finds that it does not contain the required visual information, it continues parsing other page images until it finds a page image containing the required visual information or traverses through all the page images in the cached page list. If the first page image containing the required visual information is found in the cached page list, it is identified as the target page image and the required visual semantic information is extracted. If the first page image containing the required visual information is not found in the cached page list, the agent then descends through the candidate page list and searches for a page image containing the required visual information in the candidate page list in the same manner, which is referred to as the second page image. Similarly, each time the visual understanding agent parses a page image in the candidate page list, if the parsing finds that it does not contain the required visual information, it continues parsing other page images until it finds a page image containing the required visual information or traverses through all the page images in the candidate page list. When a page image containing the required visual information is found in the candidate page list, it is called a cache hit. At this time, the page image is determined as the target page image, the required visual semantic information is extracted, and the page image is added to the cache page list.

[0046] Furthermore, if no page image containing the required visual information is found in the candidate page list, one approach is to continue searching downward, that is, to continue searching the related page list and the original page list in the same manner. Another approach is for the judgment agent to trigger a new round of visual retrieval process to update the candidate page list and the related page list.

[0047] In one implementation of the embodiment of the present application, the method further includes: If the second page image cannot be found in the candidate page list, the similarity score and relevance score corresponding to each original page image are recalculated by replacing the user query text with the visual information supplement instruction to update the candidate page list.

[0048] To trigger a new round of visual retrieval, the visual information supplementation instruction replaces the user query text. The similarity and relevance scores for each original page image are recalculated as described above. Since the similarity and relevance scores are calculated between the page image and the visual information supplementation instruction, different relevant page images and candidate page images are obtained, thereby updating the candidate and relevant page lists. After completing the list update, the visual understanding agent continues to search for page images containing the required visual information from each list.

[0049] By introducing the above-mentioned multi-level visual information cache management mechanism, the visual understanding agent can prioritize parsing those page images that are more likely to contain the required visual information, thereby improving the ability to focus on key information and integrate scattered evidence in multi-source retrieval results.

[0050] As an example, Figure 2 This is a schematic diagram of the operating principle of the multi-agent collaborative multimodal reasoning method provided in an embodiment of the present application in an actual application scenario. Figure 2 The technical framework shown does not require complex preprocessing processes such as OCR and layout recognition for multimodal documents. It directly uses the original document image as input, which can effectively improve the system robustness and deployment efficiency. Figure 2 The adaptive visual information retrieval and extraction module shown on the left uses the two page image retrieval mechanisms described above to dynamically extract several page images most relevant to the user's query text from long document images. It ultimately outputs each relevant page image and each candidate page image, providing candidate input for subsequent region-level visual understanding. Figure 2 The visual-reasoning collaborative understanding module shown on the right uses a multi-agent collaborative mechanism to perform deep semantic understanding and logical reasoning based on the extracted candidate key visual content, combining the capabilities of the text reasoning agent and the visual understanding agent. It also guides the information supplement process through a judgment-type agent with a judgment and feedback mechanism, thereby achieving more accurate and complete document understanding and response text generation.

[0051] For example, in Figure 2In the example, the text reasoning agent discovered that the supporting information "profit in 2020" was missing when performing multimodal reasoning. At this time, the judgment agent generated a visual information supplement instruction of "extracting the profit data for 2019 and 2020 from the provided document" and sent the visual information supplement instruction to the visual understanding agent. The visual understanding agent first searched for the page image containing the required visual information from the cached page list for parsing, extracted and supplemented the corresponding visual semantic information "the net profit in 2019 was 240 million US dollars, and the net profit in 2020 was 250 million US dollars". After that, the text reasoning agent can continue to the next step of multimodal reasoning to calculate the company's profit growth rate.

[0052] The technical solution of the embodiment of the present application sets up a text reasoning agent, a visual understanding agent and a judgment agent, among which the text reasoning agent leads the overall text logical reasoning, the visual understanding agent is responsible for providing visual semantic information, and the judgment agent is responsible for evaluating whether the multimodal reasoning process needs to be supplemented with additional visual semantic information, and triggers the visual information supplement process if supplementation is required, guiding the visual understanding agent to extract the required visual semantic information. After obtaining a user query text related to a multimodal document, a text reasoning agent first performs multimodal reasoning based on the user query text and the visual semantic information of the generated multimodal document, generating a thought reasoning text. Then, a judgment agent determines whether the multimodal reasoning process lacks supporting information based on the thought reasoning text. If so, it generates visual information supplementation instructions and sends these instructions to the visual understanding agent. Guided by the visual information supplementation instructions, the visual understanding agent performs visual semantic understanding on the target page image of the multimodal document, supplementing and updating the visual semantic information. The text reasoning agent then continues multimodal reasoning based on the updated visual semantic information, forming a closed-loop mechanism of multimodal reasoning, supplementary judgment, and information supplementation. It can be seen that with the increase in the number of information supplementations, the visual semantic information required by the text reasoning agent for multimodal reasoning becomes richer and more complete, thereby continuously improving the overall reasoning effect and ultimately obtaining an accurate and complete response text. The above process utilizes the collaborative work of multiple intelligent agents. Without the need to perform complex preprocessing procedures, it can perform deep semantic understanding and logical reasoning on multimodal documents, thereby achieving accurate and complete document understanding and response text generation.

[0053] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0054] The above mainly describes a multi-modal reasoning method for multi-agent collaboration. The following will describe a multi-modal reasoning device for multi-agent collaboration.

[0055] See also Figure 3 , shows a multi-agent collaborative multimodal reasoning device provided by an embodiment of the present application, including: A query text acquisition module 301 is used to acquire user query text related to the multimodal document; The multimodal reasoning module 302 is used to perform multimodal reasoning based on the visual semantic information of the user query text and the generated multimodal documents through a text reasoning agent to generate a thinking and reasoning text; The visual supplementation guidance module 303 is configured to use the judgment agent to determine whether the multimodal reasoning process lacks supporting information based on the thinking and reasoning text, and to generate visual information supplementation instructions if the multimodal reasoning process lacks supporting information, and to send the visual information supplementation instructions to the visual understanding agent; The visual semantic understanding module 304 is used to perform visual semantic understanding on the target page image of the multimodal document through the visual understanding agent under the guidance of the visual information supplement instruction, update the visual semantic information, and then return to execute the multimodal reasoning through the text reasoning agent based on the user query text and the visual semantic information of the generated multimodal document to generate the thinking and reasoning text step.

[0056] In one implementation of the embodiment of the present application, the multi-agent collaborative multimodal reasoning device further includes: A page image acquisition module, used to acquire each original page image of a multimodal document; The page image selection module is used to select a page image containing required visual information from various original page images as a target page image; wherein the required visual information is determined according to the visual information supplement instruction.

[0057] In one implementation of the embodiment of the present application, the page image selection module includes: A similarity calculation unit is used to embed the user query text and each original page image through a document encoding model, and calculate the similarity score between each original page image and the user query text; A related page image selection unit is used to select each related page image from each original page image according to a similarity score between each original page image and the user query text; A relevance calculation unit, configured to calculate the relevance score of each relevant page image and the user query text using a visual language model; a candidate page image selection unit, configured to select, from among the relevant page images, page images having corresponding relevance scores higher than a first threshold value as candidate page images; The target page image selection unit is used to select a page image containing required visual information from each candidate page image as the target page image.

[0058] In one implementation of the embodiment of the present application, the relevant page image selection unit includes: A similarity sorting subunit is used to sort the similarity scores between each original page image and the user query text from high to low to obtain a descending similarity sequence, and fit the descending similarity sequence into a similarity descending curve; A straight line construction subunit is used to construct a target straight line connecting the two end points of the similarity reduction curve; a turning point determination subunit, configured to determine a turning point where the similarity score drops sharply from the similarity drop curve based on a vertical distance between each point in the similarity drop curve and the target straight line; The related page image selection subunit is used to select, from the original page images, page images whose corresponding similarity scores are higher than the similarity scores corresponding to the turning points as the related page images.

[0059] In one implementation of the embodiment of the present application, each candidate page image is added to a candidate page list; and the target page image selection unit includes: a first page image search subunit, configured to determine the first page image as a target page image if a first page image containing required visual information is found from the cache page list; The second page image search subunit is used to determine the second page image as the target page image and add the second page image to the cache page list if the first page image cannot be found from the cache page list and the second page image containing the required visual information is found from the candidate page list.

[0060] In one implementation of the embodiment of the present application, the page image selection module further includes: The candidate page list updating unit is configured to recalculate the similarity score and relevance score corresponding to each original page image by replacing the user query text with the visual information supplement instruction if the second page image cannot be found in the candidate page list, so as to update the candidate page list.

[0061] In one implementation of the embodiment of the present application, the visual supplement guidance module includes: The semantic scoring unit is used to score the semantic completeness of the thinking and reasoning text according to the built-in scoring rules of the judgment-type intelligent agent; The supporting information missing determination unit is configured to determine that the multimodal reasoning process lacks supporting information if the value of the score is lower than a second threshold.

[0062] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the multi-modal reasoning method of multi-agent collaboration described in any of the above embodiments.

[0063] An embodiment of the present application also provides a computer program product, which, when executed on an electronic device, enables the electronic device to execute the multi-agent collaborative multimodal reasoning method as described in any of the above embodiments.

[0064] Figure 4 Schematic diagram of an electronic device provided by an embodiment of the present application. Figure 4 As shown, the electronic device 4 of this embodiment includes: a processor 40, a memory 41, and a computer program 42 stored in the memory 41 and executable on the processor 40. When the processor 40 executes the computer program 42, the steps in the above-mentioned embodiments of the multi-agent collaborative multimodal reasoning method are implemented, such as Figure 1 Alternatively, when the processor 40 executes the computer program 42, the functions of the modules / units in the above-mentioned device embodiments are realized, for example, Figure 3 Functions of modules 301 - 304 of the illustrated apparatus.

[0065] The computer program 42 may be divided into one or more modules / units, which are stored in the memory 41 and executed by the processor 40 to implement the present application. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 42 in the electronic device 4.

[0066] The processor 40 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0067] The memory 41 can be an internal storage unit of the electronic device 4, such as a hard drive or memory of the electronic device 4. The memory 41 can also be an external storage device of the electronic device 4, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. equipped on the electronic device 4. Furthermore, the memory 41 can include both an internal storage unit of the electronic device 4 and an external storage device. The memory 41 is used to store the computer program and other programs and data required by the electronic device. The memory 41 can also be used to temporarily store data that has been output or is about to be output.

[0068] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0069] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0070] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0071] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0072] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0073] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0074] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0075] If the integrated unit is implemented as a software functional unit and sold or used as a standalone product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application can implement all or part of the process steps in the above-mentioned method embodiments by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content of the computer-readable medium can be appropriately increased or decreased based on the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, based on legislation and patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0076] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A multi-modal reasoning method for multi-agent collaboration, characterized in that: include: Obtaining user query text related to multimodal documents; Performing multimodal reasoning based on the user query text and the visual semantic information of the generated multimodal document through a text reasoning agent to generate a thinking and reasoning text; By means of a judgment agent, whether the multimodal reasoning process lacks supporting information is determined based on the thinking and reasoning text, and if the multimodal reasoning process lacks supporting information, a visual information supplement instruction is generated, and the visual information supplement instruction is sent to the visual understanding agent; Through the visual understanding agent, under the guidance of the visual information supplementation instruction, visual semantic understanding is performed on the target page image of the multimodal document, the visual semantic information is updated, and then the step of returning to execute the text reasoning agent to perform multimodal reasoning based on the user query text and the generated visual semantic information of the multimodal document to generate thinking and reasoning text.

2. The method according to claim 1, wherein The target page image is determined by: Acquiring original page images of the multimodal document; A page image containing required visual information is selected from the various original page images as the target page image; wherein the required visual information is determined according to the visual information supplement instruction.

3. The method according to claim 2, wherein The step of selecting a page image containing required visual information from the respective original page images as the target page image includes: Embedding the user query text and each of the original page images through a document encoding model, and calculating a similarity score between each of the original page images and the user query text; Selecting relevant page images from the original page images according to a similarity score between each original page image and the user query text; Calculating the relevance score between each of the relevant page images and the user query text using a visual language model; Selecting, from the relevant page images, page images whose corresponding relevance scores are higher than a first threshold as candidate page images; From the candidate page images, a page image containing the required visual information is selected as the target page image.

4. The method according to claim 3, wherein The selecting of relevant page images from the original page images according to the similarity score between each original page image and the user query text comprises: Arrange the similarity scores between each of the original page images and the user query text from high to low to obtain a descending similarity sequence, and fit the descending similarity sequence into a similarity descending curve; Constructing a target straight line connecting two end points of the similarity reduction curve; Determining, from the similarity reduction curve, a turning point where the similarity score drops sharply, according to a vertical distance between each point in the similarity reduction curve and the target straight line; From the original page images, page images corresponding to which the similarity scores are higher than the similarity scores corresponding to the turning points are selected as the relevant page images.

5. The method according to claim 3, wherein Each candidate page image is added to a candidate page list; The step of selecting a page image containing the required visual information from the candidate page images as the target page image includes: If a first page image containing the required visual information is found in the cache page list, the first page image is determined as the target page image; If the first page image cannot be found in the cache page list, and a second page image containing the required visual information is found in the candidate page list, the second page image is determined as the target page image and is added to the cache page list.

6. The method according to claim 5, wherein Also includes: If the second page image cannot be found in the candidate page list, the similarity score and the relevance score corresponding to each of the original page images are recalculated by replacing the user query text with the visual information supplement instruction to update the candidate page list.

7. The method according to any one of claims 1 to 6, wherein: The determining, based on the thinking and reasoning text, whether the multimodal reasoning process lacks supporting information includes: Scoring the semantic completeness of the thinking and reasoning text according to the built-in scoring rules of the judgment-type intelligent agent; If the value of the score is lower than a second threshold, it is determined that the multimodal reasoning process lacks supporting information.

8. A multi-agent collaborative multimodal reasoning device, characterized in that: include: A query text acquisition module is used to obtain user query text related to the multimodal document; A multimodal reasoning module is configured to perform multimodal reasoning based on the user query text and the visual semantic information of the generated multimodal document through a text reasoning agent to generate a thinking and reasoning text; a visual supplementation guidance module, configured to determine, by means of a judgment agent, whether the multimodal reasoning process lacks supporting information based on the thinking and reasoning text, and to generate a visual information supplementation instruction if the multimodal reasoning process lacks supporting information, and to send the visual information supplementation instruction to the visual understanding agent; A visual semantic understanding module is used to perform visual semantic understanding on the target page image of the multimodal document through the visual understanding agent under the guidance of the visual information supplementation instruction, update the visual semantic information, and then return to execute the step of performing multimodal reasoning based on the user query text and the generated visual semantic information of the multimodal document through the text reasoning agent to generate a thinking and reasoning text.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it implements the multi-modal reasoning method for multi-agent collaboration as described in any one of claims 1 to 7.

10. A computer program product, characterized in that When the computer program product runs on an electronic device, the electronic device executes the multi-modal reasoning method for multi-agent collaboration as described in any one of claims 1 to 7.

Images