Document question-answering system and method based on multi-agent collaboration, storage medium and computer program product

Abstract

The invention provides a document question-answering system and method based on multi-agent collaboration, a storage medium and a computer program product. The system comprises a document preprocessing module, a multi-modal retrieval related context module, a preliminary answer generation module, a key information extraction module and an answer generation module. According to the method, modular design is realized through a hierarchical agent collaborative architecture, each component can be independently optimized and reused, the expansibility is high, text and image multi-modal document information can be synchronously processed, and the scene limitation of a single-modal model is broken through. Each agent is subjected to a standardized pre-training and fine tuning process, and is matched with a multi-modal fusion and consistency verification mechanism to ensure that the answer logic is coherent and has a basis, and can efficiently serve scenes such as document retrieval, intelligent question and answer and the like.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, specifically relating to a document question-answering system, method, storage medium, and computer program product based on multi-agent collaboration. Background Technology

[0002] Traditional document question answering methods and systems have certain limitations in data processing, semantic understanding, and multimodal integration. For example, the traditional RAG model directly vectorizes the original document into chunks, lacking key meta-information such as summaries, keywords, and tags. This results in vectors that cannot accurately express the core content of the text, easily leading to inaccurate matching of core concepts during retrieval and recall failures. User questions are often concise or even vague. Traditional systems, based on keyword matching, lack deep semantic understanding capabilities and cannot proactively infer the user's true intent, easily causing vector retrieval to deviate from the target. Existing large-scale language models or large-scale visual language models can often only effectively process information of a single modality, making it difficult to simultaneously integrate textual and visual cues. Many questions require understanding both textual and image content simultaneously and establishing connections between these two modalities, which poses a significant challenge to traditional models. Summary of the Invention

[0003] To address the problems raised in the background art, this invention proposes a document question-answering system, method, storage medium, and computer program product based on multi-agent collaboration.

[0004] A document question-answering system based on multi-agent collaboration, which achieves one of the objectives of this invention, includes: Document preprocessing module: Used to preprocess user-provided documents and convert them into standardized datasets; Multimodal retrieval context module: used to retrieve from the standardized dataset the set of text segments and / or image pages most relevant to the question posed for the document; Preliminary answer generation module: This module takes the set of text segments and / or the set of image pages as input to the basic agent, performs a preliminary analysis based on the questions posed in response to the document, and outputs a preliminary answer. Key information extraction module: used to input the questions raised for the document, the set of text segments and / or the set of image pages, and the preliminary answers generated by the basic agent into the key information extraction agent to obtain key text information and / or key image information; Answer generation module: used to input the question posed for the document, the preliminary answer, the key text information and / or the key image information into the answering agent, and generate the final answer to the question.

[0005] Furthermore, it also includes a professional text processing module, used to input the question, the set of text segments, and the key text information into a professional text agent to obtain a detailed text answer; the detailed text answer is used to input the answer agent to generate the final answer to the question.

[0006] Furthermore, it also includes a professional image processing module, used to input the question, the set of image pages, and the key image information into a professional image agent to obtain a detailed visual answer; the detailed visual answer is used to input the answering agent to generate the final answer to the question.

[0007] Furthermore, the preprocessing process of the document preprocessing module includes: The text of each page in the document is extracted using OCR technology and / or PDF parsing technology. A subset of the text corpus in the standardized dataset is constructed by text segment (paragraph of text). A subset of the text corpus contains multiple text segments. Extract the original image objects corresponding to each page of the document, and construct a subset of the image corpus in the standardized dataset from the original image objects of all pages.

[0008] Furthermore, the multimodal retrieval context module includes a text retrieval unit. The text retrieval unit uses the SpanBERT model to perform semantic indexing on each text segment in the text corpus subset, calculates the semantic relevance score of each text segment to the question, sorts the text segments in descending order of the scores, and selects the top-ranked text segments as the set of text segments most relevant to the question posed for the document.

[0009] Furthermore, the multimodal retrieval context module includes an image retrieval unit. The image retrieval unit uses the PaLI-X model to generate visual embedding vectors for the original image objects in the image corpus subset, calculates the Euclidean distance between the visually associated embedding vectors and the question, normalizes the Euclidean distance to obtain a relevance score, and selects the top-ranked image pages as the set of image pages most relevant to the question raised for the document after sorting them in descending order of the scores.

[0010] Furthermore, the basic intelligent agent includes a text feature extraction component, which is used to perform semantic encoding on the sequence consisting of the set of text segments and the question, capture the association features between the set of text segments and the question based on the semantic encoding, and fuse the semantic encoding result and the association features to obtain multidimensional text features.

[0011] Furthermore, the basic intelligent agent includes a visual feature extraction component and a feature compression linear layer for feature dimension alignment. The input data of the visual feature extraction component is a preprocessed set of image pages. The multidimensional visual features output by the visual feature extraction component are output to the feature compression linear layer for dimension alignment processing, and finally multidimensional visual features with the same dimensions as the multidimensional text features are generated.

[0012] Furthermore, the methods by which the basic agent outputs a preliminary answer include: When the image page set is empty, the multidimensional text features are the preliminary answer output by the basic agent; When the set of text segments is empty, the multidimensional visual features are the preliminary answer output by the basic agent; Otherwise, the multidimensional visual features and multidimensional text features are fused through an attention mechanism to obtain a fused feature vector. The fully connected layer of the basic agent maps and transforms the fused feature vector to obtain a preliminary answer.

[0013] A document question-answering method based on multi-agent collaboration, which achieves the second objective of this invention, includes: Preprocess the user-provided documents to convert them into a standardized dataset; From the standardized dataset, retrieve the set of text segments and / or the set of image pages most relevant to the questions posed for the document; The set of text segments and / or the set of image pages are input into the basic agent, and a preliminary analysis is performed in conjunction with the questions raised in response to the document. The basic agent then outputs a preliminary answer. Input the questions posed in the document, the set of text segments and / or the set of image pages, and the preliminary answers generated by the basic agent into the key information extraction agent to obtain key text information and / or key image information; The question posed to the document, the preliminary answer, the key text information and / or the key image information are input into the response agent to generate the final answer to the question.

[0014] A non-transitory computer-readable storage medium for achieving the third objective of the present invention stores a computer program thereon, characterized in that, when the computer program is executed by a processor, it implements the steps of the document question-answering method based on multi-agent collaboration.

[0015] A computer program product for achieving the fourth objective of the present invention includes a computer program / instruction that, when executed by a processor, implements the steps of the document question-answering method based on multi-agent collaboration.

[0016] The beneficial effects of this invention include: This invention employs a hierarchical intelligent agent collaborative architecture. Through modular design of basic intelligent agents, key information extraction intelligent agents, and specialized text / image intelligent agents, it reduces system complexity. Each component can be independently optimized and reused, adapting to various multimodal document question-answering scenarios and exhibiting strong scalability. It can simultaneously process text and image-based document information, leveraging specialized intelligent agents to accurately extract text logic and image data, overcoming the limitations of single-modal models in handling multiple document types and improving question-answering coverage in complex document scenarios. Each intelligent agent utilizes a standardized pre-training and fine-tuning process, ensuring output accuracy through loss function optimization and F1 score verification. Combined with the multimodal fusion and consistency verification mechanism of the answering intelligent agent, it further eliminates information redundancy and conflicts, ensuring the final answer is logically coherent and verifiable. Furthermore, the training process is publicly available and directly reproducible, with clearly defined thresholds for model inference speed and accuracy to suit practical application needs. The final answer uses a structured format for easy understanding, making it widely applicable to document retrieval, intelligent question answering, and other scenarios, demonstrating significant practical value and applicability. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the method described in this invention; Figure 2 This is a schematic diagram of the system described in this invention. Detailed Implementation

[0018] The following detailed embodiments are provided to explain the technical solutions of the present invention, so that those skilled in the art can understand the present invention. The scope of protection of the present invention is not limited to the following specific embodiments. Any modifications or improvements made by those skilled in the art that incorporate the technical solutions of the present invention but differ from the following detailed embodiments are also within the scope of protection of the present invention.

[0019] A document question answering method based on multi-agent collaboration, such as Figure 1 As shown, it includes the following steps: S100: Obtain the documents uploaded by the user and the questions raised regarding the documents, wherein the documents are denoted as... , Containing multiple pages, represented as: N represents the total number of pages in the document, and the questions raised by users regarding this document are denoted as... q , q It is natural language text.

[0020] S200: For user-uploaded documents Preprocessing is performed, including information extraction, data normalization, and feature preservation, to convert the original documents into standardized data suitable for subsequent retrieval and agent analysis. Different techniques are used to extract information from different data structures, such as text or images. The specific steps are as follows: S201: For image-based documents, use OCR technology to recognize the document. The text contained therein; S202: For editable documents that exist in character encoding format, use PDF parsing technology to extract the document. Digital text in the text; S203: For documents Each page The extracted text can be represented as a series of text segments or paragraphs. t i ={ t i1 , t i2 ,…, t iM}, subscript i M indicates page The document contains M text segments; the text segments extracted from all pages form a subset of the document corpus. S204: If the document Includes tables or images, for each page The original image objects are preserved, with one original image object corresponding to each page, meaning the original visual layout and features must be retained. Through this comprehensive analysis utilizing both textual and visual cues, the dual representation of text and visuals forms the basis for the multimodal analysis performed by the framework; the original image objects of all pages constitute a subset of the image corpus of the document corpus. S300: Retrieve from the document corpus the question posed for the document. q The most relevant information, which needs to be considered in both text and image formats, is as follows: S301: For text retrieval, based on the subset of the text corpus, for each page All individual text segments below t ij Semantic indexing was performed using SpanBERT (Span-based Pre-training of BERT); addressing the issues raised regarding the document. q Problems generated using SpanBERT q The semantic embedding vector is then calculated, and the semantic embedding vector is correlated with all individual text segments in the entire document. t ij The semantic relevance scores of the semantic embedding vectors are sorted in descending order for all text segments in the entire document. t ijAfter performing a global sort, the set of the k most relevant text segments ranked highest across the entire document is retrieved. T q , represented as T q ={ t '1, t '2,…, t ' k}, k ≤ the total number of text segments, k is a configurable parameter, configured according to the total document length. When the total number of pages is less than 50, the value of k is 3~5; when the number of pages is greater than 50, the value of k is 5~8, to avoid excessive text leading to context fragmentation. t' i ∈ page text segment t ij It provides textual context for subsequent processing by the intelligent agent. In one embodiment, SpanBERT is fine-tuned based on a hybrid dataset of SQuAD 2.0 (Stanford Question Answering Dataset 2.0) and DocVQA (Document Visual Question Answering), with 10,000 fine-tuning iterations and a learning rate of 2e-5; S302: For image retrieval, based on a subset of the image corpus from the document corpus, for each page in the entire document... The original image object is processed by PaLI-X to generate dense visual embedding vectors, addressing the issues raised in the document. q The question is converted into a text prompt and then generated via PaLI-X. q The visual relevance embedding vector is calculated, and the Euclidean distance between this visual relevance vector and the visual embedding vectors of all page image objects in the entire document is calculated. The distance is then normalized to zero and used as the visual relevance score. After globally sorting all page image objects in the entire document in descending order of score, the top-ranked images are retrieved. k The most visually relevant image pages are selected to obtain the image page set. I q , represented as i k Indicates ranking k The highly relevant image pages are from the document. One of the pages in the original collection of pages.

[0021] S400: Utilizes the basic agent to process the set of text segments retrieved from S300. T q and / or image page collection I qA preliminary analysis is conducted to generate a preliminary answer. The specific steps are as follows: S401: Building the Basic Intelligent Agent B G This includes a text feature extraction component based on DistilBERT (Distilled BERT, a lightweight BERT model), built using the DistilBERT-base-uncased version. This version includes a 6-layer Transformer encoder with a total of 66M parameters. This lightweight version offers semantic understanding capabilities suitable for initial answer generation, avoiding inference delays caused by an overly large model. The component's output feature dimension is 768, directly meeting the requirements for fusion and adaptation with visual features. It is derived from a set of text segments. T q and user issues q The concatenated sequence (maximum length 512 tokens, truncated for semantic integrity if exceeding this limit) is input into this component and then semantically encoded layer by layer by the Transformer encoder. The semantic encoding process includes: the encoder performing basic semantic mapping on each token in the sequence that contains text segment semantic units and question keywords, generating independent basic semantic encoding results for each token; based on the independent basic semantic encoding results, the encoder achieves cross-token semantic interaction through a built-in multi-head attention mechanism, thereby capturing and generating a set of text segments. T q and user issues q The encoder integrates the basic semantic encoding results with the associated features to generate 768-dimensional text features, which are then fed into the subsequent attention mechanism module. Basic intelligent agents B G It also includes a ResNet-50-based visual feature extraction component, which is built using a ResNet-50 model pre-trained on the ImageNet dataset. It retains the first 49 layers of the feature extraction network, removes the original classification layer, and outputs 2048-dimensional visual features. Pre-trained weights ensure the accuracy of image detail extraction. Its input data is a pre-processed set of image pages. I qThe preprocessing includes scaling the image size to 224×224 pixels and normalizing RGB to [0, 1]. To achieve dimensional alignment with text features, a feature compression linear layer is constructed after the visual feature extraction component, consisting of two linear structures: 2048-dimensional to 1024-dimensional and then 1024-dimensional to 768-dimensional. The first layer uses ReLU activation function to alleviate gradient vanishing, while the second layer has no activation function to ensure linear mapping of features to 768 dimensions. Both layers are initialized using Xavier to avoid unstable training due to excessively large initialization parameters. After the standardized image is fed into the ResNet-50 network, 2048-dimensional original visual features are extracted. These original features flow directly into the feature compression linear layer, undergoing layer-by-layer dimensional compression processing to finally generate 768-dimensional visual features. These features are simultaneously fed into the attention mechanism module along with the text features.

[0022] Basic intelligent agents B G It also includes an attention mechanism module, which consists of a component with 8 attention heads, each with a dimension of 64, for a total dimension of 8×64=512. The padding parts of text features and visual features are masked with all zeros to avoid invalid features from participating in the calculation. The attention mechanism module performs a weighted fusion of the text features output by the text feature extraction component and / or the visual features output by the visual feature extraction component with a weight ratio of 7:3 to generate a 768-dimensional fused feature vector, which is directly input into the fully connected layer. The fully connected layer adopts a progressive structure of 768-dimensional to 512-dimensional, and then from 512-dimensional to 256-dimensional to the output layer. The output layer is a softmax activated text generation layer with vocab_size=30522, consistent with the tokenizer of DistilBERT, which is responsible for mapping the fused features to the preliminary answer of natural language.

[0023] In one embodiment, the training process of the basic agent includes two stages: pre-training and fine-tuning. During the pre-training phase, a total of 1 million samples were mixed from SQuAD2.0 (text-based question answering data, 60%), DocVQA (document-image question answering data, 30%), and VQAv2.0 (visual question answering data, 10%). The data cleaning rules included: filtering samples with text length < 10 characters and image resolution < 224×224, and establishing association labels for multimodal samples based on text segments and image regions. Training parameters include: batch size set to 16, optimizer set to AdamW (weight decay rate 1e-4), initial learning rate 2e-5, learning rate decay strategy set to linear decay, training to 80% of the total number of iterations and then decreasing to 1e-6, with a total of 8000 iterations. The loss function adopts a joint loss function design, including text semantic loss, visual feature loss and fusion feature loss; the text semantic loss adopts cross-entropy loss with a weight of 0.5; the visual feature loss adopts MSE loss with a weight of 0.3; the fusion feature loss adopts cosine similarity loss with a weight of 0.2. Through this joint loss function, the training objectives of the three dimensions of text, vision and fusion are kept consistent.

[0024] During the fine-tuning phase, the fine-tuning dataset consists of 5000 multimodal samples containing text, images, and standard answers, selected from the DocVQA test set. These samples are divided into training, validation, and test sets in an 8:1:1 ratio. To avoid losing pre-training knowledge, the parameters of the first 30 layers of DistilBERT and ResNet-50 are frozen, and fine-tuning is performed only on linear layers, attention mechanism modules, and fully connected layers. The initial answer accuracy on the validation set is used as the core metric. Accuracy is defined as the percentage of samples where the semantic similarity between the predicted answer and the standard answer is not less than 0.8. Fine-tuning stops if the validation set accuracy does not improve after three consecutive rounds, and the optimal model weights are saved as the inference weights of the basic agent. After fine-tuning, the initial answer accuracy of the basic agent on the test set is not less than 75%, and the inference speed is no more than 0.5 seconds per sample, which meets the design goal of quickly generating reliable initial answers. S402: Receives the text segment set obtained from S301 and S302. T q and image page collection I q Simultaneously, it receives questions regarding the document. q These serve as basic intelligent agents B G Input; S403: Basic Intelligent Agent B G The following process is used to obtain questions raised by users. q Related preliminary answers : Collection of text segments T q All text segments are concatenated into a complete context text. The built-in lightweight pre-trained language model DistilBERT (Bidirectional Encoder Representations from Transformers) is then invoked to combine this context text with the user question. q After concatenation, 768-dimensional text features are generated; The built-in ResNet-50 visual feature extraction component is invoked to extract features from the image page collection. I q2048-dimensional original visual features are extracted from the image, and then the visual features are subjected to the following dimension adaptation processing: the 2048-dimensional original visual features are compressed to 768-dimensional through a linear layer, and the dimensions are aligned with the 768-dimensional text features generated by DistilBERT. An attention mechanism is used to fuse the above text and image features. This attention mechanism is configured with 8 attention heads, each with a dimension of 64. During fusion, the feature importance is assigned according to a weight of 3:7 (visual:text). This weight is optimized by the SQuAD2.0 validation set to ensure that the fusion logic is based on text semantics and assisted by visual cues, and to avoid the problem being deviated from due to the excessive proportion of visual features. Finally, the basic agent uses a built-in fully connected layer to map and transform the fused feature vectors, outputting a preliminary answer. .

[0025] S500: This will retrieve user questions. q The retrieved collection of text segments T q and basic intelligent agents B G The preliminary answer given As another key information extraction intelligent agent The input is further analyzed to extract the most critical information essential for accurately answering the question. The specific steps are as follows: S501: Constructing and training a key information extraction agent B c ,include: Key information extraction intelligent agent B c Using the RoBERTa-base model based on the Transformer architecture as the core of text semantic understanding, the overall architecture of the RoBERTa-base model consists of three parts: an embedding layer, a Transformer encoder, and an output projection layer. The data transmission chain is as follows: the input text sequence is input to the embedding layer for dimension mapping, then passes through the Transformer encoder for layer-by-layer feature enhancement, and finally is input to the output projection layer for task adaptation; specifically including: The embedding layer is used to transform the input multimodal features into 768-dimensional embedding vectors in the format [batch_size, sequence_length, dimension]. This layer contains 30,522 neurons to ensure that the model can cover common text symbols in document question answering scenarios. Each neuron outputs 768-dimensional features, and the final output dimension is [batch_size, sequence_length, 768], where sequence_length is the length of the input text sequence, with a maximum of 512 tokens, and 768 dimensions is the unified hidden layer dimension of subsequent layers to ensure the consistency of signal transmission dimensions. Transformer encoder: It consists of 12 encoder layers, which are connected in sequence. The output of the previous layer is used as the input of the next layer. Each layer has a completely identical structure and independent parameters. The input and output dimensions of each layer are [batchsize, sequencelength, 768]. Each encoder layer contains a multi-head self-attention sub-layer and a feedforward neural network sub-layer. The feedforward neural network sub-layer receives the output of the multi-head self-attention sub-layer. The multi-head self-attention sublayer comprises 12 parallel attention heads, i.e., 12 independent attention computation units. Each attention head has an input dimension of 768 dimensions, which is split into 64 dimensions through linear projection. Each attention head independently calculates the semantic association weights between tokens, outputting 64-dimensional attention features. The output features of the 12 attention heads are concatenated and then restored to 768-dimensional features through a linear transformation involving 768 neurons, completing the fusion computation of multi-head attention. This sublayer captures the dependencies between tokens from different semantic perspectives through the parallel operation of 12 attention heads, thus avoiding the limitations of a single attention mechanism.

[0026] The feedforward neural network sublayer is used to non-linearly enhance the output features of the multi-head self-attention sublayer. It consists of two fully connected network layers: the first layer contains 3072 neurons, which transform the 768-dimensional feature vector into a 3072-dimensional feature vector through a linear transformation. The activation function is ReLU, which realizes feature dimension expansion and non-linear mapping; the second layer contains 768 neurons, which restore the 3072-dimensional feature vector to a 768-dimensional feature vector through a linear transformation, that is, restore it to the hidden layer dimension, to ensure that it matches the input dimension of the next layer encoder.

[0027] Before input to both the multi-head self-attention sub-layer and the feedforward neural network sub-layer, layer normalization is performed (normalization dimension is the hidden layer dimension 768). Specifically, the mean and variance of the 768-dimensional feature vector of each token are calculated independently, and the values ​​of each dimension are converted into standardized values ​​with a mean of 0 and a variance of 1, eliminating feature scale differences and stabilizing the distribution of input data. The feature vector output by the sub-layer is residually connected with the original input features to retain the original gradient signal. The gradient signal is retained through the shortcut path to alleviate the gradient vanishing problem in backpropagation, ensuring that the encoder is stably trained from the first layer to the last layer, and realizing the progressive deepening of semantic features.

[0028] The output projection layer receives the 768-dimensional output features from the Transformer encoder. It first performs a linear transformation to initially map the semantic features, converting the output format of the semantic features into the format required for the key information extraction task. Then, the mapped features are distributed in parallel to the following two functional sub-layers of the output projection layer: the text key information extraction sub-layer and the image key information association sub-layer, to complete the key information processing of different modalities in parallel. The text key information extraction sublayer contains 30,522 neurons. After receiving 768-dimensional basic features, it first maps the basic features to the label score corresponding to each token (i.e., the probability score of whether a token belongs to key information or non-key information) through linear transformation operation. Then, it integrates all tokens and their label scores in the original text order to form a label sequence. The sublayer calls the sequence optimization logic based on conditional random field technology. It uses the semantic association rules of adjacent and similar tokens in the label sequence to globally adjust the label score of each token, selects the label sequence that meets the semantic coherence, reduces the situation of contradictory labels of adjacent tokens, and finally outputs structured token-level key text information.

[0029] The semantic association rule of adjacent and similar tokens in the labeled sequence is that key information usually exists in the form of consecutive tokens. If a token is determined to be the starting point of key information, the probability that its subsequent adjacent tokens are key information should be significantly higher than that of non-key information. Conversely, if a token is non-key information, the tokens before and after it are also likely to be non-key information.

[0030] The image key information association sublayer contains 256 neurons. After receiving basic features, it directly completes the alignment of image feature dimensions and the matching of key regions through neurons, and outputs key image information that is adapted to the image features.

[0031] Key information extraction intelligent agent B c The training process is divided into two stages: pre-training and fine-tuning, specifically including: During the pre-training phase, DocVQA (Document Visual Question Answering), SQuAD2.0 (Stanford Question Answering Dataset2.0), and Flickr30K (Flickr30K Image-Caption Dataset) were mixed in a 4:3:3 ratio to form a total of 800,000 multimodal samples. Invalid samples with text annotation fragment length <3 characters or image annotation region coordinates exceeding image pixel boundaries are filtered out to ensure data quality; The training parameters are configured as follows: batch size is set to 32; the optimizer is AdamW, the weight decay rate is 1e-4; the initial learning rate is 3e-5, the learning rate decay strategy is cosine annealing decay, and the total number of iterations is 10,000. A joint loss function is adopted, which includes text annotation matching loss (CRF loss), image annotation classification loss (cross-entropy loss), and image annotation localization loss (MSE loss), to ensure the consistency of training objectives for multiple tasks; the weights are 0.6, 0.2, and 0.2 respectively.

[0032] During the fine-tuning phase, multiple (e.g., 3000) multimodal samples are selected from the DocVQA test set and divided into training, validation, and test sets in a 7:2:1 ratio. Fine-tuning strategies include: freezing the parameters of the embedding layer and the first 6 layers of the Transformer encoder in the RoBERTa-base model, and only fine-tuning the last 6 layers of the Transformer encoder, the output projection layer, and the text key information extraction sub-layer to avoid loss of pre-trained knowledge; The core metric is the F1 score on the validation set, which is calculated as the average of the text F1 and image F1 scores. The text F1 score measures the accuracy and completeness of the model in recognizing key information in text, while the image F1 score measures the accuracy and completeness of the model in locating key regions in images. During fine-tuning, this core metric is calculated after each training round. If there is no improvement after five consecutive rounds, fine-tuning stops, and the optimal model weights are saved.

[0033] After stopping fine-tuning, validate the model using a test set. The model must meet the following criteria: "Text F1 score ≥ 82%, Image F1 score ≥ 78%, Single-sample inference speed ≤ 0.3 seconds". If the criteria are not met, adjust the training strategy (such as optimizing the learning rate, expanding the dataset, etc.) and restart the fine-tuning process until the model converges and meets the criteria.

[0034] S502: When the text segment set If not empty, the questions raised in the document will be addressed. The retrieved collection of text segments The tokens are segmented into two discrete token sequences, each no longer than 512 tokens. An embedding layer maps each token in the two discrete token sequences to two corresponding multi-dimensional (768-dimensional) initial text vectors. These two initial text vectors are then combined with the initially generated answer. The multimodal features that together constitute the input embedding layer; key information extraction from the input; key information extraction agent. The embedding layer is converted into a 768-dimensional embedding vector; semantic enhancement is performed layer by layer by a 12-layer Transformer encoder, capturing semantic dependencies between tokens through 12 parallel attention heads, strengthening text fragment features strongly related to the question, and outputting semantically enhanced 768-dimensional deep semantic features; the output projection layer receives these deep semantic features, first generating a key information label score for each token (i.e., the probability score of whether a token belongs to key information or non-key information) through linear transformation, then integrating all tokens and their label scores in the original text order to form a label sequence, and then using the sequence optimization logic of Conditional Random Field (CRF) technology to globally adjust the label sequence using the semantic association law of "continuous distribution of key information", selecting factual content, data information, and logically related statements that conform to semantic coherence, and the agent... The final output contains token-level structured key text information that is crucial for accurately answering the question. This key text information is available for use. express; S503: When the image page set When not empty, extract key information from the agent. Questions raised in response to the document The collection of retrieved image pages and the initial generated answer Semantic features are aligned across modalities to obtain a multimodal input sequence; this sequence is then used to extract key information for the agent. The embedding layer converts the input vector into a 768-dimensional embedding vector, which is then input into the Transformer encoder. The Transformer encoder enhances the features of the input vector layer by layer. Each layer uses a multi-head self-attention sublayer to capture the semantic association between visual features and the question and preliminary answer, highlighting the core visual region features. These features are then processed by nonlinear enhancement, layer normalization, and residual connections in a feedforward neural network sublayer, ultimately outputting 768-dimensional deep features which are then fed into the output projection layer. The output projection layer passes the deep features to the image key information association sublayer. This sublayer, with 256 neurons, performs dimensional adaptation of visual features and key region matching, accurately locating core visual elements such as tables, charts, and key data points in the image, generating intermediate features containing region coordinates, category labels, and semantic descriptions. These intermediate features are then structurally integrated by the agent. The final output contains key image information that is crucial for accurately answering the question. This key image information is available... express.

[0035] S600: Targeting extracted key text information and related text segment collection Utilizing specialized text-based intelligent agents By analyzing the text content, a detailed text answer is generated. Available This indicates that the specific process includes: Professional text-based intelligent agents B T The core semantic understanding architecture is based on the RoBERTa-large model, which includes a 24-layer Transformer encoder and 1024-dimensional hidden layers. It is a professional text-based intelligent agent. B T Functionally, it consists of three parts: the input processing layer, the core encoding layer based on the RoBERTa-large model, and the output generation layer.

[0036] The input processing layer receives a set of text segments. T q Key text information T c User issues q The concatenated sequence has a maximum length of 1024 tokens. If the length exceeds the limit, it will be truncated according to the semantic integrity of the paragraph. It is encoded by the tokenizer (text segmentation encoder) that is paired with the RoBERTa-large model and converted into an input format that is adapted to the model, outputting 512-dimensional initial text features. The core encoding layer based on the RoBERTa-large model serves as the core processing component of the agent. It contains 24 Transformer encoder layers and 1024-dimensional hidden layers. It captures cross-paragraph semantic dependencies through 16 attention heads, strengthens the association weight between core information and the question, and performs layer-by-layer semantic enhancement on the initial text features output by the input processing layer, ultimately outputting 1024-dimensional deep semantic features. Output generation layer as a professional text intelligent agent B T Deep semantic features are mapped to natural language output through two-layer linear transformation (1024-dimensional to 512-dimensional to vocab_size). The vocab_size is consistent with the tokenizer vocabulary size of the RoBERTa-large model, ensuring the fluency and accuracy of text generation.

[0037] Professional text-based intelligent agents B T The training includes two phases: pre-training and fine-tuning. During the pre-training phase, SQuAD2.0, DocVQA text subsets, and RACE long text understanding datasets were mixed in a ratio of 6:3:1 to form a massive amount of pre-training samples to cover long text semantic understanding scenarios in multiple domains. Filter out invalid samples with text length < 20 characters and answers that are not related to the context, and preprocess the pre-trained samples according to the format of "question, context, standard answer"; The training parameters are set as follows: batch size is 16; the optimizer is AdamW, the weight decay rate is 1e-4; the initial learning rate is 3e-5, the learning rate decay strategy is linear decay, and the total number of iterations is 6000. Based on the above training parameters, the pre-processed pre-training samples are input into the professional text intelligent agent in batches. B T During training, the model semantically encodes the input text sequence and outputs the predicted answer. At this time, the cross-entropy loss function is used to quantify the semantic difference between the model's predicted answer and the standard answer provided by the sample, and obtain a specific error value. This error value is then passed to the AdamW optimizer, which updates the model's trainable parameters through backpropagation, allowing the model to gradually learn the ability to "accurately match the question with the context and generate semantically consistent answers" until the total number of iterations is reached, at which point the pre-training phase ends.

[0038] During the fine-tuning phase, the Longformer-QA dataset and a custom document question-answering dataset were used to select 3,000 long text question-answering samples, which were then divided into training set, validation set, and test set in an 8:1:1 ratio. The following fine-tuning strategy is adopted: freeze the parameters of the first 12 Transformer encoder layers of the RoBERTa-large model, and only fine-tune the last 12 encoder layers and the output generation layer to avoid the loss of pre-trained knowledge, while adapting to the custom document question answering scenario; The F1 score of the validation set text is used as the core evaluation metric. This core metric is calculated after each round of training. If there is no improvement after 4 consecutive rounds, fine-tuning is stopped and the optimal model weights are saved. After stopping fine-tuning, validate the model using a test set. The model must meet the following requirements: text F1 score ≥ 85% and single-sample inference speed ≤ 0.8 seconds. If the requirements are not met, adjust the training strategy (such as optimizing the learning rate, expanding the dataset, etc.) and restart the fine-tuning process until the model converges and meets the requirements.

[0039] After the model training is complete, it will address the questions raised in the document. q Text segment collection and key text information T c Input the trained professional text intelligent agent B T The intelligent agent first leverages the semantic understanding capabilities of the core coding layer (RoBERTa-large) to process the set of text segments. T q Perform semantic segmentation and logical organization, combining key textual information. T c The core information paragraphs are located, and the model's 16 built-in attention heads are used to focus on the questions posed in the document. q The relevant content is analyzed through causal analysis and logical deduction of factual verification, and finally, a detailed text answer is generated through the output generation layer. , It includes problem analysis, logical deduction process, data support, and clear conclusions.

[0040] S700: For receiving and extracting key image information and related image context Utilizing specialized image intelligence agents By analyzing image content, it generates detailed visual answers, which can be used... express.

[0041] Professional Image Intelligent Agent The construction methods include: The image feature extraction architecture, based on the ResNet-50 model, consists of the following three layers: Input processing layer: receives a collection of image pages. I q Key image information I cThe input image is preprocessed as follows: the image is uniformly scaled to a size of 224×224 pixels, and the pixel values ​​are normalized to zero and one to eliminate the influence of image size and brightness differences on the model. Finally, a 3-channel (RGB) normalized image is output. The core feature extraction layer (i.e., the ResNet-50 model) contains 50 layers of convolutional and residual connection structures. It captures low, medium and high-level features of the image through layer-by-layer convolutional operations, and finally outputs 2048-dimensional deep image semantic features to enhance the feature representation of key regions of the image (such as table cells and chart data areas in documents). Output parsing layer: Through two layers of linear transformation (2048-dimensional to 1024-dimensional, and then to the output dimension), the deep image semantic features are mapped to the image key information parsing results. The output dimension matches the number of labeled categories of image key information (such as "table", "chart", "text region", "key data point"), and finally outputs structured image key information containing region coordinates, content category and semantic description.

[0042] Professional Image Intelligent Agent The training process includes: 1. Pre-training phase A massive amount of multi-scene image samples for pre-training were constructed in a 6:3:1 ratio, including COCO (CommonObjectsinContext, a general object detection dataset), Flickr30KImage-Caption (an image-text annotation dataset), and DocBank (a document image annotation dataset), covering general image and document image scenarios; Filter out invalid samples with blurry images, missing annotations, or invalid region coordinates; scale all images to 224×224 pixels; and standardize them according to the "image-key region annotation-semantic description" format to ensure data quality. The training parameters are set as follows: batch size is 32; the optimizer is AdamW with a weight decay rate of 1e-4; the initial learning rate is 2e-5; the learning rate decay strategy is cosine annealing; and the total number of iterations is 5000. Based on the above training parameters, the preprocessed multi-scene image samples are input into the professional image intelligent agent in batches. B I Training is then performed. During training, the image data of each sample is input into the core feature extraction layer of the model to learn image features. The "key region annotation (category + coordinates)" that comes with the sample serves as the standard answer and is compared with the prediction result (predicted category + predicted coordinates) output by the model. The loss function employs a joint loss function to simultaneously accomplish the two core tasks of image key region classification and region coordinate localization. Specifically, it comprises two parts: first, the image key region classification loss, using cross-entropy loss, is used to determine whether an image region belongs to a predefined category such as "table," "chart," or "text area," with a weight of 0.7; second, the region localization loss, using MSE loss, is used to accurately predict the coordinate positions of key regions, such as the four corner coordinates of a table, with a weight of 0.3. This weight allocation prioritizes classification accuracy while also considering localization precision, ensuring consistency in the training objectives across multiple tasks.

[0043] The difference between the sample prediction result and the standard answer is quantified by the loss function to obtain the specific error value. Then, the error value is backpropagated through the optimizer to update the trainable parameters of the model, so that the model gradually masters the core capabilities of feature extraction, category judgment and coordinate localization, until the total number of iterations is completed and the pre-training stage ends.

[0044] 2. Fine-tuning stage We selected a massive number of document image samples from the DocVQA (DocumentVisualQuestionAnswering) image test set, supplemented them with custom document image annotation samples, and divided them into training set, validation set, and test set in an 8:1:1 ratio. The following fine-tuning strategy is set: freeze the parameters of the first 20 convolutional layers of the ResNet-50 model to retain the general image feature extraction capability, and only fine-tune the parameters of the last 30 convolutional layers and the output parsing layer parameters to avoid the loss of pre-training knowledge, while adapting to the specific features of the document image. The F1 score of the images in the validation set is used as the core evaluation metric. This metric is calculated after each round of training. If there is no improvement after 4 consecutive rounds, fine-tuning is stopped, and the model weights at the highest metric during training are saved. If the fine-tuned model achieves an F1 score ≥ 80% on the test set and a single-image inference speed ≤ 1 second, then it is considered a professional image-based intelligent agent. If the training is successful, otherwise the learning rate can be adjusted or the sample size of the document images can be increased, and the fine-tuning process can be restarted.

[0045] After training is complete, questions will be raised regarding the document. q The collection of retrieved image pages I q and key image information I C Input the trained professional image agent B I The visual answer is generated through the following process: First, based on professional image intelligent agents B I The core feature extraction layer extracts the context of the retrieved images.I q Perform key area location and screening, combined with key image information. I C The core analysis area is identified; then, the chart data within the area is extracted through the output parsing layer and converted into structured text, while simultaneously parsing the core content of the image; finally, the relationship between the image content, structured data, and user questions (such as data corroboration and concept correspondence) is analyzed to generate a detailed visual answer. , It includes image content analysis, chart data conversion results, explanations of the relationship between visual information and the problem, and supporting conclusions.

[0046] S800: Receives and analyzes data from the basic agent. B G Professional text intelligent agent and professional image intelligence agents B I The answer, using the responding intelligent agent B s This ultimately generates a comprehensive answer that can be used. express.

[0047] Answering agent B s The following three-layer functional architecture is adopted: Input fusion layer: receives user questions q The detailed text answers output by the professional text intelligent agent and the visual answers output by the professional image intelligent agent are concatenated and standardized in the format of "question-text information-visual information" and converted into a unified feature format that the model can process, with a dimension of 768. Core decision layer: Employs a lightweight Transformer encoder with 6 layers and 8 attention heads to perform correlation analysis on the vectors output from the input fusion layer, filtering those relevant to user questions. q Highly relevant core information, handling a small number of conflicting information, and strengthening the logical coherence of the answer; Output Integration Layer: Through a linear transformation and natural language generation module, the fusion features output by the core decision layer are converted into a structured final answer that conforms to the user's reading habits. The output format is "problem restatement - core conclusion - text support - visual support - supplementary explanation", ensuring that the answer is clear, complete and verifiable.

[0048] The answering agent The training process includes: 1. Pre-training phase A massive multimodal sample was constructed by mixing DocVQA (Document Visual Question Answering Dataset) and Flickr30KImage-Caption (Image-Text Annotation Dataset) in a 6:4 ratio, covering collaborative question answering scenarios that combine text and images; The massive multimodal samples were preprocessed to filter out invalid samples with missing answers or no correlation between text and image information, and standardized according to the format of "user question - text answer - visual answer - standard unified answer" to ensure data quality. The training parameters are set as follows: batch size is 32, the optimizer is AdamW, the weight decay rate is 1e-4, the initial learning rate is 3e-5, the learning rate decay strategy is linear decay, and the total number of iterations is 4000. Based on the batch size of 32, the preprocessed multimodal samples are divided into training batches and input into the response agent batch by batch for pre-training. During training of each batch, the "user question-text answer-visual answer" in the sample is input into the input fusion layer of the model to generate unified features, which are then passed to the core decision layer; the "standard unified answer" in the sample is used as the benchmark for the model output and is used for subsequent loss calculation. The loss function adopted is the cross-entropy loss function, which quantifies the semantic difference between the fused answer generated by the model and the "standard unified answer" provided by the sample, and obtains a specific error value. This error value is passed to the AdamW optimizer, and the trainable parameters of the model are updated through backpropagation, so that the model can gradually learn the ability to "integrate text and visual information and generate a semantically consistent unified answer" until 4000 iterations are completed, and the pre-training stage ends.

[0049] 2. Fine-tuning stage Multiple multimodal samples containing mixed text and image scenarios were selected from the custom document question-answering dataset and divided into training set, validation set and test set in a ratio of 8:1:1. The sample format is: "User question, detailed text answer output by professional text agent - detailed image answer output by professional image agent - manual labeled standard final answer"; The following fine-tuning strategy is set: freeze the parameters of the first 3 layers of the core decision layer Transformer encoder, and only fine-tune the parameters of the last 3 layers and the output integration layer to avoid the loss of pre-trained knowledge, while adapting to custom document question answering scenarios. The "F1 score" of the validation set is used as the core indicator to measure the accuracy of the generated answer and the manually labeled standard answer. This indicator is calculated after each round of training. If it does not improve after 4 consecutive rounds, the fine-tuning is stopped and the optimal model weights are saved. Training is completed when the "F1 score of the answer is ≥88% and the final answer generation speed per sample is ≤1.2 seconds" is met on the test set. Otherwise, the learning rate is adjusted or the fine-tuning sample size is increased, and the fine-tuning process is restarted.

[0050] After the agent is trained, the methods for generating the final answer include: The user's original question q The input fusion layer of the answering agent is composed of detailed text answers output by a professional text agent and visual answers output by a professional image agent. This layer completes the format splicing and feature standardization, and outputs 768-dimensional unified multimodal features to the core decision layer. The core decision layer completes the correlation analysis, consistency verification and core information filtering of text and visual information through a 6-layer Transformer encoder, eliminates information redundancy or minor conflicts, and outputs the fused core semantic features to the output integration layer. After being processed by the linear transformation and natural language generation module, the output integration layer outputs the final unified answer in a structured format of "problem restatement - core conclusion - text support - visual support - supplementary explanation". In one embodiment, if the final answer is missing information (such as not covering the user's core questions), it is fed back to a professional text / image agent to re-extract the key information and repeat the above steps; if the information is complete and logically coherent, the final answer is output directly.

[0051] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0052] This invention also provides a document question-answering system based on multi-agent collaboration, such as... Figure 2 As shown, it includes: Document preprocessing module: Used to preprocess user-provided documents and convert them into standardized datasets; Multimodal retrieval context module: used to retrieve from the standardized dataset the set of text segments and / or image pages most relevant to the question posed for the document; Preliminary answer generation module: This module takes the set of text segments and / or the set of image pages as input to the basic agent, performs a preliminary analysis based on the questions posed in response to the document, and outputs a preliminary answer. Key information extraction module: used to input the questions raised for the document, the set of text segments and / or the set of image pages, and the preliminary answers generated by the basic agent into the key information extraction agent to obtain key text information and / or key image information; Answer generation module: used to input the question posed for the document, the preliminary answer, the key text information and / or the key image information into the answering agent, and generate the final answer to the question.

[0053] In one embodiment, a professional text processing module is further included, which is used to input the question, the set of text segments, and the key text information into a professional text agent to obtain a detailed text answer; the detailed text answer is used to input the answer agent to generate a final answer to the question.

[0054] In one embodiment, a specialized image processing module is further included, which is used to input the question, the set of image pages, and the key image information into a specialized image agent to obtain a detailed visual answer; the detailed visual answer is used to input the answering agent to generate a final answer to the question.

[0055] In one embodiment, the system further includes the user input module, which is used to obtain documents uploaded by the user and questions raised by the user regarding the uploaded documents.

[0056] In one embodiment, the preprocessing process of the document preprocessing module includes: The text of each page in the document is extracted using OCR technology and / or PDF parsing technology, and a subset of the text corpus is constructed by paragraph. The original image objects corresponding to each page of the document are retained, and the original image objects of all pages are used to construct an image corpus subset; the text corpus subset and the image corpus subset together constitute the standardized dataset.

[0057] In one embodiment, the multimodal retrieval context module includes a text retrieval unit. The text retrieval unit uses the SpanBERT model to perform semantic indexing on each text segment in the text corpus subset, calculates the semantic relevance score of each text segment to the question, sorts the text segments in descending order of the scores, and selects the top-ranked text segments as the set of text segments most relevant to the question posed for the document.

[0058] In one embodiment, the multimodal retrieval context module includes an image retrieval unit. The image retrieval unit uses the PaLI-X model to generate visual embedding vectors for the original image objects in the image corpus subset, calculates the Euclidean distance between the visually associated embedding vectors and the question, normalizes the Euclidean distance to obtain a relevance score, and selects the top-ranked image pages as the set of image pages most relevant to the question posed in the document after sorting them in descending order of the scores.

[0059] In one embodiment, the basic intelligent agent includes a text feature extraction component, which is used to perform semantic encoding on the sequence consisting of a set of text segments and the question, capture the association features between the set of text segments and the question based on the semantic encoding, and fuse the semantic encoding result and the association features to obtain multidimensional text features.

[0060] In one embodiment, the basic intelligent agent includes a visual feature extraction component and a feature compression linear layer for feature dimension alignment. The input data of the visual feature extraction component is a preprocessed set of image pages. The multidimensional visual features output by the visual feature extraction component are output to the feature compression linear layer for dimension alignment processing, and finally multidimensional visual features with the same dimensions as the multidimensional text features are generated.

[0061] In one embodiment, the method by which the basic agent outputs a preliminary answer includes: When the image page set is empty, the multidimensional text features are the preliminary answer output by the basic agent; When the set of text segments is empty, the multidimensional visual features are the preliminary answer output by the basic agent; Otherwise, the multidimensional visual features and multidimensional text features are fused through an attention mechanism to obtain a fused feature vector. The fully connected layer of the basic agent maps and transforms the fused feature vector to output a preliminary answer.

[0062] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0063] This invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the various steps of the method described in this invention.

[0064] This invention also provides a non-transitory computer-readable storage medium storing a computer program, which includes program instructions that, when executed by a processor, implement the various steps of the method described in this invention, which will not be elaborated further here.

[0065] The computer-readable storage medium can be the data transmission apparatus or the internal storage unit of a computer device provided in any of the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium can also be the external storage device of the computer device, such as the plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on the computer device.

[0066] Furthermore, the computer-readable storage medium may include both internal storage units and external storage devices of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that is to be output or has already been output.

[0067] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0068] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0069] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0070] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0071] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

Claims

1. A document question-answering system based on multi-agent collaboration, characterized in that, include: Document preprocessing module: Used to preprocess user-provided documents and convert them into standardized datasets; Multimodal retrieval context module: used to retrieve from the standardized dataset the set of text segments and / or image pages most relevant to the question posed for the document; Preliminary answer generation module: used to input the set of text segments and / or the set of image pages into the basic agent, perform preliminary analysis in conjunction with the question, and output a preliminary answer from the basic agent; Key information extraction module: used to input the question, the set of text segments and / or the set of image pages, and the preliminary answer generated by the basic agent into the key information extraction agent to obtain key text information and / or key image information; Answer generation module: used to input the question, the preliminary answer, the key text information and / or the key image information into the answering agent, and generate the final answer to the question.

2. The document question-answering system based on multi-agent collaboration as described in claim 1, characterized in that, It also includes a professional text processing module, which is used to input the question, the set of text segments and the key text information into a professional text agent to obtain a detailed text answer; the detailed text answer is used to input the answer agent to generate the final answer to the question.

3. The document question-answering system based on multi-agent collaboration as described in claim 1 or 2, characterized in that, It also includes a professional image processing module, which is used to input the question, the set of image pages and the key image information into a professional image agent to obtain a detailed visual answer; the detailed visual answer is used to input the answering agent to generate the final answer to the question.

4. The document question-answering system based on multi-agent collaboration as described in claim 1, characterized in that, The preprocessing process of the document preprocessing module includes: extracting the text from each page of the document and constructing a subset of the text corpus in the standardized dataset by text segment.

5. The document question-answering system based on multi-agent collaboration as described in claim 1 or 4, characterized in that, The preprocessing process of the document preprocessing module includes: extracting the original image objects corresponding to each page of the document, and the original image objects of all pages constitute a subset of the image corpus in the standardized dataset.

6. The document question-answering system based on multi-agent collaboration as described in claim 4, characterized in that, The multimodal retrieval context module includes a text retrieval unit. The text retrieval unit performs semantic indexing on each text segment in the text corpus subset, calculates the semantic relevance score of each text segment to the question, sorts the text segments in descending order of the scores, and selects the top-ranked text segments as the set of text segments most relevant to the question.

7. The document question-answering system based on multi-agent collaboration as described in claim 5, characterized in that, The multimodal retrieval context module includes an image retrieval unit. The image retrieval unit generates visual embedding vectors based on the original image objects in the image corpus subset, calculates the Euclidean distance between each visual embedding vector and the visual association embedding vector of the question, normalizes the Euclidean distance to obtain a relevance score, and selects the top-ranked original images as the set of image pages most relevant to the question after sorting them in descending order of the scores.

8. The document question-answering system based on multi-agent collaboration as described in claim 1, characterized in that, The basic intelligent agent includes a text feature extraction component, which is used to perform semantic encoding on the sequence consisting of a set of text segments and the question, capture the association features between the set of text segments and the question based on the semantic encoding, and fuse the semantic encoding results and association features to obtain multidimensional text features.

9. The document question-answering system based on multi-agent collaboration as described in claim 8, characterized in that, The basic intelligent agent includes a visual feature extraction component and a feature compression linear layer for feature dimension alignment. The input data of the visual feature extraction component is a preprocessed set of image pages, and the output data of the visual feature extraction component is multidimensional visual features. The multidimensional visual features are then output to the feature compression linear layer for dimension alignment processing, ultimately generating multidimensional visual features with the same dimensions as the multidimensional text features.

10. The document question-answering system based on multi-agent collaboration as described in claim 9, characterized in that, The methods by which a basic intelligent agent outputs a preliminary answer include: When the image page set is empty, the multidimensional text features are the preliminary answer output by the basic agent; When the set of text segments is empty, the multidimensional visual features are the preliminary answer output by the basic agent; Otherwise, the multidimensional visual features and multidimensional text features are fused through an attention mechanism to obtain a fused feature vector. The fully connected layer of the basic agent maps and transforms the fused feature vector to output a preliminary answer.

11. A document question-answering method based on multi-agent collaboration in the system as described in claim 1, characterized in that, include: Preprocess the user-provided documents to convert them into a standardized dataset; From the standardized dataset, retrieve the set of text segments and / or the set of image pages most relevant to the questions posed for the document; The set of text segments and / or the set of image pages are input into the basic agent, and a preliminary analysis is performed in conjunction with the question. The basic agent then outputs a preliminary answer. Input the question, the set of text segments and / or the set of image pages, and the preliminary answer generated by the basic agent into the key information extraction agent to obtain key text information and / or key image information; The question, the preliminary answer, the key text information, and / or the key image information are input into the answering agent to generate the final answer to the question.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the document question-answering method based on multi-agent collaboration as described in claim 11.

13. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the document question-answering method based on multi-agent collaboration as described in claim 11.

Images