Multi-modal intelligent question answering system and method based on large model

Abstract

The invention discloses a multi-modal intelligent question answering system and method based on a large model. The system comprises a signaling processing module, a document preprocessing module, a voice preprocessing module, a retrieval recall module, an RAG processing chain module and a knowledge base module. The method comprises the following steps: (1) a knowledge base offline construction step; (2) a knowledge base online construction step; (3) a real-time question and answer step; according to the system and the method provided by the invention, personalized intelligent question and answer service can be provided for audiences by combining materials which are uploaded by a speaker in advance and contain rich contents such as characters, pictures and charts and real-time presentation contents and utilizing multi-mode presentation information such as characters, pictures and voice, so that instant questions of the audiences can be answered, and the audiences can be more interested. Extensible knowledge supplementation can be provided according to the presentation content and materials.

Description

Technical Field

[0001] The present invention relates to a large-model-based multimodal intelligent question-answering system and method, belonging to the field of information technology, and in particular to the field of intelligent question-answering technology for lecture scenarios. Background Art

[0002] In today's digital age, real-time communication (RTC) systems are booming at an unprecedented rate, profoundly changing how people communicate and collaborate. From everyday remote work and online education to various business meetings and academic exchanges, audio and video communication has become an indispensable and critical tool. In recent years, with the rapid development of artificial intelligence (AI) and large-scale modeling technologies, these audio and video communication systems have increasingly been integrated with AI-powered intelligent assistants to improve programming and office efficiency. These intelligent assistants strive to provide users with more intelligent and personalized services. The real-time subtitle feature, leveraging speech recognition and natural language processing, enables real-time conversion between Chinese and English subtitles, breaking down language barriers and making cross-border conference communication smoother and more seamless. Using speech recognition technology, voice content in meetings can be converted into text transcripts in real time, allowing users to easily review key points and avoid missing important information. Furthermore, the intelligent minutes feature automatically organizes meeting content, extracts key information, and generates clearly organized minutes, helping users quickly grasp the core of the meeting and significantly improving meeting efficiency. Furthermore, AI-enabled intelligent meeting management, such as intelligent meeting scheduling, precise reminders, and comprehensive record-keeping, further optimizes meeting processes and provides users with an unprecedentedly convenient experience. However, the intelligent assistant features of current audio and video communication systems on the market mostly focus on intelligent summarization and minutes, intelligent transcription and translation, and intelligent management and collaboration, primarily serving commercial office scenarios.

[0003] Lectures are an important means of disseminating knowledge and communicating. Audiences not only seek to gain basic knowledge from lectures, but also crave a deeper understanding of topics they care about. Traditional lecture models, which rely on one-way information transmission and fixed interaction patterns, have numerous limitations, including limited time, an inability to meet the personalized needs of audiences, and a lack of real-time interaction. These one-way communication methods are unable to meet the growing interactive needs and personalized content demands of modern audiences.

[0004] For lecture scenarios, how to combine large model technology with modern audio and video communication technology to construct an intelligent question-and-answer system that integrates multimodal information. This system can provide customized interaction and question-and-answer services to the audience based on the lecture content. This has become a technical problem that urgently needs to be solved in the field of intelligent question-and-answer. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to invent an intelligent question-answering system and method with multimodal information fusion, which can effectively solve the problems of insufficient interaction and differences in personalized needs in traditional lectures, so that the audience can ask questions and get answers in real time, enhance the sense of participation and lecture effect, and provide a simple access method to meet the real-time, interactive and personalized needs of modern lecture scenes.

[0006] To achieve the above objectives, the present invention proposes a multimodal intelligent question-answering system based on a large model, which includes the following modules:

[0007] The signaling processing module is the core module of the system. It is responsible for receiving, parsing and processing requests from the signaling server and coordinating other modules in the system to complete corresponding tasks. Its main functions include:

[0008] (1) Receive lecture material upload requests, lecture start / end requests, user question requests, and multi-round dialogue list acquisition requests sent by the signaling server; parse the request content; extract key parameters, including lecture room ID, user ID, question content, etc.;

[0009] (2) Based on the request type sent by the signaling server, the processing flow of other modules in the system is triggered, specifically:

[0010] Speech material upload request: triggers the document pre-processing module to process the uploaded speech materials;

[0011] Lecture start / end request: coordinates the speech preprocessing module to receive real-time speech stream and dynamically updates the knowledge base;

[0012] User question request: the retrieval and recall module and the RAG processing chain module are called to generate the answer and return it to the user;

[0013] Request to obtain a multi-turn conversation list: This triggers the database module, queries the historical question-answer database, and returns the multi-turn conversation context information.

[0014] (3) Return the processing result to the client through the signaling server;

[0015] The document preprocessing module has the following main functions: (1) file decompression: receiving the compressed package of lecture materials sent by the signaling processing module and decompressing the PDF, PPTX, DOCX and other documents therein; (2) page-to-image conversion: converting the pages in the decompressed document into PNG image blocks with appropriate resolution, preserving the font and layout features; (3) metadata annotation: adding metadata to each image block to record additional information of the document, including page number, title, chart type, etc.; the processing results of the document preprocessing module are sent to the retrieval and recall module;

[0016] The speech preprocessing module receives speech streams in real time and transcribes them into text, ensuring that speech content can be accurately recognized and converted into text, providing basic support for subsequent processing. This module supports real-time speech stream processing to dynamically update the knowledge base in the subsequent retrieval and recall modules, ensuring that the question and answer content is synchronized with the progress of the speech.

[0017] Retrieval and Recall Module: This module's main function is to build the retrieval-enhanced knowledge base required by the system and complete online retrieval. This module's functions are divided into two stages: online and offline. The offline stage is responsible for processing pre-input lecture materials containing multimodal information after being processed by the document preprocessing module; the online stage functions to process real-time incoming lecture content and user queries.

[0018] RAG processing chain module: This module uses the user's question and the image blocks returned by the retrieval and recall module as context. It then feeds this context and the historical question-answer pairs retrieved by the database module into the multimodal large model via the Prompt template to generate high-quality answers.

[0019] Knowledge base module: The function of this module is to build the system's knowledge base through the storage, retrieval and management of multi-round dialogue history of embedded vectors; this module receives the embedded vectors and their metadata generated by the retrieval and recall module through an efficient data storage and query mechanism, and stores them in the database to ensure that the system can quickly respond to query requests and support the retrieval of embedded vectors and the contextual history memory of multi-round dialogues; this module includes two databases, namely the speech embedding vector database and the historical question and answer pair database; the speech embedding vector database is used to store the embedded vector data generated by the speech for retrieval, and the historical question and answer pair database is used to store the contextual information of multi-round dialogues.

[0020] The speech preprocessing module processes real-time speech streams based on streaming speech recognition technology and an efficient data processing framework. The processing process includes the following three main steps:

[0021] (s1) Voice reception: Receives real-time voice streams via UDP packets and extracts the RTP payload data therefrom. Specifically, the signaling processing module parses the SDP information from the speech start request and extracts the IP address, port number, and other information required for audio reception. Receives the real-time voice stream forwarded by the SFU server in the form of RTP packets, parses the UDP packets using the UDP receiving and parsing tool library, extracts the RTP payload data therefrom, and stores the extracted voice data in the audio segment list.

[0022] (s2) Voice format conversion: converting the received voice data into an audio format suitable for voice recognition, performing format conversion on the voice data to ensure that the voice data meets the requirements of voice recognition, and converting the voice data into an audio segment object in WAV format;

[0023] (s3) Speech transcription: Use streaming speech recognition technology to transcribe speech content into text in real time. Periodically extract speech data from the audio segment list, merge and save it as a temporary WAV file, perform speech transcription on the temporary WAV file, and generate the transcribed text content.

[0024] The specific processing content of the offline stage of the retrieval and recall module is as follows: first, multi-vectors are generated through delayed interaction, and the multimodal materials input in the offline stage are pre-mapped to a unified vector space through multimodal alignment processing, that is, text embedding and visual embedding are mapped to a unified vector space to ensure that information of different modalities can be compared for similarity; then, the centroid is selected and the embedded vector is compressed by clustering the centroid to reduce the storage space occupied by the embedded vector and retain sufficient semantic information. At the same time, the embedded vector cluster center and the restored vector information are stored for use in the online stage.

[0025] The specific processing content of the online phase of the retrieval and recall module is as follows: (a) processing the real-time incoming lecture content and user queries, and quickly screening candidate documents through the clustering centroid strategy to generate the final retrieval results; encoding the real-time incoming lecture content, generating text embedding, mapping the text embedding to the unified vector space in the offline phase, and performing online data stream clustering on the newly generated embedding vector; (b) encoding the user query, generating query embedding, and mapping it to the unified vector space in the offline phase, quickly screening candidate vectors through the clustering centroid strategy, performing residual decompression on the screened candidate vectors, restoring the original embedding vector representation, and using a delayed interaction mechanism to calculate the fine-grained similarity between the query and the candidate documents. For each embedding vector in the query, calculate its maximum similarity with all embedding vectors in the candidate documents, add the similarity scores of all embedding vectors in the query, generate a total similarity score between the user query and the candidate documents, sort the candidate documents according to the total similarity score, and return the most relevant documents.

[0026] The present invention also proposes a multimodal intelligent question-answering method based on a large model, the method comprising the following steps:

[0027] (1) Offline knowledge base construction step: Before the lecture begins, the system receives an initialization request from the signaling server, processes the lecture materials uploaded by the user, and builds a knowledge base to support subsequent retrieval;

[0028] (2) Online knowledge base construction steps: During the lecture, the system continuously receives the real-time audio stream forwarded by the SFU, and transcribes the voice content into text in real time through automatic speech recognition technology. After processing, it is dynamically updated to the knowledge base to ensure that the knowledge base content is synchronized with the progress of the lecture;

[0029] (3) Real-time question-answering step: When a user submits a question request through the client, the signaling server forwards the question to the system. The system first retrieves the context fragments related to the question from the knowledge base, and then calls the external large model for semantic understanding and answer generation. Finally, the generated answer is returned to the user who asked the question through the signaling server, realizing efficient and accurate real-time question-answering interaction.

[0030] The step (1) of building the knowledge base offline specifically includes the following steps:

[0031] (11) The user sends an initialization request including the upload of lecture materials to the signaling server through the user client; the signaling server sends the lecture material upload request to the signaling processing module, and sends the lecture materials to the signaling processing module in a streaming upload manner;

[0032] (12) The signaling processing module triggers the document pre-processing module to process the uploaded lecture materials and send the processing results to the retrieval and recall module;

[0033] (13) The offline phase function of the retrieval and recall module processes the pre-input speech materials containing multimodal information after being processed by the document preprocessing module to generate embedding vectors and their metadata;

[0034] (14) The knowledge base module receives the embedding vector and its metadata generated by the retrieval and recall module and stores them in the speech embedding vector database in the knowledge base module;

[0035] (15) After completing the above operations, the signaling processing module returns the processing results to the user client through the signaling server.

[0036] The step (2) of building the knowledge base online specifically includes the following steps:

[0037] (21) The user sends a speech start request to the signaling server through the client. The signaling server sends a media channel creation request to the SFU server. The SFU server creates and returns media channel information to the signaling server. The signaling server forwards this information to the signaling processing module. The signaling processing module triggers the voice preprocessing module to start receiving voice streams. Finally, it sends a successful start response to the signaling server.

[0038] (22) The voice preprocessing module completes the process of converting the voice RTP stream into text through voice reception, voice format conversion and voice transcription, and transmits the transcribed text content to the retrieval and recall module in real time;

[0039] (23) The knowledge base module reads the embedding vectors and metadata generated by the offline phase of the retrieval and recall module from the lecture embedding vector database;

[0040] (24) The retrieval and recall module processes the speech text content transmitted in real time by the speech preprocessing module in the online stage to obtain the updated embedding vector and its metadata;

[0041] (25) The retrieval and recall module triggers the knowledge base module to store the updated embedding vector and its metadata into the speech embedding vector database in the knowledge base module;

[0042] (26) When the speech ends, the signaling processing module receives the speech end request sent by the signaling server, sends a successful end response, and initiates a stop operation to the voice preprocessing module and the retrieval and recall module.

[0043] The step (3) of real-time question and answer includes the following steps:

[0044] (31) The user sends a request to obtain a multi-round conversation list to the signaling server through the user client. After receiving the request, the signaling server forwards the request to the signaling processing module.

[0045] (32) The signaling processing module receives the multi-round conversation list acquisition request, parses the request content to extract key parameters, including user ID and speech ID, triggers the knowledge base module and transmits these key parameters;

[0046] (33) The knowledge base module retrieves the corresponding multi-round dialogue context information from the historical question-answer pair database using the user ID and speech ID passed by the signaling processing module, and passes it to the signaling processing module;

[0047] (34) The signaling processing module responds to the multi-round conversation list acquisition request initiated by the signaling server, returns the context information of the multi-round conversation to the signaling server, and the signaling server returns it to the client of the user who asked the question;

[0048] (35) The user sends a user question request to the signaling server through the user client. After receiving the question request, the signaling server forwards the question request to the signaling processing module;

[0049] (36) The signaling processing module receives the question request, parses the request content to extract key parameters, including user ID, lecture ID and question content, and triggers the retrieval and recall module, knowledge base module and RAG processing chain module;

[0050] (37) The knowledge base module reads the corresponding embedding vector and metadata generated by the previous retrieval and recall module from the speech embedding vector database based on the user ID and speech ID, and passes it to the retrieval and recall module;

[0051] (38) The retrieval and recall module receives the question content, obtains the most relevant content to the user query through the online stage, and passes it to the RAG processing chain module as the retrieval and recall content;

[0052] (39) The knowledge base module passes the corresponding multi-round dialogue context information retrieved from the historical question-answer pair library using the user ID and speech ID transmitted by the signaling processing module to the RAG processing chain module;

[0053] (310) The RAG processing chain module retrieves the recalled related content and the corresponding multi-round dialogue context information based on the user's question content, constructs the complete question content through the Prompt template, and passes it to the signaling processing module;

[0054] (311) The signaling processing module packages the complete question content into a request and sends it to the multimodal large model, receives its response, and parses the answer content;

[0055] (312) The signaling processing module responds to the user question request initiated by the signaling server and returns the multimodal large model answer content to the signaling server, and the signaling server returns it to the client of the user who asked the question;

[0056] (313) The knowledge base module records the current question-answer pair into the historical question-answer pair library to support context memory for multiple rounds of dialogue.

[0057] The beneficial effect of the present invention is that the intelligent question-answering system and method of multimodal information fusion proposed in the present invention can combine the materials containing rich content such as text, pictures, charts, etc. uploaded in advance by the speaker, as well as the real-time lecture content, and use multimodal lecture information such as text, pictures, and voice to provide the audience with personalized intelligent question-answering services. It can not only answer the audience's immediate questions, but also provide extended knowledge supplements based on the lecture content and materials, breaking through the functional boundaries of traditional audio and video communication systems and promoting the effective dissemination and in-depth understanding of knowledge. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a schematic diagram of the module structure of a multimodal intelligent question-answering system based on a large model proposed in the present invention.

[0059] Figure 2 This is a flowchart of a multimodal intelligent question-answering method based on a large model proposed by the present invention.

[0060] Figure 3 It is a schematic diagram of the specific operation steps of the offline knowledge base construction step in an embodiment of the present invention.

[0061] Figure 4 It is a schematic diagram of specific operation steps of the online knowledge base construction step in an embodiment of the present invention.

[0062] Figure 5 1 is a schematic diagram of specific operation steps of the real-time question-and-answer step in an embodiment of the present invention. DETAILED DESCRIPTION

[0063] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings.

[0064] See also Figure 1 , introduces a multimodal intelligent question answering system based on a large model proposed by the present invention, the system includes the following modules:

[0065] The signaling processing module is the core module of the system, responsible for receiving, parsing and processing requests from the signaling server, and coordinating other modules in the system to complete corresponding tasks;

[0066] In this embodiment, the signaling processing module is implemented based on the FastApi framework, using a high-performance web services architecture that supports multi-threaded concurrent processing and can respond to requests from multiple clients simultaneously, ensuring efficient and real-time system operation. The module interacts with the signaling server via the HTTP protocol, receiving and sending network requests and triggering processing flows in other modules within the system.

[0067] The main functions of the signaling processing module include:

[0068] (1) Receive lecture material upload requests, lecture start / end requests, user question requests, and multi-round dialogue list acquisition requests sent by the signaling server; parse the request content; extract key parameters, including lecture room ID, user ID, question content, etc.;

[0069] The signaling server is a server used to establish connections and exchange information in real-time communication. It is the core hub of the speech system and is responsible for message transmission and coordination. The system of the present invention relies on the signaling server as a transit hub and uses the HTTP protocol to send interactive requests.

[0070] (2) Based on the request type sent by the signaling server, the processing flow of other modules in the system is triggered, specifically:

[0071] Speech material upload request: triggers the document pre-processing module to process the uploaded speech materials;

[0072] Lecture start / end request: coordinates the speech preprocessing module to receive real-time speech stream and dynamically updates the knowledge base;

[0073] User question request: the retrieval and recall module and the RAG processing chain module are called to generate the answer and return it to the user;

[0074] Request to obtain a multi-turn conversation list: This triggers the database module, queries the historical question-answer database, and returns the multi-turn conversation context information.

[0075] (3) Return the processing result to the client through the signaling server;

[0076] In the embodiment, the details of the HTTP interface provided by the system of the present invention are shown in Table 1:

[0077] Table 1 System external interfaces

[0078]

[0079] The document preprocessing module has the following main functions: (1) file decompression: receiving the compressed package of the lecture materials sent by the signaling processing module and decompressing the PDF, PPTX, DOCX and other documents therein; (2) page-to-image conversion: converting the pages in the decompressed document into PNG image blocks with a suitable resolution (preserving the font / layout features); (3) metadata annotation: adding metadata to each image block to record additional information of the document, including page number, title, chart type, etc.; the processing results of the document preprocessing module are sent to the retrieval and recall module;

[0080] The speech preprocessing module receives speech streams in real time and transcribes them into text, ensuring that speech content can be accurately recognized and converted into text, providing basic support for subsequent processing. This module supports real-time speech stream processing to dynamically update the knowledge base in the subsequent retrieval and recall modules, ensuring that the question and answer content is synchronized with the progress of the speech.

[0081] Retrieval and Recall Module: This module's main function is to build the retrieval-enhanced knowledge base required by the system and complete online retrieval. This module's functions are divided into two stages: online and offline. The offline stage is responsible for processing pre-input lecture materials containing multimodal information after being processed by the document preprocessing module; the online stage functions to process real-time incoming lecture content and user queries.

[0082] RAG processing chain module: This module uses the user's question and the image blocks returned by the retrieval and recall module as context. It then feeds this context and the historical question-answer pairs retrieved by the database module into the multimodal large model via the Prompt template to generate high-quality answers.

[0083] In the embodiment, the RAG processing chain module is implemented through the LangChain framework. The prompt template is shown in Table 2:

[0084] Table 2: Prompt template

[0085]

[0086] Knowledge base module: The function of this module is to build the system's knowledge base through the storage, retrieval and management of multi-round dialogue history of embedded vectors; this module receives the embedded vectors and their metadata generated by the retrieval and recall module through an efficient data storage and query mechanism, and stores them in the database to ensure that the system can quickly respond to query requests and support the retrieval of embedded vectors and the contextual history memory of multi-round dialogues; this module includes two databases, namely the speech embedding vector database and the historical question and answer pair database; the speech embedding vector database is used to store the embedded vector data generated by the speech for retrieval, and the historical question and answer pair database is used to store the contextual information of multi-round dialogues.

[0087] In the embodiment, the knowledge base module is implemented based on the PostgreSQL database and its pgvector extension, the speech embedding vector database design is shown in Table 3, and the historical question and answer database design is shown in Table 4.

[0088] Table 3 Speech embedding vector data table

[0089]

[0090] Table 4 Historical question-answer data

[0091]

[0092] The speech preprocessing module processes real-time speech streams based on streaming speech recognition technology and an efficient data processing framework. The processing process includes the following three main steps:

[0093] (s1) Voice reception: Receive real-time voice streams through UDP data packets and extract the RTP payload data therein. Specifically, the signaling processing module receives the SDP information parsed from the speech start request, and extracts the IP address, port number and other information required for audio reception; receives the real-time voice stream forwarded by the SFU server, which is in the form of RTP data packets, uses the UDP receiving and parsing tool library (such as scapy) to parse the UDP data packets, extracts the RTP payload data therein, and stores the extracted voice data in the audio segment list; the SFU server refers to the Selective Forwarding Unit (SFU), which is a media stream forwarding server. During the speech, the SFU server acts as a transit station for media data and is responsible for forwarding the audio streams of different users to the intelligent question-and-answer system.

[0094] (s2) Voice format conversion: convert the received voice data into an audio format suitable for voice recognition (such as WAV). Use a library such as pydub to convert the voice data format to ensure that the voice data meets the requirements of voice recognition. Convert the voice data into an audio segment object in WAV format.

[0095] (s3) Speech transcription: Use streaming speech recognition technology (such as faster-whisper) to transcribe speech content into text in real time. Periodically extract speech data from the audio segment list, merge and save it as a temporary WAV file. In the embodiment, the whisper library is used to transcribe the temporary WAV file to generate the transcribed text content.

[0096] The specific processing content of the offline stage of the retrieval and recall module is as follows: first, multi-vectors are generated through delayed interaction, and the multimodal materials input in the offline stage are pre-mapped to a unified vector space through multimodal alignment processing, that is, text embedding and visual embedding are mapped to a unified vector space to ensure that information of different modalities can be compared for similarity; then, the centroid is selected and the embedded vector is compressed by clustering the centroid to reduce the storage space occupied by the embedded vector and retain sufficient semantic information. At the same time, the embedded vector cluster center and the restored vector information are stored for use in the online stage.

[0097] The specific processing content of the online phase of the retrieval and recall module is as follows: (a) processing the real-time incoming lecture content and user queries, and quickly screening candidate documents through the clustering centroid strategy to generate the final retrieval results; encoding the real-time incoming lecture content, generating text embedding, mapping the text embedding to the unified vector space in the offline phase, and performing online data stream clustering on the newly generated embedding vector; (b) encoding the user query, generating query embedding, and mapping it to the unified vector space in the offline phase, quickly screening candidate vectors through the clustering centroid strategy, performing residual decompression on the screened candidate vectors, restoring the original embedding vector representation, and using a delayed interaction mechanism to calculate the fine-grained similarity between the query and the candidate documents. For each embedding vector in the query, calculate its maximum similarity with all embedding vectors in the candidate documents, add the similarity scores of all embedding vectors in the query, generate a total similarity score between the user query and the candidate documents, sort the candidate documents according to the total similarity score, and return the most relevant documents.

[0098] Traditional retrieval methods often struggle to fully utilize the connections between different modalities, resulting in poor accuracy and comprehensiveness in retrieval results. The multimodal retrieval enhancement generation component addresses this issue by comprehensively considering multiple modalities such as text, images, and charts to achieve a deep understanding of multimodal information and effective retrieval. During the multimodal generation embedding process, the connections between different modalities are fully integrated, enabling the system to better capture the complex semantic relationships in multimodal information. This improves the accuracy and comprehensiveness of multimodal retrieval recall when users ask questions, providing users with retrieval results that better meet their needs.

[0099] In an embodiment, the image in the document is first processed using the visual-language dual encoder model SigLIP. The SigLIP visual encoder divides the image into multiple image blocks (patches) and generates an embedding vector for each image block. The embedding vectors of these image blocks capture the local visual features of the image. Then, the embedding vectors of these image blocks are input into the PaliGemma model, and multimodal content alignment is performed based on the PaliGemma model. PaliGemma-3B itself is a large visual language model that combines the visual Transformers (ViTs) model and the large language Transformer model. The PaliGemma language model combines these visual embedding vectors and the text information in the document to generate a contextualized language model output embedding vector. This process achieves a preliminary fusion of multimodal information (text and image), so that the generated embedding vector can contain both textual and visual semantic information.

[0100] To achieve multimodal semantic alignment, a mapping layer is added to the multimodal information mapping process. The mapping layer is responsible for mapping the embedding vector output generated by the PaliGemma model to a lower-dimensional (128-dimensional) vector space. This dimensional mapping helps reduce computational complexity and storage space, while also making subsequent similarity calculations more efficient. During the mapping process, the mapping layer uses linear projection to map text and image embeddings into a unified vector space, converting high-dimensional image embedding vectors into low-dimensional vectors. This ensures that the mapped vectors retain the key information in the original embedding vectors while adapting to the computational requirements of the delayed interaction model.

[0101] By leveraging a visual language model (VLM), embeddings are generated for the visual information (i.e., image patches) and textual information of a document. These image patches are treated as tokens, just like how words are treated in a language model. PaliGemma-3B combines the Visual Transformer (ViT) for visual data with the Gemma language model for textual data, creating a shared representation space for both. The Visual Transformer decomposes the input image into smaller blocks (16x16 pixels), each of which is treated as a separate token. These blocks are then passed through ViT, similar to how text blocks are processed.

[0102] In an embodiment, a document is generated as a sequence of image patches, where each patch is a part of the entire document image. A visual transformer (ViT) processes each patch independently and generates an embedding vector: these embeddings are then compared with the embedding of the text query, all in the same shared space.

[0103] When text is input, it is converted into a series of tokens and passed through the language model. The same large model architecture is used to embed text queries into the same latent space as image embeddings. This shared space ensures that query embeddings can be directly compared with document image embeddings.

[0104] In order to perform effective retrieval later, it is necessary to align the representations of these different modalities so that text and image data can be compared in a common space. The VLM model is able to compare text queries with visual elements in documents. However, the embeddings representing text and image data need to exist in the same dimensional space. Text embeddings are generated by LLM-based models and are typically 768 or 1024 dimensions. Image embeddings are generated by visual ViT and typically have larger dimensions, such as 1024 or 2048 dimensions. These embeddings encode the visual features of the image patch.

[0105] Text embeddings and image embeddings are mapped to a shared space via linear projection, a technique used to bring vectors from different spaces into a common latent space. This shared space supports cross-modal information comparison, meaning that text queries can retrieve relevant document images based on their visual content, such as tables, charts, or other visual elements. To maintain a balance between memory efficiency and computational cost while minimizing performance degradation, a 128-dimensional embedding was chosen. This dimension is also used in the multi-vector reranking process, so the smaller the better, but a proper balance must be struck with respect to performance.

[0106] With the continuous increase in the amount of data, the storage and processing of multimodal information face huge challenges. Traditional methods may lead to reduced retrieval efficiency, and the multi-vector representations generated by pre-calculation of the delayed interaction mechanism in the offline stage are also prone to excessive occupation of storage space. The present invention adopts a clustered centroid compression method, the core idea of which is to utilize the locality of vectors in the embedding space, that is, similar vectors are usually clustered together in the embedding space. By clustering vectors, a central vector can be assigned to each cluster, and then the residual of each vector relative to its cluster center is stored. Since the residual is usually much smaller than the original vector, the storage space can be greatly reduced. Using this strategy can not only retain key information, but also improve the efficiency of storage and retrieval and reduce retrieval latency.

[0107] The offline phase of the retrieval and recall module uses offline clustering, which is specifically as follows: (1) Initial centroid selection: The size of the cluster center set is the power of 2 closest to the total number of embeddings generated by the document, that is, it is proportional to the total number of embeddings in the corpus. A portion of the embedding set of all document blocks is sampled for k-means clustering to obtain C cluster centers. Each embedding vector index is encoded into the cluster center closest to it and an embedding vector for subsequent approximate residual calculations. When encoding the embedding vector, each dimension is quantized to 1 to 2 bits, which significantly reduces the storage cost. (2) Transformation list mapping: A transformation list is saved at the same time to support fast neighbor vector retrieval, and the centroids are mapped to their corresponding document blocks (here, mapping to document blocks instead of embedding vectors is chosen because the number of document blocks is much smaller than the number of embeddings, which can make the overall storage space of the inverted list smaller).

[0108] The online phase of the retrieval and recall module uses online clustering. The specific content is as follows: For newly input speech content (such as text transcribed in real time), it is mapped into a vector embedding set, and the centroid is dynamically updated with reference to the StreamKM++ algorithm. StreamKM++ is very suitable for processing large-scale data streams. While maintaining high clustering quality, it can significantly reduce the complexity of calculation. Due to its efficient sampling and core set technology, StreamKM++ has excellent performance in real-time data processing and streaming analysis. The new embedding vector is assigned to the nearest centroid, and the position of the centroid and the related residual approximation vector are updated. If the distance between the new embedding vector and all centroids exceeds a certain threshold, a new centroid is generated. Specifically: (1) Dynamic cluster update: Exponentially decay each centroid, and update the weight to reduce the influence of the old centroid and adapt to changes in data distribution. For each new embedding vector, find its nearest centroid and update the centroid and weight according to the new weight. (2) Transformation list maintenance: For newly added centroids, add them and their associated content blocks to the transformation list, and recalculate the associated residual for the updated centroid.

[0109] The details of the user query function in the online phase of the retrieval recall module are as follows: (1) Initial candidate set generation: Given the query embedding matrix and the list of centroid vectors in the index, matrix multiplication is performed to calculate the token-level query-centroid similarity, find the top-t centroids, and identify document blocks close to the specified number of centroids as candidate sets (if the document block has more than one token, the centroids in the top-t are used as centroids); (2) Candidate set refinement (centroid interaction): Through the document block candidate set, a centroid-based approximate score D is constructed, and then it is used to rank the candidate documents, taking the top n most relevant document blocks; (3) Candidate set refinement (centroid pruning): As an additional optimization step, low-magnitude centroid scores are first pruned before constructing the candidate set approximate score. In this filtering stage, only tokens whose maximum corresponding centroid score meets a certain threshold are scored, and those below this threshold are filtered out. The original embedding vectors of the final candidate paragraph set are then reconstructed through the residual decompression mechanism, and the final recall ranking of these embeddings is performed using the delayed interaction scoring method.

[0110] The details of the delayed interaction mechanism used by the retrieval and recall module are as follows: Traditional retrieval models can be mainly divided into dual-encoder and cross-encoder structures. The dual-encoder retrieval model often uses a simple single-vector approach to process the interaction between queries and documents, which cannot accurately capture the semantic similarity between them. The cross-encoder retrieval model, on the other hand, calculates the full similarity of the question and text embedding vectors in real time, which is computationally expensive and cannot meet the system's real-time requirements.

[0111] The delayed interaction retrieval mechanism is designed to perform token-level similarity matching between queries and candidate passages. Unlike traditional retrieval models that reduce everything to a single embedding vector for comparison, the delayed interaction mechanism operates on multiple embedding vectors, preserving fine-grained details and improving retrieval accuracy. The total similarity score between the query and the document is the sum of the maximum similarities across all query tokens.

[0112] The original embeddings of the final candidate set are reconstructed through the residual decompression mechanism to obtain the reconstructed embedding vectors of the final candidate set, and these embeddings are ranked using the maximum similarity. Let D be the reconstructed embedding vector of the final candidate set after decompression. The final score LI(Q,D) is then calculated using the following formula, and the similarity ranking is performed in turn, recalling the top-ranked paragraphs:

[0113]

[0114] Q represents the set of embedding vectors for a query. In natural language processing, a query can be a question, a phrase, or a keyword. Embedding a query yields a set of embedding vectors that represent the semantic information of the query.

[0115] Qi is the i-th embedding vector in the query Q. This represents the semantic representation of the query at the token (usually word or subword unit) level, and the embedding vector corresponding to each word is Qi.

[0116] Dj represents the jth embedding vector in the reconstructed embedding vector set of the final candidate set after decompression.

[0117] See also Figure 2 A multimodal intelligent question answering method based on a large model, the method comprising the following steps:

[0118] (1) Offline knowledge base construction step: Before the lecture begins, the system receives an initialization request from the signaling server, processes the lecture materials uploaded by the user, and builds a knowledge base to support subsequent retrieval;

[0119] (2) Online knowledge base construction steps: During the lecture, the system continuously receives the real-time audio stream forwarded by the SFU, and transcribes the voice content into text in real time through automatic speech recognition technology. After processing, it is dynamically updated to the knowledge base to ensure that the knowledge base content is synchronized with the progress of the lecture;

[0120] (3) Real-time question-answering step: When a user submits a question request through the client, the signaling server forwards the question to the system. The system first retrieves the context fragments related to the question from the knowledge base, and then calls the external large model for semantic understanding and answer generation. Finally, the generated answer is returned to the user who asked the question through the signaling server, realizing efficient and accurate real-time question-answering interaction.

[0121] See also Figure 3 The step (1) of building the knowledge base offline specifically includes the following steps:

[0122] (11) The user sends an initialization request including the upload of lecture materials to the signaling server through the user client; the signaling server sends the lecture material upload request to the signaling processing module, and sends the lecture materials to the signaling processing module in a streaming upload manner;

[0123] (12) The signaling processing module triggers the document pre-processing module to process the uploaded lecture materials and send the processing results to the retrieval and recall module;

[0124] (13) The offline phase function of the retrieval and recall module processes the pre-input speech materials containing multimodal information after being processed by the document preprocessing module to generate embedding vectors and their metadata;

[0125] (14) The knowledge base module receives the embedding vector and its metadata generated by the retrieval and recall module and stores them in the speech embedding vector database in the knowledge base module;

[0126] (15) After completing the above operations, the signaling processing module returns the processing results to the user client through the signaling server.

[0127] See also Figure 4 The step (2) of building the knowledge base online specifically includes the following steps:

[0128] (21) The user sends a speech start request to the signaling server through the client. The signaling server sends a media channel creation request to the SFU server. The SFU server creates and returns media channel information to the signaling server. The signaling server forwards this information to the signaling processing module. The signaling processing module triggers the voice preprocessing module to start receiving the RTP voice stream and finally sends a successful start response to the signaling server.

[0129] (22) The voice preprocessing module completes the process of converting the voice RTP stream into text through voice reception, voice format conversion and voice transcription, and transmits the transcribed text content to the retrieval and recall module in real time;

[0130] (23) The knowledge base module reads the embedding vectors and metadata generated by the offline phase of the retrieval and recall module from the lecture embedding vector database;

[0131] (24) The retrieval and recall module processes the speech text content transmitted in real time by the speech preprocessing module in the online stage to obtain the updated embedding vector and its metadata;

[0132] (25) The retrieval and recall module triggers the knowledge base module to store the updated embedding vector and its metadata into the speech embedding vector database in the knowledge base module;

[0133] (26) When the speech ends, the user client sends a request to end the speech to the signaling server, which forwards the request to end the speech to the signaling processing module. The signaling processing module receives the request to end the speech from the signaling server, sends a successful end response, and initiates a stop operation to the voice preprocessing module and the retrieval and recall module.

[0134] See also Figure 5 The step (3) of real-time question and answer includes the following steps:

[0135] (31) The user sends a request to obtain a multi-round conversation list to the signaling server through the user client. After receiving the request, the signaling server forwards the request to the signaling processing module.

[0136] (32) The signaling processing module receives the multi-round conversation list acquisition request, parses the request content to extract key parameters, including user ID and speech ID, triggers the knowledge base module and transmits these key parameters;

[0137] (33) The knowledge base module retrieves the corresponding multi-round dialogue context information from the historical question-answer pair database using the user ID and speech ID passed by the signaling processing module, and passes it to the signaling processing module;

[0138] (34) The signaling processing module responds to the multi-round conversation list acquisition request initiated by the signaling server, returns the context information of the multi-round conversation to the signaling server, and the signaling server returns it to the client of the user who asked the question;

[0139] (35) The user sends a user question request to the signaling server through the user client. After receiving the question request, the signaling server forwards the question request to the signaling processing module;

[0140] (36) The signaling processing module receives the question request, parses the request content to extract key parameters, including user ID, lecture ID and question content, and triggers the retrieval and recall module, knowledge base module and RAG processing chain module;

[0141] (37) The knowledge base module reads the corresponding embedding vector and metadata generated by the previous retrieval and recall module from the speech embedding vector database based on the user ID and speech ID, and passes it to the retrieval and recall module;

[0142] (38) The retrieval and recall module receives the question content, obtains the most relevant content to the user query through the online stage, and passes it to the RAG processing chain module as the retrieval and recall content;

[0143] (39) The knowledge base module passes the corresponding multi-round dialogue context information retrieved from the historical question-answer pair library using the user ID and speech ID transmitted by the signaling processing module to the RAG processing chain module;

[0144] (310) The RAG processing chain module retrieves the recalled related content and the corresponding multi-round dialogue context information based on the user's question content, constructs the complete question content through the Prompt template, and passes it to the signaling processing module;

[0145] (311) The signaling processing module packages the complete question content into a request and sends it to the multimodal large model, receives its response, and parses the answer content;

[0146] (312) The signaling processing module responds to the user question request initiated by the signaling server and returns the multimodal large model answer content to the signaling server, and the signaling server returns it to the client of the user who asked the question;

[0147] (313) The knowledge base module records the current question-answer pair into the historical question-answer pair library to support context memory for multiple rounds of dialogue.

[0148] The inventors conducted a large number of simulation experiments on the system proposed in the present invention, and the experimental results are as follows:

[0149] The functional test used POSTMAN software to simulate the signaling server sending business requests to the system, uploading speech materials, starting / ending speech, asking user questions, and obtaining multi-round dialogue lists. These functional interface-related tests were performed. All modules were executed as expected. The specific test results are shown in Table 5.

[0150] Table 5 Experimental results

[0151]

[0152]

[0153] The above experimental results prove that the system and method proposed in the present invention are feasible and effective.

Claims

1. A multimodal intelligent question-answering system based on a large model, characterized by: The system includes the following modules: The signaling processing module is the core module of the system, responsible for receiving, parsing and processing requests from the signaling server, and coordinating other modules in the system to complete corresponding tasks; Its main functions include: (1) Receive lecture material upload requests, lecture start / end requests, user question requests, and multi-round dialogue list acquisition requests sent by the signaling server; parse the request content; extract key parameters, including lecture room ID, user ID, question content, etc.; (2) Based on the request type sent by the signaling server, the processing flow of other modules in the system is triggered, specifically: Speech material upload request: triggers the document pre-processing module to process the uploaded speech materials; Lecture start / end request: coordinates the speech preprocessing module to receive real-time speech stream and dynamically updates the knowledge base; User question request: the retrieval and recall module and the RAG processing chain module are called to generate the answer and return it to the user; Request to obtain a multi-turn conversation list: This triggers the database module, queries the historical question-answer database, and returns the multi-turn conversation context information. (3) Return the processing result to the client through the signaling server; The document preprocessing module has the following main functions: (1) file decompression: receiving the compressed package of lecture materials sent by the signaling processing module and decompressing the PDF, PPTX, DOCX and other documents therein; (2) page-to-image conversion: converting the pages in the decompressed document into PNG image blocks with appropriate resolution, preserving the font and layout features; (3) metadata annotation: adding metadata to each image block to record additional information of the document, including page number, title, chart type, etc.; the processing results of the document preprocessing module are sent to the retrieval and recall module; The speech preprocessing module receives speech streams in real time and transcribes them into text, ensuring that speech content can be accurately recognized and converted into text, providing basic support for subsequent processing. This module supports real-time speech stream processing to dynamically update the knowledge base in the subsequent retrieval and recall modules, ensuring that the question and answer content is synchronized with the progress of the speech. Retrieval and Recall Module: This module's main function is to build the retrieval-enhanced knowledge base required by the system and complete online retrieval. This module's functions are divided into two stages: online and offline. The offline stage is responsible for processing pre-input lecture materials containing multimodal information after being processed by the document preprocessing module; the online stage functions to process real-time incoming lecture content and user queries. RAG processing chain module: This module uses the user's question and the image blocks returned by the retrieval and recall module as context. It then feeds this context and the historical question-answer pairs retrieved by the database module into the multimodal large model via the Prompt template to generate high-quality answers. Knowledge base module: The function of this module is to build the system's knowledge base through the storage, retrieval and management of multi-round dialogue history of embedded vectors; this module receives the embedded vectors and their metadata generated by the retrieval and recall module through an efficient data storage and query mechanism, and stores them in the database to ensure that the system can quickly respond to query requests and support the retrieval of embedded vectors and the contextual history memory of multi-round dialogues; this module includes two databases, namely the speech embedding vector database and the historical question and answer pair database; the speech embedding vector database is used to store the embedded vector data generated by the speech for retrieval, and the historical question and answer pair database is used to store the contextual information of multi-round dialogues.

2. The multimodal intelligent question-answering system based on a large model according to claim 1, characterized in that: The speech preprocessing module processes real-time speech streams based on streaming speech recognition technology and an efficient data processing framework. The processing process includes the following three main steps: (s1) Voice reception: Receives real-time voice streams via UDP packets and extracts the RTP payload data therefrom. Specifically, the signaling processing module parses the SDP information from the speech start request and extracts the IP address, port number, and other information required for audio reception. Receives the real-time voice stream forwarded by the SFU server in the form of RTP packets, parses the UDP packets using the UDP receiving and parsing tool library, extracts the RTP payload data therefrom, and stores the extracted voice data in the audio segment list. (s2) Voice format conversion: converting the received voice data into an audio format suitable for voice recognition, performing format conversion on the voice data to ensure that the voice data meets the requirements of voice recognition, and converting the voice data into an audio segment object in WAV format; (s3) Speech transcription: Use streaming speech recognition technology to transcribe speech content into text in real time. Periodically extract speech data from the audio segment list, merge and save it as a temporary WAV file, perform speech transcription on the temporary WAV file, and generate the transcribed text content.

3. The multimodal intelligent question-answering system based on a large model according to claim 1, characterized in that: The specific processing content of the offline stage of the retrieval and recall module is as follows: first, multi-vectors are generated through delayed interaction, and the multimodal materials input in the offline stage are pre-mapped to a unified vector space through multimodal alignment processing, that is, text embedding and visual embedding are mapped to a unified vector space to ensure that information of different modalities can be compared for similarity; then, the centroid is selected and the embedded vector is compressed by clustering the centroid to reduce the storage space occupied by the embedded vector and retain sufficient semantic information. At the same time, the embedded vector cluster center and the restored vector information are stored for use in the online stage.

4. The multimodal intelligent question-answering system based on a large model according to claim 1, characterized in that: The specific processing content of the online phase of the retrieval and recall module is as follows: (a) processing the real-time incoming speech content and user queries, and quickly screening candidate documents through the clustering centroid strategy to generate the final retrieval results; (b) encoding the real-time incoming speech content to generate text embeddings, mapping the text embeddings to the unified vector space used in the offline phase, and performing online data stream clustering on the newly generated embedding vectors; (b) For user queries, encode them, generate query embeddings, and map them to a unified vector space in the offline stage. Quickly screen candidate vectors through the cluster centroid strategy, perform residual decompression on the screened candidate vectors, restore the original embedding vector representation, and use a delayed interaction mechanism to calculate the fine-grained similarity between the query and candidate documents. For each embedding vector in the query, calculate its maximum similarity with all embedding vectors in the candidate documents, add up the similarity scores of all embedding vectors in the query, and generate a total similarity score between the user query and the candidate documents. Then, sort the candidate documents according to the total similarity score and return the most relevant documents.

5. A multimodal intelligent question-answering method based on a large model, characterized by: The method comprises the following steps: (1) Offline knowledge base construction step: Before the lecture begins, the system receives an initialization request from the signaling server, processes the lecture materials uploaded by the user, and builds a knowledge base to support subsequent retrieval; (2) Online knowledge base construction steps: During the lecture, the system continuously receives the real-time audio stream forwarded by the SFU, and transcribes the voice content into text in real time through automatic speech recognition technology. After processing, it is dynamically updated to the knowledge base to ensure that the knowledge base content is synchronized with the progress of the lecture; (3) Real-time question-answering step: When a user submits a question request through the client, the signaling server forwards the question to the system. The system first retrieves the context fragments related to the question from the knowledge base, and then calls the external large model for semantic understanding and answer generation. Finally, the generated answer is returned to the user who asked the question through the signaling server, realizing efficient and accurate real-time question-answering interaction.

6. The multimodal intelligent question answering method based on a large model according to claim 5, characterized in that: The step (1) of building the knowledge base offline specifically includes the following steps: (11) The user sends an initialization request including the upload of lecture materials to the signaling server through the user client; the signaling server sends the lecture material upload request to the signaling processing module, and sends the lecture materials to the signaling processing module in a streaming upload manner; (12) The signaling processing module triggers the document pre-processing module to process the uploaded lecture materials and send the processing results to the retrieval and recall module; (13) The offline phase function of the retrieval and recall module processes the pre-input speech materials containing multimodal information after being processed by the document preprocessing module to generate embedding vectors and their metadata; (14) The knowledge base module receives the embedding vector and its metadata generated by the retrieval and recall module and stores them in the speech embedding vector database in the knowledge base module; (15) After completing the above operations, the signaling processing module returns the processing results to the user client through the signaling server.

7. The multimodal intelligent question answering method based on a large model according to claim 5, characterized in that: The step (2) of building the knowledge base online specifically includes the following steps: (21) The user sends a speech start request to the signaling server through the client. The signaling server sends a media channel creation request to the SFU server. The SFU server creates and returns media channel information to the signaling server. The signaling server forwards this information to the signaling processing module. The signaling processing module triggers the voice preprocessing module to start receiving voice streams. Finally, it sends a successful start response to the signaling server. (22) The voice preprocessing module completes the process of converting the voice RTP stream into text through voice reception, voice format conversion and voice transcription, and transmits the transcribed text content to the retrieval and recall module in real time; (23) The knowledge base module reads the embedding vectors and metadata generated by the offline phase of the retrieval and recall module from the lecture embedding vector database; (24) The retrieval and recall module processes the speech text content transmitted in real time by the speech preprocessing module in the online stage to obtain the updated embedding vector and its metadata; (25) The retrieval and recall module triggers the knowledge base module to store the updated embedding vector and its metadata into the speech embedding vector database in the knowledge base module; (26) When the speech ends, the signaling processing module receives the speech end request sent by the signaling server, sends a successful end response, and initiates a stop operation to the voice preprocessing module and the retrieval and recall module.

8. The multimodal intelligent question answering method based on a large model according to claim 5, characterized in that: The step (3) of real-time question and answer includes the following steps: (31) The user sends a request to obtain a multi-round conversation list to the signaling server through the user client. After receiving the request, the signaling server forwards the request to the signaling processing module. (32) The signaling processing module receives the multi-round conversation list acquisition request, parses the request content to extract key parameters, including user ID and speech ID, triggers the knowledge base module and transmits these key parameters; (33) The knowledge base module retrieves the corresponding multi-round dialogue context information from the historical question-answer pair database using the user ID and speech ID passed by the signaling processing module, and passes it to the signaling processing module; (34) The signaling processing module responds to the multi-round conversation list acquisition request initiated by the signaling server, returns the context information of the multi-round conversation to the signaling server, and the signaling server returns it to the client of the user who asked the question; (35) The user sends a user question request to the signaling server through the user client. After receiving the question request, the signaling server forwards the question request to the signaling processing module; (36) The signaling processing module receives the question request, parses the request content to extract key parameters, including user ID, lecture ID and question content, and triggers the retrieval and recall module, knowledge base module and RAG processing chain module; (37) The knowledge base module reads the corresponding embedding vector and metadata generated by the previous retrieval and recall module from the speech embedding vector database based on the user ID and speech ID, and passes it to the retrieval and recall module; (38) The retrieval and recall module receives the question content, obtains the most relevant content to the user query through the online stage, and passes it to the RAG processing chain module as the retrieval and recall content; (39) The knowledge base module passes the corresponding multi-round dialogue context information retrieved from the historical question-answer pair library using the user ID and speech ID transmitted by the signaling processing module to the RAG processing chain module; (310) The RAG processing chain module retrieves the recalled related content and the corresponding multi-round dialogue context information based on the user's question content, constructs the complete question content through the Prompt template, and passes it to the signaling processing module; (311) The signaling processing module packages the complete question content into a request and sends it to the multimodal large model, receives its response, and parses the answer content; (312) The signaling processing module responds to the user question request initiated by the signaling server and returns the multimodal large model answer content to the signaling server, and the signaling server returns it to the client of the user who asked the question; (313) The knowledge base module records the current question-answer pair into the historical question-answer pair library to support context memory for multiple rounds of dialogue.

Images