A large-model voice question answering system for multi-source heterogeneous local knowledge base

Abstract

The present invention discloses a large-model voice question-answering system for multi-source heterogeneous local knowledge bases, including: a multi-source heterogeneous local knowledge base processing module, which considers the diversity of local knowledge base data sources, structures, types, etc., and designs a cutting method based on semantic integrity to construct a vector database; a large-model reasoning module based on the local knowledge base, which provides a matching threshold exclusive to the industry knowledge base, performs incremental dynamic updates, and realizes adaptive context matching with the industry knowledge base to solve the problem of local knowledge redundancy or scarcity in large-model reasoning; a human-computer voice interaction module, which introduces a virtual digital human and a voice interaction interface to realize "artificial" communication between the model and the user, and realizes good human-computer interaction. The present invention uniformly processes and adaptively matches the complex and diverse local knowledge bases of various industries, and realizes low-cost, high-efficiency, one-stop deployment of a large-model voice question-answering system.

Description

Technical Field

[0001] The present invention relates to the technical field of vertical large-scale model applications, and in particular to a large-scale model voice question-answering system for multi-source heterogeneous local knowledge bases. Background Art

[0002] In recent years, generative AI tools, such as ChatGPT, have exploded in popularity. Their exceptional performance in natural language processing, computer vision, and reinforcement learning has marked a historic milestone in the era of general artificial intelligence. With the rapid evolution and continuous improvement of models, general generative AI has demonstrated tremendous application potential and commercial value, attracting widespread attention across various fields.

[0003] Today, large models are shifting from closed scenarios to open tasks. Based on diverse application scenarios and incorporating local and professional knowledge bases, they are transforming from general-purpose AI to specialized AI serving various industries. However, existing data within industries varies greatly, and much local data presents issues of sensitivity and security that require consideration. This has led to the current AI production paradigm being a typical hand-crafted design model, where specialized AI deployment requires the development of unique processing flows for each specific scenario. Furthermore, vertical applications of large models lack a one-stop deployment system with good human-computer interaction. This, in turn, results in long AI development cycles and high implementation costs, a constraint that urgently needs to be overcome for the large-scale application of generative large speech models. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of current generative artificial intelligence applications and provide a large-model voice question-answering system for multi-source heterogeneous local knowledge bases to solve the current problems of large differences in the deployment of vertical large models, high costs, and limited dialogue security and accuracy.

[0005] To achieve the above objectives, the present invention provides a technical solution: a large-model voice question-answering system for a multi-source heterogeneous local knowledge base, comprising: a multi-source heterogeneous local knowledge base processing module, a large-model reasoning module based on the local knowledge base, and a human-computer voice interaction module;

[0006] The multi-source heterogeneous local knowledge base processing module provides a unified processing flow for local knowledge bases adapted to different industries, solves the problem of diversity in local knowledge base data sources, structures, types, and formats, and builds a standard local knowledge base;

[0007] The large model inference module uses the proposed adaptive knowledge base matching method to perform incremental dynamic updates based on Gaussian functions to the knowledge base's exclusive threshold. Based on the exclusive threshold, the L1 norm of the semantic cluster center is calculated to achieve similarity measurement. Based on the diversity of the industry knowledge base and the relevance of questions and answers, dynamic context matching with no fixed number of items is performed to achieve dedicated large model inference based on multi-source heterogeneous local knowledge bases and meet industry security and accuracy requirements.

[0008] The human-computer voice interaction module is used to realize the voice and visual interaction between the large model and the user. Based on the industry-specified service portrait, voice is used to drive the facial expressions and lip shapes of the virtual digital human, realizing a transparent "artificial" interaction mode of the model.

[0009] Furthermore, the multi-source heterogeneous local knowledge base processing module considers the diversity of local data sources, structures, types, and formats, and uniformly converts complex and diverse industry data according to semantic integrity, aiming to meet the adaptability of multiple industries, including the following steps:

[0010] 1) Collect industry Q&A related knowledge base: For multimodal data scenarios, extract the text knowledge base of the required modality to avoid information redundancy and retrieval difficulties, and match the Q&A requirements to build a local text knowledge base: D = {D1, D2, ... D i ,…,D n}, where D i represents the i-th multi-source heterogeneous text knowledge base in a specific industry. Considering the actual existence of the industry knowledge base, D i Contains different data types and multiple file formats, including tables and documents, and file formats including JSON, PDF, and XML;

[0011] 2) Semantic integrity segmentation: divide text information into structured and unstructured data, specify the upper limit threshold of the length of a single sub-data as τ, and perform text segmentation based on semantic integrity to segment the knowledge base: D i ={d i1 ,d i2 ,…,d ij ,…,d in}, where d ij Represents the jth cutting sub-data in the i-th knowledge base;

[0012] 3) Database conversion: Use the vector embedding model Embed(·) to convert text data into vector data and construct text-vector tuples:

[0013] v ij =Embed M (d ij )

[0014] D i ={(d i1 ,v i1 ),(d i2 ,v i2 ),…,(d ij ,v ij ),…,(d in ,v in )}

[0015] In the formula, Embed M (d ij ) means embedding the unit text information of the specified length upper limit threshold τ into the local knowledge base vector data v with a fixed length of M ij , the sub-dataset d ij Convert to text-vector tuple (d ij ,v ij ).

[0016] Furthermore, the semantic integrity segmentation divides local data into structured and unstructured data. For structured data, segmentation is performed according to independent data units; for unstructured data, segmentation is performed according to the priority: sub-data length upper limit threshold τ> segment> sentence, that is, within the length upper limit threshold, semantic segmentation is performed in units of segments and sentences to maximize the retention of the semantic integrity of the converted sub-data.

[0017] Furthermore, the large model inference module considers the differences in the amount of contextual information required by different industry data and different query questions, and designs an adaptive knowledge base matching method to solve the problem of local knowledge redundancy or scarcity when inferring large language models, including the following steps:

[0018] 1) Convert user query speech into query text and embed the query text into a vector for context matching:

[0019] v Q =Embed M (Q s )

[0020] Where Q s The initial query text is processed by the same Embed as the local knowledge base. M (·) model, the initial query text is converted into vector data of length M, that is, the query vector v Q ;

[0021] 2) Vector Similarity Distance Calculation: The query text is mapped to vector data of the same dimension. Fasis vector matching is introduced to cluster the dimensions of the data according to semantics to improve matching accuracy and efficiency. The similarity between texts is expressed as:

[0022] ds =||v Q -v ij ||1

[0023] Where ||·||1 is the L1 norm between vectors, d s Represents the query vector v Q With the local knowledge base vector data v ij The similarity between two vectors. The smaller the norm between vectors, the greater the similarity. When the L1 norm is 0, it means that the two vectors are exactly the same.

[0024] 3) Set the industry knowledge base-specific similarity threshold, and practitioners generate the threshold sequence {μ1,μ2,…,μ i ,…,μ n}, based on the threshold sequence, Gaussian function modeling is used to obtain the hierarchical knowledge base threshold that can be updated incrementally:

[0025]

[0026] δ∈{μ+p*σ|p=0,±1,±2,…}

[0027] Where μ i represents the matching threshold of the i-th knowledge base in the industry, μ represents the mean of the threshold sequence, σ is the modified standard deviation of the threshold Gaussian distribution, δ is the industry similarity matching threshold, and p is a hyperparameter that controls the value of the matching threshold. Based on the probability density distribution of the Gaussian function, the similarity matching threshold δ is divided into multiple levels with the standard deviation as the unit. The specific level is selected according to the industry needs.

[0028] 4) Local knowledge base matching, design of adaptive knowledge base matching method:

[0029]

[0030] Where Q represents the initial query text Q s The combination of similarity retrieval text is initialized to Q = Q s ,concat(Q,d ij ) indicates text concatenation, d s_max It represents the unit data text with the highest similarity to the query text in the knowledge base, δ is the similarity threshold of the industry knowledge base, and Q is generated by iteratively in the vector database;

[0031] 5) Large language model reasoning combined with local knowledge base:

[0032] R=LLM θ (Q)

[0033] Where R represents the reasoning result based on the local knowledge base, LLM θ(·) indicates an optional large model interface, which supports local deployment of fine-tuned large models and remote calls to the large model API. θ represents the parameters of the pre-trained large language model.

[0034] Furthermore, the adaptive knowledge base matching method is based on a similarity threshold specific to the industry knowledge base. If the calculated L1 norm is lower than the industry knowledge base similarity threshold δ, the data below the threshold should be uniformly used as context input. If the L1 norms of the given knowledge base data and the query vector are both higher than the threshold, the data with the closest similarity is selected for input. The adaptive knowledge base matching method overcomes the problem of industry information redundancy or scarcity in the context matching process to achieve adaptive matching between the local knowledge base and the query context requirements.

[0035] Furthermore, the human-computer voice interaction module provides a voice-to-text conversion interface for text-to-speech switching and drives the digital human's expression and lip shape based on the open source model SadTalker, including the following steps:

[0036] 1) Use the text-to-speech interface to convert the output text of the large model into speech:

[0037] a {1,2,…,t} =TTS(R)

[0038] In the formula, TTS(·) represents the text-to-speech conversion interface, which directly calls the existing Python open source library. {1,2,…,t} Represents the generated t-segment speech sequence;

[0039] 2) The industry provides digital human portraits in picture or video format β0;

[0040] 3) Combine the speech sequence with the digital human portrait to drive the changes in the digital human's expression and lip shape, and obtain a video of the digital human talking to the user. The generation method is:

[0041] β {1,2,...,t} =Φ(Ψ A (a {1,2,...t} ),β0)

[0042] Where, β {1,2,…,t} represents the generated digital human sequence, Ψ A (·) represents the audio encoder based on the residual network, which embeds the audio features into a latent space, takes the digital human portrait β0 as a condition, and uses the network Φ(·) as a feature decoder to generate the dialogue sequence β {1,2,…,t} .

[0043] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0044] 1. The present invention provides a one-stop system for deploying question-answering services for large models in actual industries. It is transparent to actual industries and specific models, has universal applicability, and improves the deployment efficiency of large models in actual applications.

[0045] 2. The present invention is aimed at multi-source heterogeneous local knowledge bases. Taking into account the diversity of sources, structures, types, etc. of actual production data environments, it constructs a unified processing method for local knowledge bases that retains semantic integrity and adapts to the complex and diverse actual industry knowledge base environments.

[0046] 3. The present invention designs an adaptive knowledge base matching method based on local knowledge base retrieval, provides matching thresholds exclusive to industry knowledge bases, and performs incremental dynamic updates to meet the differences between the knowledge base and the query questions, improve the accuracy and efficiency of local knowledge base matching, and provide a new solution for local deployment.

[0047] 4. The present invention provides a question-and-answer platform for human-computer voice interaction and a large model open interface, combining the most cutting-edge large model technology, digital human technology and model deployment technology, with good human-computer interaction capabilities, realizing the practical application of cutting-edge technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is an overall flow chart of a large-model voice question-answering system for a multi-source heterogeneous local knowledge base according to an embodiment of the present invention.

[0049] Figure 2 A schematic diagram of a method for processing a multi-source heterogeneous local knowledge base according to an embodiment of the present invention.

[0050] Figure 3 This is a flowchart of a large model reasoning based on a local knowledge base in an embodiment of the present invention.

[0051] Figure 4 This is a schematic diagram of human-computer voice interaction between a user and an optional large model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The present invention will be further described below with reference to specific embodiments.

[0053] like Figure 1 As shown, this embodiment provides a large-model voice question-answering system for multi-source heterogeneous local knowledge bases, including: a multi-source heterogeneous local knowledge base processing module, a large-model reasoning module based on the local knowledge base, and a human-computer voice interaction module.

[0054] The multi-source heterogeneous local knowledge base processing module provides a unified processing flow for local knowledge bases adapted to different industries, solves the problem of diversity in local knowledge base data sources, structures, types, and formats, and builds a standard local knowledge base;

[0055] The large model inference module uses the proposed adaptive knowledge base matching method to perform incremental dynamic updates based on Gaussian functions to the knowledge base's exclusive threshold. Based on the exclusive threshold, the L1 norm of the semantic cluster center is calculated to achieve similarity measurement. Based on the diversity of the industry knowledge base and the relevance of questions and answers, dynamic context matching with no fixed number of items is performed to achieve dedicated large model inference based on multi-source heterogeneous local knowledge bases and meet industry security and accuracy requirements.

[0056] The human-computer voice interaction module is used to realize the voice and visual interaction between the large model and the user. Based on the industry-specified service portrait, voice is used to drive the facial expressions and lip shapes of the virtual digital human, realizing a transparent "artificial" interaction mode of the model.

[0057] Furthermore, taking the intelligent question-and-answer service of traffic control as an example, when a car owner inquires about his or her vehicle violations and their penalties, the system needs to provide corresponding question-and-answer services. The local multi-source heterogeneous knowledge base processing realizes the transformation of the complex and diverse multi-source heterogeneous local knowledge base into a unified vector database that can interact with the large model. Specifically, Figure 2 As shown in the figure, the local knowledge base involved in completing the Q&A requirements includes photos of violations captured during driving and PDF documents of corresponding penalty measures. For the captured violation images, different from the text modality required for Q&A, the text information is first extracted based on the industry Q&A requirements; for the images captured for violations, the extracted information is as follows:

[0058] {

[0059] License plate number: XXX

[0060] Owner's name: XXX

[0061] Violation location: XXX

[0062] Violation: XXX

[0063] };

[0064] The multimodal data is converted into the text information required by the industry question-answering service through a text converter to obtain the industry original database D = {D1, D2, ... D N}, where D iRepresents the i-th multi-source heterogeneous text knowledge base in the traffic question-answering service. Considering the integrity of information and the accuracy of retrieval, the converted text is divided into structured data and unstructured data according to the independence of different data entries; for the illegal capture information, which belongs to structured data, it is directly divided according to the independent data structure; for the penalty measure PDF text, which belongs to unstructured text data, semantic integrity segmentation is implemented. When performing text segmentation, the upper limit threshold τ of the text segmentation length is specified, and the units are paragraphs and sentences. For any text knowledge base D i ={d i1 ,d i2 ,...,d in}, traverse the text data from the beginning. Within a unit of the cutting length limit, if the entire text exists, the entire text is cut into a piece of data; if the entire text does not exist, it is split at the end of the sentence closest to the end. The text cutting point is used as the new text starting point, and traversal continues until the text segmentation is completed. Finally, the unit text data is embedded into a vector using the Text2vec model, where any text knowledge base D i ={(d i1 ,v i1 ),(d i2 ,v i2 ),…,(d ij ,v ij ),…,(d in ,v in )}. At this point, the local multi-source heterogeneous database has completed the unified vector database conversion.

[0065] Furthermore, a large model reasoning module based on a local knowledge base is required. Based on the local knowledge base constructed by the multi-source heterogeneous local knowledge base processing module, the reasoning capabilities of a large model need to be combined to meet industry-specific question-and-answer requirements. Due to factors such as industry barriers, intellectual property rights, and user security, even fine-tuned large models often struggle to meet actual industry needs in the absence of memory and context. However, large model reasoning combined with a local knowledge base can obtain relevant industry knowledge and related information from the local knowledge base based on real-time user queries, thereby overcoming issues such as lack of expertise and inaccurate answers.

[0066] Specifically, such as Figure 3 As shown in the figure: When a user performs a query on a question, the query text is first converted into a query vector through a word embedding model. Due to the randomness of the query question, the length of the converted query vector is not fixed; and because the size of the local knowledge base is not fixed, a more efficient similarity retrieval algorithm is required. In the adaptive knowledge base matching method designed by the present invention, the Faiss vector matching algorithm is first introduced to calculate the similarity between the query vector and the local vector database. Faiss uses the same EmbeddedM (·) The query question is embedded into M-dimensional vector data, and the data of each dimension is represented by the semantic cluster center ID; when performing query similarity matching, the difference in the cluster center IDs of the corresponding dimensions of the two is calculated as the similarity between the vectors. The closer the difference in the cluster center IDs, the higher the semantic similarity between the two. Since the distance between the vectors of different knowledge bases is different from the actual matching vectors, the specific local knowledge base in the embodiment should be modeled with a similarity distribution Gaussian function. The similarity matching threshold δ is divided into five levels in the interval [μ-2σ,u+2σ] with the standard deviation as the unit, and the specific level is determined according to the corresponding query. The initial query text Q s The combination Q of similarity retrieval text is:

[0067]

[0068] Where concat(Q,d ij ) indicates text concatenation, d s_max Represents the local data text with the highest similarity to the query text. When multiple matches are found with distances below the upper threshold, the text corresponding to all vectors below the upper threshold is used as the query context. When the distance between all vectors in the knowledge base and the query vector is above the threshold, the text corresponding to the vector with the highest similarity is input as the query context and concatenated with the query text as the input to the large model.

[0069] Furthermore, if Figure 4 As shown, the human-computer voice interaction module provides a method for large-scale model voice and visual interaction. Specifically, it first introduces a pre-trained digital human model. In this embodiment, the open source model SadTalker is used. A corresponding service profile is set based on the traffic question-and-answer service image, thereby shielding the model and allowing direct interaction with the user. Next, a speech-to-text conversion interface is connected. In this embodiment, a Python open source library is used. When users make queries, they can directly input them via voice. The speech-to-text interface converts the user's voice query into a text query. When the large model outputs, the output text is first converted into speech and then sent to the virtual digital human model. The speech drives the generation of the digital human's facial expressions and lip movements, enabling professional "artificial" interaction between the model and the human. Considering the rapid iteration of models and the different focuses of different industries, the present invention provides a multi-model deployment interface. The system provides input and output interfaces for the large model while being transparent to the actual model. The system is applicable to various large models, including general-purpose, fine-tuned, and locally deployed. The specific model can be selected based on actual industry needs.

[0070] The above embodiments are intended only to illustrate the technical concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. They are not intended to limit the scope of protection of the present invention. Any equivalent changes or modifications made in accordance with the spirit of the present invention are intended to be covered by the scope of protection of the present invention.

Claims

1. A large-model voice question answering system for multi-source heterogeneous local knowledge base, characterized by: include: Multi-source heterogeneous local knowledge base processing module, large model reasoning module based on local knowledge base, and human-computer voice interaction module; The multi-source heterogeneous local knowledge base processing module provides a unified processing flow for local knowledge bases adapted to different industries, solves the problem of diversity in local knowledge base data sources, structures, types, and formats, and builds a standard local knowledge base; The large model inference module uses the proposed adaptive knowledge base matching method to perform incremental dynamic updates based on the Gaussian function for the knowledge base's exclusive threshold. Based on the exclusive threshold, the L1 norm of the semantic clustering center is calculated to achieve similarity measurement. Based on the diversity of the relevance between the industry knowledge base and the question and answer, dynamic context matching with no fixed number of items is performed. The human-computer voice interaction module is used to realize the interaction between the large model and the user in voice and vision. Based on the service portrait specified by the industry, the module uses voice to drive the facial expressions and lip movements of the virtual digital human. The large model reasoning module considers the differences in the amount of contextual information required by different industry data and different query questions, and designs an adaptive knowledge base matching method, including the following steps: 1) Convert user query speech into query text and embed the query text into a vector for context matching: v Q =Embed M (Q s ) Where Q s The initial query text is processed by the same Embed as the local knowledge base. M (·) model, the initial query text is converted into vector data of length M, that is, the query vector v Q ; 2) Vector Similarity Distance Calculation: The query text is mapped to vector data of the same dimension. Fasis vector matching is introduced to cluster the dimensions of the data according to semantics to improve matching accuracy and efficiency. The similarity between texts is expressed as: d s =||v Q -v ij ||1 Where ||·||1 is the L1 norm between vectors, d s Represents the query vector v Q With the local knowledge base vector data v ij The similarity between two vectors. The smaller the norm between vectors, the greater the similarity. When the L1 norm is 0, it means that the two vectors are exactly the same. 3) Set the industry knowledge base-specific similarity threshold, and practitioners generate the threshold sequence {μ1,μ2,…,μ i ,…,μ n }, based on the threshold sequence, Gaussian function modeling is used to obtain the hierarchical knowledge base threshold that can be updated incrementally: δ∈{μ+p*σ|p=0,±1,±2,…} Where μ i represents the matching threshold of the i-th knowledge base in the industry, μ represents the mean of the threshold sequence, σ is the modified standard deviation of the threshold Gaussian distribution, δ is the industry similarity matching threshold, and p is a hyperparameter that controls the value of the matching threshold. Based on the probability density distribution of the Gaussian function, the similarity matching threshold δ is divided into multiple levels with the standard deviation as the unit. The specific level is selected according to the industry needs. 4) Local knowledge base matching, design of adaptive knowledge base matching method: Where Q represents the initial query text Q s The combination of similarity retrieval text is initialized to Q = Q s ,concat(Q,d ij ) indicates text concatenation, d s_max It represents the unit data text with the highest similarity to the query text in the knowledge base, δ is the similarity threshold of the industry knowledge base, and Q is generated by iteratively in the vector database; 5) Large language model reasoning combined with local knowledge base: R=LLM θ (Q) Where R represents the reasoning result based on the local knowledge base, LLM θ (·) indicates an optional large model interface, which supports local deployment of fine-tuned large models and remote calls to the large model API. θ represents the parameters of the pre-trained large language model.

2. A large-model voice question-answering system for multi-source heterogeneous local knowledge bases according to claim 1, characterized in that: The multi-source heterogeneous local knowledge base processing module considers the diversity of local data sources, structures, types, and formats, and uniformly converts complex and diverse industry data according to semantic integrity, including the following steps: 1) Collect industry Q&A related knowledge base: For multimodal data scenarios, extract the text knowledge base of the required modality to avoid information redundancy and retrieval difficulties, and match the Q&A requirements to build a local text knowledge base: D = {D1, D2, ... D i ,…,D n }, where D i represents the i-th multi-source heterogeneous text knowledge base in a specific industry. Considering the actual existence of the industry knowledge base, D i Contains different data types and multiple file formats, including tables and documents, and file formats including JSON, PDF, and XML; 2) Semantic integrity segmentation: divide text information into structured and unstructured data, specify the upper limit threshold of the length of a single sub-data as τ, and perform text segmentation based on semantic integrity to segment the knowledge base: D i ={d i1 ,d i2 ,…,d ij ,…,d in }, where d ij Represents the jth cutting sub-data in the i-th knowledge base; 3) Database conversion: Use the vector embedding model Embed(·) to convert text data into vector data and construct text-vector tuples: v ij =Embed M (d ij ) D i ={(d i1 ,v i1 ),(d i2 ,v i2 ),…,(d ij ,v ij ),…,(d in ,v in )} In the formula, Embed M (d ij ) means embedding the unit text information of the specified length upper limit threshold τ into the local knowledge base vector data v with a fixed length of M ij , the sub-dataset d ij Convert to text-vector tuple (d ij ,v ij ).

3. A large-model voice question-answering system for multi-source heterogeneous local knowledge bases according to claim 2, characterized in that: The semantic integrity segmentation divides local data into structured and unstructured data. For structured data, segmentation is performed according to independent data units; for unstructured data, segmentation is performed according to the priority: sub-data length upper limit threshold τ> segment> sentence, that is, within the length upper limit threshold, semantic segmentation is performed preferentially in units of segments and sentences to maximize the retention of the semantic integrity of the converted sub-data.

4. A large-model voice question-answering system for multi-source heterogeneous local knowledge bases according to claim 1, characterized in that: The human-computer speech interaction module provides a speech-to-text conversion interface, performs text-to-speech switching, and drives the digital human's expression and lip shape based on the open source model SadTalker, including the following steps: 1) Use the text-to-speech interface to convert the output text of the large model into speech: a {1,2,…,t} =TTS(R) In the formula, TTS(·) represents the text-to-speech conversion interface, which directly calls the existing Python open source library. {1,2,…,t} Represents the generated t-segment speech sequence; 2) The industry provides digital human portraits in picture or video format β0; 3) Combine the speech sequence with the digital human portrait to drive the changes in the digital human's expression and lip shape, and obtain a video of the digital human talking to the user. The generation method is: b {1,2,…,t} =Φ(Ψ A (a {1,2,…t} ),β0) Where, β {1,2,…,t} represents the generated digital human sequence, Ψ A (·) represents the audio encoder based on the residual network, which embeds the audio features into a latent space, takes the digital human portrait β0 as a condition, and uses the network Φ(·) as a feature decoder to generate the dialogue sequence β {1,2,…,t} .

Images