Dialogue method and system based on cross-document association retrieval and global intention recognition

Abstract

The invention provides a dialogue method and system based on cross-document association retrieval and global intention recognition. The dialogue method comprises the following steps: S1, importing unstructured or semi-structured texts in various formats; s2, converting the unstructured or semi-structured text into a structured text, and segmenting the structured text; s3, based on a double-flow parallel word embedding method, generating an embedding vector of the text segment; s4, obtaining a query text input by a user, and processing the query text by a calculation method based on double-flow parallel word embedding to generate two query vectors; s5, performing similarity calculation on the query vector and the embedded vector to obtain a similarity score, and selecting Top K text segments according to the similarity score; s6, filling the selected Top K text segments into a Prompt template, and calling the Prompt template into a large language model to generate an answer; the global intention of the user can be effectively identified, the continuity of the context can be kept in multiple rounds of dialogues, and the intelligent level and the user experience of the dialogue system are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a dialogue method and system based on cross-document association retrieval and global intent recognition. Background Art

[0002] Large language models (LLMs) are the hottest and relatively mature multi-round dialogue solutions in academia, and are widely used in daily chats, manual customer service, knowledge question and answer, and other scenarios. Although these models perform well in generating natural language, they still face some challenges, such as how to update knowledge, process long-tail data, and reduce data leakage. In order to cope with these problems, several new technical methods have emerged in recent years, including external association retrieval technologies such as retrieval-augmented generation (RAG) and LlamaIndex. These technologies solve the bottleneck of traditional LLM in processing large-scale knowledge by introducing non-parametric memory and external knowledge base.

[0003] With the rapid development of artificial intelligence technology, dialogue-based artificial intelligence systems have been widely used in all walks of life. However, existing dialogue systems generally have several technical problems: on the one hand, most systems rely on local information for knowledge acquisition and cannot effectively associate different fields or knowledge points across documents, resulting in poor accuracy of answers and poor contextual coherence; on the other hand, existing dialogue systems often cannot fully understand the user's global intentions, easily lose context in complex dialogue situations, and cannot provide accurate and intelligent responses.

[0004] The existing technologies to solve this technical problem are based on the architecture of knowledge-enhanced dialogue systems, which provide richer context and more accurate answers by combining large language models with external knowledge bases. Typical solutions include cross-document retrieval technology that combines document-level retrieval and generative models. The system supplements the shortcomings of generative models by retrieving relevant knowledge from multiple documents, avoiding the generated content being limited to the fixed knowledge base of the pre-trained model.

[0005] For example, similar solutions use cross-document knowledge retrieval technology. Based on the query entered by the user, the system can extract relevant information from multiple external documents and combine it with a large language model to generate answers that better meet the user's needs. Although these systems can effectively enhance the accuracy of conversations in some scenarios, they usually still have problems such as insufficient cross-document information integration and difficulty in context management.

[0006] Therefore, the current technologies on how to solve cross-document association retrieval and thus better implement global intent recognition still need to be further improved. Summary of the invention

[0007] In view of this, the present invention proposes a dialogue method and system based on cross-document association retrieval and global intent recognition, which solves the technical problems of information isolation, insufficient knowledge integration and inaccurate intent recognition in the dialogue system in the prior art.

[0008] The technical solution of the present invention is achieved in this way: The present invention provides a dialogue method based on cross-document association retrieval and global intent recognition, comprising the following steps: S1, imports unstructured or semi-structured texts in various formats based on the local knowledge base; S2, converting the unstructured or semi-structured text in step S1 into structured text, and segmenting the converted structured text to obtain different text segments; S3, generating an embedding vector of the text segment in step S2 based on a dual-stream parallel word embedding method; S4, obtaining the query text input by the user, and processing it by the calculation method based on dual-stream parallel word embedding described in step S3 to generate two query vectors; S5, performing similarity calculation on the query vector obtained in step S4 and the embedding vector obtained in step S3 to obtain a similarity score, and selecting the top K text segments according to the similarity score; S6, fill the Top K text segments selected in step S5 into the Prompt template, and call the large language model to generate an answer.

[0009] On the basis of this solution, further preferably, the dual-stream parallel word embedding method includes the following contents: Based on the BERT model and the BM25 model, dual-stream parallel word embedding processing is performed on the text segment.

[0010] Based on this solution, further preferably, the BERT model is used to generate an embedding vector of the text segment, the BM25 model is used to calculate the similarity between the text segment and the structured text, and the embedding vector is stored in a vector library.

[0011] On the basis of this solution, further preferably, the multiple formats of unstructured or semi-structured texts include PDF, xlsx, xls, csv, word, txt and markdown (md) texts.

[0012] On the basis of this solution, it is further preferred that the similarity calculation in step S5 comprises the following steps: The query vector obtained in step S4 and the embedding vector obtained in step S3 are weightedly fused using cosine similarity and BM25 score to generate a final similarity score.

[0013] On the basis of this solution, further preferably, the formula of the cosine similarity is: ; The calculation formula of BM25 is: .

[0014] Among them, A and B represent vectors, is the term in the query, D is the text segment, k 1 is an adjustable parameter, b is 0.0-1.0, and N is the number of all text segments in D.

[0015] Based on this solution, it is further preferred that after the similarity score is generated, if the similarity score is greater than 0.7, it means there is no reference; if the similarity score is less than 0.7, the top K text segments are selected. Based on this solution, it is further preferred that the large language model is the LLaMA3 ChatQA-8B large model.

[0016] Based on this solution, further preferably, the method supports multiple rounds of dialogue.

[0017] In addition, the present invention also provides a dialogue system based on cross-document association retrieval and global intent recognition, including: Import module, used to import unstructured or semi-structured texts in various formats into local knowledge base; A conversion module, used for converting the unstructured or semi-structured text in the import module into structured text, and segmenting the converted structured text to obtain different text segments; An embedding module, used for generating an embedding vector of the text segment in the conversion module by using a dual-stream parallel word embedding method; The query module is used to obtain the query text input by the user and generate two query vectors after being processed by the embedding module; A calculation module, used to calculate the similarity between the query vector obtained by the query module and the embedding vector obtained by the embedding module to obtain a similarity score, and select the top K text segments according to the similarity score; The result module is used to fill the Top K text segments selected by the calculation module into the Prompt template and call the large language model to generate answers.

[0018] The dialogue method and system based on cross-document association retrieval and global intent recognition of the present invention have the following beneficial effects compared with the prior art: In response to the prior art, the method provided by the present invention increases the recognition and reading structure, converts unstructured formats such as excel and doc into structured formats such as sql and json, and accelerates question and answering; increases training samples in the customer service field to better meet specific scenarios; and adds an intent recognition model before answering, greatly reducing the probability of answering irrelevant questions or answering incorrectly.

[0019] On the other hand, the system of the present invention can also solve the problems of information isolation, insufficient knowledge integration, inaccurate intent recognition, etc. in the existing dialogue system. Through cross-document knowledge association retrieval, the system can obtain relevant information more comprehensively, and through global intent recognition technology, accurately understand the user's deep intentions in multiple rounds of dialogue, thereby achieving a more intelligent, smooth and highly context-aware dialogue experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 This is a flow chart of the dialogue method based on cross-document association retrieval and global intent recognition described in Example 1 of the present invention; Figure 2 This is a diagram illustrating a conversation method based on cross-document association retrieval and global intent recognition as described in Example 2 of the present invention; Figure 3 The process of selecting the Top K calculation method described in Example 2 of the present invention; Figure 4 This is a framework diagram of the dialogue system based on cross-document association retrieval and global intent recognition described in Example 3 of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] It should be understood that the terms used in this description of the embodiments of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the embodiments of the present invention. As used in the description of the embodiments of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms of "a", "an" and "the" are intended to include plural forms.

[0024] The existing technology has limited retriever formats, which makes it impossible to convert data in multiple file formats such as excel and doc. At the same time, the model generalization ability is insufficient. LLM and RAG technologies are only trained in general fields, and lack the generalization ability for specific field tasks, requiring additional adjustment and training; therefore, the present invention provides a dialogue method and system based on cross-document association retrieval and global intent recognition, which solves the problems of information isolation, insufficient knowledge integration, inaccurate intent recognition, etc. in the existing dialogue system. Through cross-document knowledge association retrieval, the system can obtain relevant information more comprehensively, and through global intent recognition technology, accurately understand the user's deep intentions in multiple rounds of dialogue, thereby achieving a more intelligent, smooth and highly context-aware dialogue experience.

[0025] Example 1 A dialogue method based on cross-document association retrieval and global intent recognition, such as Figure 1 It can be seen that the following steps are included: S1, imports unstructured or semi-structured texts in various formats based on the local knowledge base; Before explaining step S1, some terms in the present invention are briefly explained. Multi-round dialogue refers to continuous dialogue based on contextual content to achieve the purpose of solving a certain type of specific task. Generally, a round of messages from one dialogue party to the start of sending messages to the start of others sending messages is defined as the message sender's message.

[0026] Specifically, import local knowledge base files in various formats, including PDF, xlsx, xls, csv, word, txt and md files. PDF, xlsx, xls, csv, word, txt and md files contain unstructured or semi-structured text; S2, converting the unstructured or semi-structured text in step S1 into structured text, and segmenting the converted structured text to obtain different text segments; Specifically, the unstructured or semi-structured text in step S1 is converted into structured text, for example, the PDF file is segmented according to punctuation marks, the xlsx, xls, and csv files are formed into text according to the content of each line, and the md file is converted according to the title in HTML format; The method provided by the present invention supports user-defined rules and segments the converted structured text to obtain different text segments.

[0027] S3, generating an embedding vector of the text segment in step S2 based on a dual-stream parallel word embedding method; Specifically, a dual-stream parallel word embedding method is introduced to avoid the impact of a single word embedding on the accuracy of text embedding; BERT word vector (text2vec): Starting from the embedding of the word itself, combined with the sentence order and language habits, the embedding vector of the text segment is generated; BM25 algorithm: Based on word frequency, sentence length and the number of non-repeated words, the BM25 model is used to calculate the similarity between the text segment and the structured text, and store the embedding vector in the vector library; S4, obtaining the query text input by the user, and processing it by the calculation method based on dual-stream parallel word embedding described in step S3 to generate two query vectors; Specifically, the user inputs a query statement, obtains the query text input by the user, performs text word embedding calculation on it in two parallel streams, and generates two query vectors; S5, performing similarity calculation on the query vector obtained in step S4 and the embedding vector obtained in step S3 to obtain a similarity score, and selecting the top K text segments according to the similarity score; Specifically, the query vector obtained in step S4 and the embedding vector obtained in step S3 are weightedly fused using cosine similarity and BM25 score to generate a final similarity score.

[0028] In this implementation, specifically, the formula for the cosine similarity is: ; The calculation formula of BM25 is: .

[0029] Among them, A and B represent vectors, is the term in the query, D is the text segment, k 1 is an adjustable parameter, b is 0.0-1.0, and N is the number of all text segments in D.

[0030] More specifically, after the similarity scores are generated, if the similarity score is greater than 0.7, it means that there is no reference; if the similarity score is less than 0.7, the top K text segments are selected.

[0031] S6, fill the Top K text segments selected in step S5 into the Prompt template, and call the large language model to generate an answer.

[0032] Specifically, the retrieved knowledge points are filled into the Prompt template, ready to be used as the input of the large model, and the LLaMA3 ChatQA-8B large model is used to generate answers; Answering rule: If the knowledge stored in the large model itself cannot generate an answer, then answer: "Sorry. I cannot find the answer based on the context.".

[0033] Supports multiple rounds of conversations, and historical conversations will be saved in the record; under the premise of sufficient video memory, supports no less than 10 rounds of continuous conversations, and the minimum running video memory requirement is 32GB.

[0034] Example 2 A dialogue method based on cross-document association retrieval and global intent recognition, such as Figure 2 It can be seen that the following steps are included: First, import the local knowledge base. The knowledge base file format is changed from the original txt format to PDF, md, excel (xlsx, xls, csv, etc.) format files. The unstructured or semi-structured text content is converted into structured text (PDF is segmented according to punctuation marks, xlsx, xls, csv are formed according to lines, and md is formatted according to HTML titles). The file can also be divided into different text segments or text segments in a custom format; Then, a dual-stream parallel word embedding method is introduced to avoid the influence of single word embedding on text embedding. Here, Bert word vector (text2vec, sota of text similarity) and sentence and document relevance algorithm based on BM25 are selected. The text2vec word vector mainly starts from the embedding of the word itself, superimposes the sentence word order, language habits, etc. to obtain the embedding vector of the text segment, while BM25 starts from the word frequency, sentence length, and number of non-repeated words in the sentence in the document, calculates the relevance score between the text segment and the entire document (or multiple documents), and stores it in the vector library.

[0035] Specifically, text2vec is a text vectorization tool that can represent text (including words, sentences, paragraphs) as vector matrices for tasks such as text similarity calculation. It implements multiple text representation and text similarity calculation models, similar to word2vec: The core principle of text2vec is to map text into vectors of fixed dimensions, and then evaluate the similarity between texts by calculating the cosine similarity between these vectors.

[0036] Specifically, the calculation formula of cosine similarity is:

[0037] Among them, A and B are vector representations of two text segments, θ is the angle between the two text segments, and ||A|| and ||B|| are the norms of the vectors.

[0038] Text2vec is trained using CoSENT, a supervised sentence vector solution that improves the accuracy of text similarity tasks by optimizing cosine similarity. Compared with traditional Sentence-BERT, the training process of CoSENT is more consistent with the prediction task because it directly optimizes cosine similarity instead of indirectly affecting similarity through classification tasks. The core of CoSENT is a new loss function. This paper applies CoSENT to English dataset training for the first time. The formula is:

[0039] in, , is the set of all positive and negative sample pairs, that is, the set of text fields that match correctly and the set of text fields that do not match. λ is a hyperparameter, usually set to 20. The purpose of the loss function is to make the cosine similarity of any matched question and answer in a batch greater than the cosine similarity of the unmatched question and answer, and through the calculation of cosine similarity, the model is allowed to focus on improving the similarity score of the matched question and answer, ensuring that the similarity score of the unmatched question and answer is reduced.

[0040] Specifically, the BM25 algorithm is an algorithm for calculating the relevance between sentences and documents. Its principle is: the input sentence is segmented, and then the relevance between each word in the sentence and the document doc is calculated separately, and then the weighted sum is performed. The relevance score between the sentence and the document is obtained. The core of the BM25 algorithm is to perform a weighted summation of the frequency (Term Frequency, TF) of the query terms in the document and the rarity (Inverse Document Frequency, IDF) in the entire document collection to evaluate the relevance between the document and the query.

[0041] Specifically, the scoring formula of BM25 is as follows:

[0042] Where D represents a document or text segment, Q represents a query, Represents the terms in the query, k is the number of words qi contained in the document or text segment D. 1 is an adjustable parameter, usually k 1=1.2 to 2.0, b is 0.0 to 1.0, len(D) is the length of the document, and avg_len is the average document length of the document collection.

[0043]

[0044] From the idf formula, we can see that The larger the denominator, the smaller the numerator, and the smaller the IDF value. This is because if a word appears in multiple documents, it can be said to be a commonly used word that exists in any sentence, which cannot reflect the particularity of the sentence, so it is given a smaller IDF value.

[0045] Where N represents the number of all documents in D, that is, the total number of documents used to calculate the relevance score with sentence, and 0.5 is the smoothing factor, which is a hyperparameter to prevent qi from not appearing in the text segment, resulting in the denominator being 0.

[0046] Thus, in the process of two-stream parallel text word embedding, the similarity matching score is ,in .

[0047] When a user enters a query statement, two query vectors of the text are calculated through dual-stream parallel text word embedding, and the similarity score is calculated by comparing with the vector library.

[0048] If the score is lower than 0.7 (similarity score between 0-1), there is no reference knowledge point. If the score is greater than 0.7, select the first k similar knowledge points; Specifically, the Top K selection method is as follows: Figure 3 As shown, including the following: Initial search: Enter all text segments (i.e. Figure 3 In the example, x represents a text segment, and multiple text segments and external contexts that are most relevant to the input query are retrieved through a search engine.

[0049] Similarity weighting and weighted fusion: The similarity scores of the query and the retrieved text are calculated by combining two main methods: Cosine Similarity: Measures the vector similarity between the query and the text segment.

[0050] BM25 score: Calculates the relevance of a text segment to a query based on term frequency and document length.

[0051] The final score of similarity matching is calculated by weighted fusion:

[0052] Among them, α and β are adjustable weight coefficients.

[0053] Re-ranking: Re-rank the initially retrieved text segments according to the fusion scores. Figure 3 As shown in Figure 3, the reranking module aligns the query vector with the retrieved text vectors, calculates the final score, and outputs the Top k most relevant text segments.

[0054] Transformer embedding and context fusion: The selected text segment and external context are input into the Transformer-Based Embedding model to further generate {v1,v2,v3...vn}.

[0055] Loss function optimization: The optimization objectives include (context constraint loss) and (Distribution Constraint Loss) to improve the quality of context matching.

[0056] Finally passed and Supervised learning is performed on the generated output to ensure that it matches the standard answer GoldLabels.

[0057] As can be seen above, the system completes the similarity matching between text segments and queries and answer extraction through the process of preliminary retrieval, similarity weighting and weighted fusion, Transformer embedding, reranking and context fusion. In this process, the accuracy of retrieval results and the overall performance of the system are improved by optimizing context matching and output loss.

[0058] The similarity score is calculated by the similarity matching between the query and the text segment, and the top k most relevant text segments are selected. In the model, k is set to 3.

[0059] Specifically, in this model, k is set to 3, and the knowledge points are filled into the prompt template. The large model LLaMA3ChatQA-8B needs to use the information stored in the large model itself to answer the question. If you do not have relevant knowledge, you can answer "Sorry. I cannot find the answer based on the context.", and the question and answer history will be saved in the historical dialogue; If there is sufficient video memory, the number of rounds of dialogue can be no less than 10 rounds.

[0060] Example 3 A dialogue system based on cross-document association retrieval and global intent recognition, such as Figure 4 Visible, including: Import module, used to import unstructured or semi-structured texts in various formats into local knowledge base; A conversion module, used for converting the unstructured or semi-structured text in the import module into structured text, and segmenting the converted structured text to obtain different text segments; An embedding module, used for generating an embedding vector of the text segment in the conversion module by using a dual-stream parallel word embedding method; The query module is used to obtain the query text input by the user and generate two query vectors after being processed by the embedding module; A calculation module, used to calculate the similarity between the query vector obtained by the query module and the embedding vector obtained by the embedding module to obtain a similarity score, and select the top K text segments according to the similarity score; The result module is used to fill the Top K text segments selected by the calculation module into the Prompt template and call the large language model to generate answers.

[0061] In summary, the present invention provides a dialogue method and system based on cross-document association retrieval and global intent recognition. Compared with the existing solutions, by introducing global intent recognition technology, the user's global intent can be effectively identified. It not only relies on local information, but also can maintain the consistency of context in multiple rounds of dialogues, further improving the intelligence level of the dialogue system and user experience.

[0062] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A dialogue method based on cross-document association retrieval and global intent recognition, characterized in that: The following steps are involved: S1, imports unstructured or semi-structured texts in various formats based on the local knowledge base; S2, converting the unstructured or semi-structured text in step S1 into structured text, and segmenting the converted structured text to obtain different text segments; S3, generating an embedding vector of the text segment in step S2 based on a dual-stream parallel word embedding method; S4, obtaining the query text input by the user, and processing it by the calculation method based on dual-stream parallel word embedding described in step S3 to generate two query vectors; S5, performing similarity calculation on the query vector obtained in step S4 and the embedding vector obtained in step S3 to obtain a similarity score, and selecting the top K text segments according to the similarity score; S6, fill the Top K text segments selected in step S5 into the Prompt template, and call the large language model to generate an answer.

2. The dialogue method based on cross-document association retrieval and global intent recognition as claimed in claim 1, characterized in that: The dual-stream parallel word embedding method includes the following contents: Based on the BERT model and the BM25 model, dual-stream parallel word embedding processing is performed on the text segment.

3. The dialogue method based on cross-document association retrieval and global intent recognition as claimed in claim 2, characterized in that: The BERT model is used to generate an embedding vector for the text segment, and the BM25 model is used to calculate the similarity between the text segment and the structured text, and store the embedding vector in a vector library.

4. The dialogue method based on cross-document association retrieval and global intent recognition as claimed in claim 1, characterized in that: The multiple formats of unstructured or semi-structured texts include PDF, xlsx, xls, csv, word, txt and md text.

5. The dialogue method based on cross-document association retrieval and global intent recognition as claimed in claim 1, characterized in that: The similarity calculation in step S5 includes the following steps: The query vector obtained in step S4 and the embedding vector obtained in step S3 are weightedly fused using cosine similarity and BM25 score to generate a final similarity score.

6. The dialogue method based on cross-document association retrieval and global intent recognition as claimed in claim 5, characterized in that: The formula for the cosine similarity is: ； The calculation formula of BM25 is: ； Among them, A and B represent vectors, is the term in the query, D is the text segment, k1 is an adjustable parameter, b is 0.0-1.0, and N is the number of all text segments in D.

7. The dialogue method based on cross-document association retrieval and global intent recognition as claimed in claim 5, characterized in that: After the similarity scores are generated, if the similarity score is greater than 0.7, it means there is no reference; if the similarity score is less than 0.7, the top K text segments are selected.

8. The dialogue method based on cross-document association retrieval and global intent recognition as claimed in claim 1, characterized in that: The large language model is the LLaMA3 ChatQA-8B large model.

9. The dialogue method based on cross-document association retrieval and global intent recognition as claimed in claim 8, characterized in that: The method supports multi-round dialogue.

10. A dialogue system based on cross-document association retrieval and global intent recognition, characterized in that: include: Import module, used to import unstructured or semi-structured texts in various formats into local knowledge base; A conversion module, used for converting the unstructured or semi-structured text in the import module into structured text, and segmenting the converted structured text to obtain different text segments; An embedding module, used for generating an embedding vector of the text segment in the conversion module by using a dual-stream parallel word embedding method; The query module is used to obtain the query text input by the user and generate two query vectors after being processed by the embedding module; A calculation module, used to calculate the similarity between the query vector obtained by the query module and the embedding vector obtained by the embedding module to obtain a similarity score, and select the top K text segments according to the similarity score; The result module is used to fill the Top K text segments selected by the calculation module into the Prompt template and call the large language model to generate answers.

Images