A multi-round dialogue processing method and system based on RAG and knowledge graph

Abstract

The present invention belongs to the field of artificial intelligence technology, and provides a multi-round dialogue processing method and system based on RAG and knowledge graph. The method includes: according to the proprietary knowledge graph, the entities associated with the user's question are retrieved and identified to obtain the associated entities to build the reasoning link and convert it into the reasoning text sequence; the text, picture, and audio data related to the user's question are retrieved through the multimodal large model to obtain the relevant multimodal data; the relevant multimodal data, the current reasoning text sequence, the user's question and the generated answer text are integrated as the model input, and the multimodal large model is input to obtain the model answer; the relevant multimodal data is screened out, and the screened multimodal data is used as the context and the problem to be processed as the model input of the multimodal large model to obtain the answer text corresponding to the problem to be processed. The present invention can effectively align the user's intention with the answer text while reducing the interaction.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multi-round dialogue processing method and system based on RAG and knowledge graph. Background Art

[0002] The existing multi-round dialogue methods are mainly based on the large language model technology (LLM). LLM is an important breakthrough in the current field of artificial intelligence. It is based on deep learning technology and can achieve efficient understanding and generation of natural language by training massive text data. Its structure usually adopts the transformer layer (Transformer) or its variants, and uses the self-attention mechanism to capture contextual information to understand the meaning of language. When building a multi-round dialogue system, first collect text data related to the dialogue task through data collection and preprocessing, and perform necessary cleaning and annotation. Then, use the pre-trained large language model for fine-tuning training to adapt it to the dialogue task in a specific field. In the dialogue generation stage, the system can receive user input, use the trained model to generate appropriate responses, and interact with the user in multiple rounds. However, since the large language model generates segments based on probability statistics, it is prone to hallucinations, which may lead to inaccurate or irrelevant answers. At the same time, due to the high cost of training and the requirement for privacy protection, large language models are limited in the selection and scale of training data. When facing specific fields or highly specialized problems, large language models may lack the necessary expertise.

[0003] In addition, retrieval-augmented generation (RAG) is an advanced artificial intelligence technology that combines the advantages of information retrieval and deep learning models to address the challenges of knowledge limitations and "hallucination" problems in large language models. The main structure of RAG includes three key parts: retrieval, utilization, and generation. In RAG, first, the system retrieves information related to the current task from a large document collection or knowledge base. Then, this information is used to guide subsequent text generation to ensure that the generated text content is not only reasonable but also accurate. Finally, in the generation stage, the system generates the final text content based on the retrieved knowledge and the capabilities of the language model. However, retrieval-augmented generation (RAG) also faces many challenges in practical applications. The quality of user queries varies, making it difficult to accurately parse user intent and retrieve appropriate information; while reducing interactions with LLM (Large Language Model), it is difficult to effectively align user intent with high-quality structured knowledge; the retrieved knowledge is prone to over-correlation and lack of diversity, so it is difficult to query all information related to the question. In addition, there is still a lot of room for improvement in many aspects such as how to improve retrieval speed and obtain more accurate answer texts.

[0004] Therefore, it is necessary to provide a multi-round dialogue processing method and system based on RAG and knowledge graph to solve the above problems. Summary of the invention

[0005] The present invention intends to provide a multi-round dialogue processing method and system based on RAG and knowledge graph to solve the technical problems in the prior art that it is difficult to accurately parse user intentions and retrieve appropriate information, how to effectively align user intentions with high-quality structured knowledge while reducing interactions with LLM (Large Language Model), and the retrieved knowledge is prone to over-correlation and lack of diversity. The technical problems to be solved by the present invention are achieved through the following technical solutions.

[0006] The first aspect of the present invention proposes a multi-round dialogue processing method based on RAG and knowledge graph, including: step S101: obtaining the user question of the current user, and based on the user question, a multimodal large model generates an answer text; step S102: performing entity recognition on the user question and the generated answer text to obtain entities associated with the user question; step S103: according to the proprietary knowledge graph, the entity associated with the user question is retrieved and identified to obtain the associated entity, or the re-entered response information is returned to the client; step S104: according to the obtained associated entity and the proprietary knowledge graph, an inference link is constructed, and the inference link is converted into an inference text sequence; step S105: based on the inference text sequence and the user question, the text, picture, and audio data related to the user question are retrieved through the multimodal large model to obtain relevant multimodal data; step S106: the relevant The multimodal data, the current inference text sequence, the user question and the generated answer text are integrated as model input, input into the multimodal large model, obtain the model answer corresponding to the user question, and put the user question and its model answer into the historical conversation list; step S107: receive the pending question of the current user, filter the multimodal data related to the pending question from the historical conversation list, when the multimodal data related to the pending question is not filtered out, repeat steps S101 to S106; when the multimodal data related to the pending question is filtered out, enter step S108; step S108: use the filtered multimodal data as context and together with the pending question of the current user as the model input of the multimodal large model, repeat steps S101 to S103 to obtain the answer text corresponding to the pending question.

[0007] The second aspect of the present invention proposes a multi-round dialogue processing system based on RAG and knowledge graph, which adopts the multi-round dialogue processing method based on RAG and knowledge graph described in the first aspect of the present invention, and the multi-round dialogue processing system includes: a multi-round dialogue processing system based on RAG and knowledge graph, which adopts the multi-round dialogue processing method based on RAG and knowledge graph described in the first aspect of the present invention, and the multi-round dialogue processing system includes: a text acquisition module, which is used to obtain the user question of the current user, and based on the user question, a multimodal large model generates an answer text; an entity recognition module, which is used to perform entity recognition on the user question and the generated answer text to obtain the entity associated with the user question; a retrieval and recognition module, which retrieves and recognizes the entity associated with the user question according to the proprietary knowledge graph, obtains the associated entity, or returns the re-entered response information to the client; a construction module, which constructs an inference link according to the obtained associated entity and the proprietary knowledge graph, and converts the inference link into an inference text sequence; a data processing module, which is based on the inference text sequence and the user question The first model processing module is used to integrate the relevant multimodal data, the current inference text sequence, the user question and the generated answer text as the model input, input the multimodal large model, obtain the model answer corresponding to the user question, and put the user question and its model answer into the historical conversation list; the screening module is used to receive the pending question of the current user, screen the multimodal data related to the pending question from the historical conversation list, and when the multimodal data related to the pending question is not screened out, return to the step executed by the text acquisition module; when the multimodal data related to the pending question is screened out, enter the next step; the second model processing module uses the screened multimodal data as the context and together with the pending question of the current user as the model input of the multimodal large model, returns to the step executed by the text acquisition module to obtain the answer text corresponding to the pending question.

[0008] The third aspect of the present invention provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the multi-round dialogue processing method based on RAG and knowledge graph described in the first aspect of the present invention.

[0009] The fourth aspect of the present invention provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the multi-round dialogue processing method based on RAG and knowledge graph described in the first aspect of the present invention.

[0010] The embodiments of the present invention include the following advantages:

[0011] Compared with the prior art, the present invention performs entity recognition on user questions and generated answer texts to obtain entities associated with user questions. According to the proprietary knowledge graph, the entities associated with the user questions are retrieved and identified to obtain associated entities; according to the obtained associated entities and the proprietary knowledge graph, an inference link is constructed, and the inference link is converted into an inference text sequence, and further related multimodal data is obtained. The related multimodal data, the current inference text sequence, the user question and the generated answer text are integrated as model input, and input into the multimodal large model to obtain a model answer corresponding to the user question. The current user's pending question is received, and the multimodal data related to the pending question is filtered out from the historical conversation list. When the multimodal data related to the pending question is not filtered out, the step of generating the model answer is repeated; when the multimodal data related to the pending question is filtered out, the filtered multimodal data is used as the context and together with the current user's pending question as the model input of the multimodal large model to obtain the answer text corresponding to the pending question. Specifically, the self-attention mechanism in the transformer layer (Transformer) of the multimodal large model is replaced by a test-time trained Kolmogorov Arnold network (TTT-KAN). The core principle of the test-time trained Kolmogorov Arnold network (TTT-KAN) is to replace the hidden state in the traditional recurrent neural network (RNN) with a machine learning model, and to compress the context information by using the actual gradient descent of the input label (i.e., token), which not only simplifies the model structure, but also achieves a significant improvement in performance. The present invention can effectively align user intent with the answer text while reducing interactions, thereby more accurately parsing user intent.

[0012] In addition, the DPE-MNER framework is adopted to improve the accuracy and efficiency of named entity recognition.

[0013] In addition, based on the proprietary knowledge graph, the entities associated with the user questions are retrieved and identified to obtain related entities, so as to construct reasoning links to form reasoning text sequences, and further retrieve relevant multimodal data, so as to obtain more accurate model answers while diversifying the model input.

[0014] In addition, by combining the parallel feature extraction part of the third-order transformation operator and convolution, the ability to extract high-order information is improved through parallel processing, which can effectively capture higher-order local features in the image data of the user's questions and related multimodal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1It is a flowchart of an example of a multi-round dialogue processing method based on RAG and knowledge graph of the present invention;

[0016] Figure 2 is a schematic diagram of user input in an application example of the multi-round dialogue processing method based on RAG and knowledge graph of the present invention;

[0017] Figure 3 It is a schematic diagram of the principle of the third-order transformation convolution processing part of the alignment neural network in the multi-round dialogue processing method based on RAG and knowledge graph of the present invention;

[0018] Figure 4 It is a schematic diagram of the principle of the sliding attention part of the alignment neural network in the multi-round dialogue processing method based on RAG and knowledge graph of the present invention;

[0019] Figure 5 This is a partial flow chart of another perspective of the multi-round dialogue processing method based on RAG and knowledge graph of the present invention.

[0020] Figure 6 It is a structural block diagram of the multi-round dialogue processing system based on RAG and knowledge graph of the present invention;

[0021] Figure 7 is a schematic structural diagram of an electronic device embodiment according to the present invention;

[0022] Figure 8 It is a schematic diagram of the structure of a computer-readable medium embodiment according to the present invention. DETAILED DESCRIPTION

[0023] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0024] In view of the above problems, the present invention proposes a multi-round dialogue processing method and system based on RAG and knowledge graph, which obtains entities associated with user questions by performing entity recognition on user questions and generated answer texts. According to the proprietary knowledge graph, the entities associated with the user questions are retrieved and identified to obtain related entities, or re-entered response information is returned to the client; based on the obtained related entities and proprietary knowledge graph, an inference link is constructed, and the inference link is converted into an inference text sequence, and further related multimodal data is obtained. The related multimodal data, the current inference text sequence, the user question and the generated answer text are integrated as model input, and input into the multimodal large model to obtain a model answer corresponding to the user question. Receive the pending questions of the current user, filter the multimodal data related to the pending questions from the historical conversation list, and when the multimodal data related to the pending questions are not filtered out, repeat the step of generating the model answer; when the multimodal data related to the pending questions are filtered out, use the filtered multimodal data as the context and together with the pending questions of the current user as the model input of the multimodal large model to obtain the answer text corresponding to the pending questions. Specifically, the self-attention mechanism in the Transformer layer of the multimodal large model is replaced with the Kolmogorov Arnold network (TTT-KAN). The core principle of the TTT model is to replace the hidden state in the traditional RNN with a machine learning model, and use the actual gradient descent of the input label (i.e., token) to compress the context information, which not only simplifies the model structure, but also achieves a significant improvement in performance.

[0025] Example 1

[0026] Refer to the following Figures 1 to 5 , the contents of the present invention will be described in detail.

[0027] Figure 1 This is a step flow chart of an example of the multi-round dialogue processing method based on RAG and knowledge graph of the present invention. Figure 2 It is a schematic diagram of user input in an application example of the multi-round dialogue processing method based on RAG and knowledge graph of the present invention.

[0028] like Figure 1 As shown, in step S101, the user question of the current user is obtained, and based on the user question, the multimodal large model generates an answer text.

[0029] Specifically, user questions include text and images.

[0030] It should be noted that since the answer text generated by the multimodal large model based on user queries in the prior art may have insufficient professional knowledge or hallucination problems, the present invention improves the multimodal large model and introduces proprietary knowledge graphs related to the applied application scenarios in subsequent steps to optimize the retrieval process, and combines related text, voice, image, and video to form multimodal data, so as to gradually expand and enhance the model's answers, thereby obtaining more accurate answer texts.

[0031] The multimodal large model of the present invention is obtained by improving the existing open source multimodal large model. Specifically, the self-attention mechanism in the Transformer layer of the multimodal large model is replaced with the Kolmogorov Arnold network (TTT-KAN). The core principle of the TTT model is to replace the hidden state in the traditional RNN with a machine learning model and use the actual gradient descent of the input label (i.e., token) to compress context information. Through the above improvements, not only the model structure is simplified, but also the performance is significantly improved.

[0032] In traditional recurrent neural networks (RNNs), hidden states are used to store and transmit contextual information in sequences. The TTT model uses a different mechanism to capture and represent dependencies in sequence data by replacing this traditional hidden state with a machine learning model. The TTT model directly uses the gradient information of input data (such as tokens) during training to more efficiently compress and encode contextual information. This helps reduce the model's reliance on a large number of parameters, thereby improving computational efficiency and model performance.

[0033] Specifically, the Kolmogorov Arnold Network (TTT-KAN) replaces the self-attention mechanism in the Transformer, unlocking the potential of the linear complexity architecture, allowing training in the context of user questions to contain larger orders of magnitude of labels (such as millions or even billions of tokens), thereby achieving larger multimodal models with higher accuracy and a wider range of applications.

[0034] In a specific implementation, a user searches for a user question about investing in a fund, wherein the user question specifically includes text and images. Figure 2 .

[0035] Input the user's question into the multimodal big model, and get the answer text (i.e., the model answer), which will be recorded in the historical conversation list. Specifically, each time a user question is input, an answer text is obtained, which will be recorded once in the historical conversation list. When the improved multimodal big model receives the input of the user's question, it queries the historical conversation list to see whether there is multimodal data such as video, text, and image related to the user's question.

[0036] If there is multimodal data such as video, text, image, etc. related to the user's question, the multimodal data and the user's current input are used as the user's question. If there is no multimodal data such as video, text, image, etc. related to the user's question, or the historical conversation list is empty, only the user's question is used as the model input.

[0037] Preferably, the user question is matched with a preset prompt template (Prompt), and an appropriate preset prompt template is selected to gradually guide the multimodal large model to gradually generate an answer text (i.e., a possible answer). The answer text contains model "hallucinations" or lacks professional knowledge, so it only serves as the initial answer of the model and will be modified later.

[0038] The preset prompt templates are as follows: "Based on the following text, reason step by step to answer the questions.", "Based on the latest economic data and policy changes, evaluate the potential risks of investing in the following funds.", "Using the following market data, predict the stock market trend in the next quarter and give reasons.", "Considering the current economic environment and expected rate of return, suggest how to adjust my current investment portfolio to optimize risk and return."

[0039] Specifically, by calculating the cosine similarity between the user question and the preset prompt template, the preset prompt template most relevant to the user question is selected for input into the multimodal large model together with the user question.

[0040] In another specific implementation, in a legal dialogue application scenario, the query request of the current user is obtained, and the first user question is obtained from the query request, which specifically includes a text part "I have recently encountered some labor disputes, and the company did not pay my salary on time. What should I do?" and a picture part of "Labor Contract".

[0041] Next, the user question is input into the multimodal large model to generate an answer text, such as "When facing a labor dispute, you first need to understand the relevant legal provisions and procedures, and identify the content of the labor contract. Usually, labor disputes can be resolved through negotiation. If negotiation is ineffective, legal means can be sought."

[0042] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.

[0043] Next, in step S102, entity recognition is performed on the user question and the generated answer text to obtain entities associated with the user question.

[0044] Specifically, the user's questions and the generated answer texts are identified through the multimodal large model's named entity recognition to identify the named entities (i.e., key entities) of each application scenario and add them to the corresponding named entity set. When performing multimodal named entity recognition, for example, the DPE-MNER framework is used to improve the accuracy and efficiency of named entity recognition.

[0045] It should be noted that the DPE-MNER framework is an innovative iterative reasoning framework that dynamically integrates diverse multimodal representations through a "decomposition, priority, and elimination" strategy. The core of this framework is to decompose the fusion of multimodal information into hierarchical and interconnected fusion layers, emphasizing the gradual transition from simple to complex and from macro to micro, and explicitly modeling cross-modal correlations to exclude irrelevant information that may mislead predictions.

[0046] For example, entity recognition is performed on the user question and the generated answer text to obtain key entities associated with the user question, such as "dizziness" and "reason". For example, "labor dispute" and "salary".

[0047] For example, user questions and generated answer texts are used as model inputs, and the named entity recognition model is called to identify entities with specific meanings from the model inputs, such as names of people, places, organizations, dates, times, proper nouns, etc. related to medicine, law, finance, and education.

[0048] It should be noted that since the multimodal large model has the function of named entity recognition, the multimodal large model can be used directly to complete the named entity recognition task. For specific professional fields, such as medicine, law, finance, education and other application directions, the named entity recognition model of the above fields can also be trained separately, or the multimodal large model can be fine-tuned on the data in the field.

[0049] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.

[0050] Next, in step S103, the entities associated with the user question are searched and identified according to the proprietary knowledge graph to obtain associated entities, or re-entered response information is returned to the client.

[0051] Specifically, according to the proprietary knowledge graph, the entities associated with the user's question are searched and identified to obtain the associated entities in the proprietary knowledge graph, or the re-entered response information is returned to the client. The proprietary knowledge graph is a knowledge graph related to multiple application directions such as medical, legal, medical, legal, financial, and educational. It can be an existing knowledge graph, or a proprietary knowledge graph can be self-constructed based on the existing knowledge graph. The associated entities are, for example, names of people, places, organizations, dates, times, proper nouns, etc. related to multiple application directions such as medical, legal, medical, legal, financial, and educational.

[0052] Specifically, through the multimodal large model, semantic analysis is performed to locate the entities (i.e. key entities) corresponding to the user questions (i.e. user query questions) in the proprietary knowledge graph, and the entities directly connected to the user questions are placed in the named entity set.

[0053] If the entity extracted from the output answer text is indirectly connected to the key entity in the user's question through, for example, four intermediate entities, then the entity extracted from the output answer text is considered to be related to the key entity in the user's question and is retained in the named entity set. Otherwise, the entity is removed from the named entity set. Finally, the entity in the named entity set is used as an associated entity.

[0054] Optionally, the reasoning link containing an indirect relationship includes one or two entity nodes having the indirect relationship.

[0055] The associated entities matched in the proprietary knowledge graph are divided into two categories: associated entities that match user questions; and associated entities that match the generated answer text (i.e., initial answers).

[0056] For example, entities related to "labor disputes" are found in the proprietary knowledge graph, including "Labor Contract Law", "Labor Arbitration", "Court Litigation", "Wages", "Adjustment" and other entities. Among them, "wages" are a common point of dispute in labor disputes. For example, related entities related to "anemia" are found in the proprietary knowledge graph, such as "iron-containing foods", specifically "red meat", "green leafy vegetables", "beans", etc.

[0057] In an optional implementation, if an entity associated with the user question cannot be retrieved in the proprietary knowledge graph, a re-entry response message is returned to the current user's client, that is, the current user is asked to re-enter the query text or user question.

[0058] Based on the identified entities, entities associated with the user's question are identified in the proprietary knowledge graph, irrelevant entities are discarded, and the named entity set is updated.

[0059] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.

[0060] Next, in step S104, a reasoning link is constructed based on the obtained associated entities and proprietary knowledge graph, and the reasoning link is converted into a reasoning text sequence.

[0061] In this embodiment, a reasoning link is constructed based on the obtained associated entities and proprietary knowledge graph, and the reasoning link is converted into a reasoning text sequence.

[0062] Build reasoning links by matching entities in the named entity set with entities in the proprietary knowledge graph. Find the relationships between entities and how they are connected to each other in the proprietary knowledge graph. Then convert the constructed reasoning links into natural language text through a multimodal large model.

[0063] Specifically, a first type of related entity matching the problem to be processed is selected from the proprietary knowledge graph, and a subgraph is constructed using an entity selection algorithm with the first type of related entity as the center to generate multiple reasoning links.

[0064] The entity picking algorithm is used to construct a subgraph to generate multiple reasoning links, which specifically includes the following steps:

[0065] Step S201: From the proprietary knowledge graph, select related entities that match the problem to be processed and whose entity distance is less than a first specific value M to form a first set E1.

[0066] Step S202: From the proprietary knowledge graph, further select related entities that match the initial answer of the model and whose entity distance is less than a second specific value L to form a second set E2, and the first specific value M is greater than the second specific value L.

[0067] Step S203: Select nodes in the first set and the second set that intersect with the first set to form a subgraph, and generate multiple triples according to the subgraph to construct an inference link.

[0068] Specifically, based on the subgraph, a triple (including entity, relationship, entity) is generated, and one or more triples are used to build a reasoning link. Each node corresponds to an entity.

[0069] Furthermore, the constructed reasoning link is input into the multimodal large model to obtain a corresponding reasoning text sequence.

[0070] For example, the reasoning link is in the form of a triple <entity, relationship, entity>. For example, for a financial knowledge graph, the reasoning link is <interest rate, impact, bond price>, <company, issue, bond>, <bond, has, interest rate>. Thus, the generated reasoning text sequence is "In the financial market, companies issue bonds to investors in order to raise funds. These bonds have a fixed interest rate, that is, the interest rate that the company promises to pay regularly during the bond term... When making investment decisions, investors will pay close attention to changes in market interest rates and their potential impact on bond prices and expected returns."

[0071] For example, based on the symptoms of dizziness, possible causes and items that need further examination can be inferred. For example, the reasoning link includes two links, specifically "anemia"-"examination"-"hemoglobin level", "hypoglycemia-"examination"-"blood sugar level".

[0072] It should be noted that the above is only described as an optional example and should not be construed as limiting the present invention.

[0073] Next, in step S105, based on the inference text sequence and the user question, the multimodal big model is used to retrieve text, pictures, and audio data related to the user question to obtain relevant multimodal data.

[0074] Specifically, the relevant multimodal data includes text, pictures, voice data, and video data.

[0075] When the embedding model of the multimodal large model is called, the inference text sequence, the user question and the multimodal data are embedded into the same vector space, and each multimodal data storage block calculates the corresponding cosine similarity with the embedding vectors of the inference text sequence and the user question respectively to obtain all relevant multimodal data.

[0076] In a specific implementation, the stored multimodal data is divided into several sub-blocks, and each sub-block is preprocessed. When the mapping model of the multimodal large model is called, the inference text sequence, user questions and multimodal data are mapped to the same vector space, and the corresponding cosine similarity is calculated for each sub-block of the multimodal data block with the mapping vectors of the inference text sequence and the user question, and the retrieved multimodal data is sorted from high to low according to the vector similarity, and the first 10 segments of data are obtained as the relevant multimodal data.

[0077] In another specific embodiment, the multimodal data in the knowledge base matching the current user is divided to form multimodal data segments. When the mapping model of the multimodal large model is called, the multimodal data segments, the inference text sequence of the current user, and the user question are mapped into the same vector space, and the vector similarity of each multimodal data segment and the mapping vector of the inference text sequence and the user question is calculated (for example, the cosine similarity is calculated) to filter out the relevant multimodal data related to the user question from the multimodal data segments.

[0078] Specifically, when the multimodal big model is used to retrieve multimodal data related to the user's question, relevant pictures, videos, texts, and voice data are retrieved, such as descriptions of common symptoms of anemia, first aid measures for hypoglycemia, etc.

[0079] According to the proprietary knowledge graph, the entities associated with the user's question are retrieved and identified to obtain related entities, so as to construct reasoning links to form reasoning text sequences, and further retrieve relevant multimodal data, so as to obtain more accurate model answers while diversifying the model input.

[0080] It should be noted that the above is only described as an optional example and should not be construed as limiting the present invention.

[0081] Next, in step S106, relevant multimodal data, the current inference text sequence, user questions and the generated answer text are integrated as model input, input into the multimodal large model, and a model answer corresponding to the user question is obtained, and the user question and its model answer are placed in a historical conversation list.

[0082] Specifically, the relevant multimodal data, the current inference text sequence, the user question and the generated answer text are respectively represented by vectors, wherein the relevant multimodal data are mapped to the text vector space using an aligned neural network, specifically including combining a third-order transformation operator with a convolution operation to perform parallel extraction of high-order local features.

[0083] For example, multimodal data such as images, voices, and videos are passed through their respective encoders to obtain corresponding encoding vectors. After the alignment neural network, the encoding vectors of the multimodal data are mapped to the text vector space, so that the cosine similarity of the corresponding vector representations of text, images, voices, videos, and other information with similar content is as large as possible. The image encoding vector and the text vector are aligned through a matrix.

[0084] The alignment neural network includes a third-order transformation convolution processing part, a sliding attention part and a convolution attention part.

[0085] For the third-order transform convolution processing part, such as Figure 3As shown, the third-order transform convolution processing part is a combination of a third-order transform operator and a convolution (corresponding to Figure 3 The parallel feature extraction part of the "convolution operator" shown in the figure improves the ability to extract high-order information through parallel processing, and can effectively capture higher-order local features in the image data of the user's question and the image data of the related multimodal data.

[0086] A third-order transformation operator is used to perform three transformations on the image data in the related multimodal data. Specifically, three function transformations are performed on each pixel to extract high-order features of the image, wherein the functions corresponding to the three transformations are all learned by the aligned neural network.

[0087] Specifically, the third-order transform convolution part (TConv) is mainly divided into two branches. The difference between the two branches lies in whether the high-order feature extraction is performed through the cubic operator. In the secondary branch, the feature map passes through the convolution layer, batch normalization layer and activation function in sequence, which is equivalent to the residual module. Compared with the simple use of residual connection, the structure of the third-order transform convolution part achieves higher accuracy. In the main branch, the feature map first passes through the third-order transform operator. The third-order transform operator (Cubic Operator) performs cubic transformations on the feature map pixel by pixel to extract the high-order information of the image. For each pixel, according to the formula y=ax+bx 2 +cx 3 Then, the transformation result is subjected to feature fusion through convolution, and then the features are normalized through the batch normalization layer. Finally, SiLU is used as the activation function of this layer in the hope of retaining more information. The formula is as follows:

[0088] y=A·X+B·X 2 +C·X 3 O=SiLU( BN( Conv(y) ))

[0089] Among them, X is the input tensor, X 2 is the point-wise quadratic value of the input tensor, X 3 is the point-wise cubic of the input tensor, y is the tensor of the input tensor after three transformations; · is the Hadamard product, that is, point-by-point multiplication, BN is the batch normalization layer, SiLU is the activation function, Conv is the convolutional layer, and O is the output of the module.

[0090] It should be noted that, similar to Kolmogorov-Arnold Networks (KAN), the third-order transformation operator module first transforms each pixel through a cubic function. This cubic function is learned by the aligned neural network and can be regarded as a three-time Taylor expansion of the complex transformation function, which replaces the traditional neural network linear weighted calculation method, thus improving the nonlinear ability of the network. In addition, in the main branch, the result of the third-order transformation operator, convolution layer, batch normalization layer and activation function layer is similar to the two-layer network in KAN. The difference is that in order to facilitate calculation and reduce the number of parameters, only one cubic transformation is performed on a single pixel. Since the feature map contains multiple channels, and the pixel points of each channel are obtained by complex transformation of the original image (such as the image data in the user's question and the image data of related multimodal data), only one cubic transformation of a single pixel point in each channel is sufficient to extract certain information. Since high-order nonlinearity is introduced into the network structure of the present invention, the difficulty of feature extraction will be increased. Therefore, the stability of the network is enhanced by introducing batch normalization and other regularization operations.

[0091] Next, the sliding attention part will be explained. As shown in 4, the sliding attention part includes a roll operator, an attention operator, and an upsampling operator.

[0092] Specifically, in the sliding attention part, local features are collected to form a feature map by sliding windows and cross attention. The following expression is specifically used to make the size of the feature map input to the sliding attention part and the size of the feature map output by the sliding attention part unchanged. Specifically, the feature map X input to the sliding attention part is divided into blocks in the width and height dimensions to obtain m×n sub-tensors. Each sub-tensor and its transpose are matrix multiplied with the prior feature block to obtain M k1ij .

[0093] Select the number of channels of any sub-tensor and the corresponding prior feature block P k2 The number of channels is the same. When performing matrix multiplication, each channel performs matrix multiplication separately, and each channel feature map is added along the channel direction to obtain a matrix tensor, which is specifically expressed by the following expression:

[0094] ;

[0095] Among them, M k1ij is the i-th row and j-th column subtensor of the k1-th channel of the feature map of the input sliding attention part, X c1ij is the i-th row and j-th column subvector in the m×n subtensor of the c1-th channel obtained by block division, P k1cis the matrix tensor of the c1th channel of the k1th prior feature block, i, j, c are all positive integers, and C is a positive integer;

[0096] ;

[0097] Among them, M k2ij is the i-th row and j-th column subtensor of the k2-th channel, X c2ij is the i-th row and j-th column sub-vector in the m×n sub-tensor of the c2-th channel obtained by block division, P k2c is the matrix tensor of the c2th channel of the k2th prior feature block, i, j, c are all positive integers, and C is a positive integer.

[0098] It should be noted that the sliding attention module (SAM) draws on the advantage of the convolution operation in extracting local features through sliding. To ensure that the size of the output feature map remains unchanged, the width and height of the above two are set to the same value. The prior feature block can be regarded as the key matrix transposed and multiplied by the value matrix and then transformed. Since it will pass through the batch normalization layer later, it is not normalized here. The number of prior feature blocks determines the number of channels of the output feature map, and the number of output channels is twice the number of feature blocks. In addition, the traditional attention mechanism performs similarity calculations on the query matrix and the key matrix, and then multiplies it with the value matrix to obtain the final vector representation, but this operation requires a large amount of calculation, which affects the real-time performance of the network. Therefore, this part is improved. By sliding windows and cross-attention, local features of image data such as image data in user questions and related multimodal data are collected, and finally feature maps of image data such as image data in user questions and related multimodal data are formed.

[0099] For the convolutional attention part (CAtten), the original image (such as the image data in the user question and the image data of the related multimodal data) is first subjected to a convolution operation, and features are extracted from different sub-regions of the original image to obtain a feature map of the original image. When performing the convolution operation, considering that the shapes of the objects to be predicted in the image are mostly rectangular objects in different orientations, multiple dilated convolutions are used to perform the convolution operation, so that the final effective receptive field of the convolution module presents an isotropic circle. Then, the convolution feature map is sent to the attention part. The following expression is used to represent the attention function:

[0100] ;

[0101] Among them, A(Q,K,V) is the attention function, Q is the matrix composed of all the data of the user's question (i.e., the query vector), K is the matrix composed of the key vector, and K Tis the inverse matrix of the matrix of key vectors, V is the matrix of value vectors, and d is the dimension of key vectors. Softmax is an activation function that takes the index of each point along the corresponding dimension and then normalizes it. However, unlike natural language processing, the vector with a depth of d on each pixel in the feature map is regarded as a semantic basic unit.

[0102] In the convolutional attention part, the feature maps of image data such as image data in user questions and related multimodal data undergo a series of attention processes to integrate the information within the model.

[0103] Among them, the attention mechanism can help the model learn the dependencies between different vectors of the feature map and the important areas within the feature map. Finally, the upsampling operator is used to obtain a feature map of the same size as the input, which is output through the residual module.

[0104] After the above-mentioned integration of the information within the model, a model answer corresponding to the user question is obtained, and the user question and its model answer are put into a historical conversation list.

[0105] It should be noted that the above is only described as an optional example and should not be construed as limiting the present invention.

[0106] Next, in step S107, the pending issues of the current user are received, and multimodal data related to the pending issues are filtered out from the historical conversation list. When the multimodal data related to the pending issues are not filtered out, steps S101 to S106 are repeated; when the multimodal data related to the pending issues are filtered out, step S108 is entered.

[0107] Specifically, a pending question (such as a new question) of the current user is received, and multimodal data related to the pending question is filtered from a historical conversation list.

[0108] When the multimodal data related to the problem to be processed is not screened out, the problem to be processed is matched with the preset prompt template, and a matching preset prompt template is selected to guide the multimodal large model to generate an initial answer, and then return to step S101, and repeat steps S101 to S106, that is, repeatedly generate multiple rounds of model answers.

[0109] Preferably, the user question is matched with a preset prompt template (Prompt), and a suitable preset prompt template is selected to gradually guide the multimodal large model to gradually generate the answer text (ie, the model answer).

[0110] Specifically, by calculating the cosine similarity between the user question and the preset prompt template, the preset prompt template most relevant to the user question is selected for input into the multimodal large model together with the user question.

[0111] When the multimodal data related to the problem to be processed is screened out, the process proceeds to step S108.

[0112] In a specific embodiment, Figure 5 As shown, in the user query module, the user question is input into the multimodal model, and the answer text is output. Then, the user question and the answer text are input into the multimodal large model in the named entity recognition module, and the named entity (i.e., the obtained associated entity) is output. Then, in the knowledge graph query module, based on the knowledge graph (such as a proprietary knowledge graph), a reasoning link is constructed specifically according to the obtained associated entity and the proprietary knowledge graph, and the reasoning link is converted into a reasoning text sequence. Then, in the multimodal retrieval module, the user question, the reasoning text sequence, and the multimodal data (such as pictures, videos, texts, voices, etc.) are input into the multimodal large model to obtain relevant multimodal data. Finally, in the multimodal generation module, the relevant multimodal data is used as context and as model input together with the user question (or the problem to be processed), and the multimodal large model is input to obtain the final output of the model.

[0113] It should be noted that the above is only described as an optional example and should not be construed as limiting the present invention.

[0114] Next, in step S108, the filtered multimodal data is used as the context and together with the current user's pending question as the model input of the multimodal large model, and steps S101 to S103 are repeatedly executed to obtain the answer text corresponding to the pending question.

[0115] Specifically, when multimodal data related to the problem to be processed is screened out, the screened multimodal data is used as the context and together with the current user's problem to be processed as the model input of the multimodal large model to obtain the answer text corresponding to the problem to be processed.

[0116] In another specific implementation, the screened multimodal data is used as context and integrated with the pending issues of the current user.

[0117] Specifically, the relevant multimodal data including images, voice, video, etc., the inference text sequence, the current user's pending questions, and the initial answer are input into the corresponding pre-trained encoder model to obtain the corresponding vector representation. For the vector representation of multimodal data, the multimodal data is mapped to the text vector space through the alignment neural network, that is, the multimodal data is converted into a multimodal large model (that is, a vector that the language model can understand, such as a specific latitude). Finally, it is sent to the Transformer of the multimodal large model to obtain the model output text, that is, the answer text, and the pending question and the corresponding answer text are placed in the historical conversation list.

[0118] It should be noted that when multimodal data is mapped to the text vector space, each language text may be split into different labels (i.e. tokens), and each token is a specific vector. For example, the vector [1.2, 1.4, 1.7, 2.0] can be represented as a token, and each word, such as patent, may be two tokens, which means that it will eventually be represented as two vectors. In addition, in order for the large language model to understand the image, the image needs to be converted into multiple tokens, and the large language model needs to be able to accurately understand these multiple tokens, that is, semantic alignment must be performed. The above alignment neural network is performing semantic alignment. The above is only explained as an optional example and should not be understood as a limitation of the present invention.

[0119] For example, the filtered multimodal data is used as context and integrated with the current user's pending question and then input into the multimodal large model to obtain an answer text corresponding to the pending question, such as outputting "Based on the dizziness symptoms you described, dizziness may be related to anemia or hypoglycemia. It is recommended that you go to the hospital as soon as possible to check your blood routine and blood sugar levels. At the same time, here are some related health education videos and materials that you can learn about."

[0120] For example, when a user inputs a new question, the cosine similarity is calculated through the embedding model of the multimodal large model to analyze whether the new question is correlated with the elements in the historical conversation list. If the cosine similarity of the two vectors is greater than 0.8, the two vectors are considered to be correlated, and the corresponding multimodal data is concatenated with the user question to obtain a new user question. Otherwise, the original user question is retained and sent to the multimodal large model.

[0121] The steps of generating answer texts are repeated in multiple rounds. Regardless of whether the user question and the generated answer text, i.e. the model answer, are related to the historical conversation, the interaction will be recorded in the historical conversation list, i.e. the historical conversation list will be updated to provide a reference for subsequent interactions.

[0122] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.

[0123] In addition, the drawings are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the drawings do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0124] Compared with the prior art, the present invention performs entity recognition on user questions and generated answer texts to obtain entities associated with user questions. According to the proprietary knowledge graph, the entities associated with the user questions are retrieved and identified to obtain associated entities; according to the obtained associated entities and the proprietary knowledge graph, an inference link is constructed, and the inference link is converted into an inference text sequence, and further related multimodal data is obtained. The related multimodal data, the current inference text sequence, the user question and the generated answer text are integrated as model input, and input into the multimodal large model to obtain a model answer corresponding to the user question. The current user's pending question is received, and the multimodal data related to the pending question is filtered out from the historical conversation list. When the multimodal data related to the pending question is not filtered out, the step of generating the model answer is repeated; when the multimodal data related to the pending question is filtered out, the filtered multimodal data is used as the context and together with the current user's pending question as the model input of the multimodal large model to obtain the answer text corresponding to the pending question. Specifically, the self-attention mechanism in the Transformer layer of the multimodal large model is replaced with TTT-KAN. The core principle of the TTT model is to replace the hidden state in the traditional RNN with a machine learning model, and use the actual gradient descent of the input label (i.e., token) to compress context information, which not only simplifies the model structure, but also achieves significant performance improvement. The present invention accurately parses user intentions while reducing interactions and effectively aligns user intentions with answer texts.

[0125] In addition, the DPE-MNER framework is adopted to improve the accuracy and efficiency of named entity recognition.

[0126] In addition, based on the proprietary knowledge graph, the entities associated with the user questions are retrieved and identified to obtain related entities, so as to construct reasoning links to form reasoning text sequences, and further retrieve relevant multimodal data, so that more accurate model answers can be obtained while diversifying the model input.

[0127] In addition, by combining the parallel feature extraction part of the third-order transformation operator and convolution, the ability to extract high-order information is improved through parallel processing, which can effectively capture higher-order local features in the image data of the user's questions and related multimodal data.

[0128] Example 2

[0129] The following is a system embodiment of the present invention, which can be used to implement the method embodiment of the present invention. For details not disclosed in the system embodiment of the present invention, please refer to the method embodiment of the present invention.

[0130] Figure 6 It is a structural diagram of an example of a multi-round dialogue processing system based on RAG and knowledge graph according to the present invention.

[0131] The second aspect of the present disclosure provides a multi-round dialogue processing system 400 based on RAG and knowledge graph, which adopts the multi-round dialogue processing method based on RAG and knowledge graph described in the first aspect of the present invention.

[0132] like Figure 6 As shown, the multi-round dialogue processing system 400 includes a text acquisition module 410, an entity recognition module 420, a retrieval recognition module 430, a construction module 440, a data processing module 450, a first model processing module 460, a screening module 470, and a second model processing module 480.

[0133] In a specific implementation, the text acquisition module 410 is used to obtain the user question of the current user, and based on the user question, the multimodal large model generates an answer text. The entity recognition module 420 is used to perform entity recognition on the user question and the generated answer text to obtain the entity associated with the user question; the retrieval recognition module 430 retrieves and identifies the entity associated with the user question according to the proprietary knowledge graph to obtain the associated entity, or returns the re-entered response information to the client. The construction module 440 constructs an inference link according to the obtained associated entity and the proprietary knowledge graph, and converts the inference link into an inference text sequence; the data processing module 450 retrieves the text, picture, and audio data related to the user question through the multimodal large model based on the inference text sequence and the user question to obtain relevant multimodal data. The first model processing module 460 is used to integrate the relevant multimodal data, the current inference text sequence, the user question and the generated answer text as the model input, input the multimodal large model, obtain the model answer corresponding to the user question, and put the user question and its model answer into the historical dialogue list. The screening module 470 is used to receive the pending questions of the current user, screen the multimodal data related to the pending questions from the historical conversation list, and when the multimodal data related to the pending questions is not screened out, return to the text acquisition module to perform the steps; when the multimodal data related to the pending questions is screened out, enter the next step. The second model processing module 480 uses the screened multimodal data as context and together with the pending questions of the current user as the model input of the multimodal large model, returns to the text acquisition module, executes the steps of the text acquisition module, and obtains the answer text corresponding to the pending questions.

[0134] Specifically, when the multimodal data related to the problem to be processed is not screened out, the problem to be processed is matched with the preset prompt template, and a matching preset prompt template is selected to guide the multimodal large model to generate an initial answer, and steps S101 to S106 (i.e., steps S101 to S106 in Example 1) are repeated, that is, multiple rounds of answers are repeatedly generated.

[0135] According to an optional implementation, relevant multimodal data, the current inference text sequence, user questions and the generated answer text are integrated as model input, input into the multimodal large model, and a model answer corresponding to the user question is obtained, and the user question and its model answer are placed in a historical conversation list.

[0136] The relevant multimodal data, the current inference text sequence, the user question and the generated answer text are respectively represented by vectors, wherein the relevant multimodal data are mapped to the text vector space using an aligned neural network, specifically including combining a third-order transformation operator with a convolution operation to perform parallel extraction of high-order local features; the aligned neural network includes a third-order transformation convolution processing part, a sliding attention part and a convolution attention part.

[0137] According to an optional implementation, a third-order transformation operator is used to perform three transformations on the image data in the related multimodal data. Specifically, three function transformations are performed on each pixel to extract high-order features of the image, wherein the functions corresponding to the three transformations are all learned by the aligned neural network.

[0138] According to an optional implementation manner, in the sliding attention part, local features are collected to form a feature map by means of sliding windows and cross attention. Specifically, the following expression is used to make the size of the feature map input to the sliding attention part and the size of the feature map output by the sliding attention part unchanged. Specifically, the feature map X input to the sliding attention part is divided into blocks in the width and height dimensions to obtain m×n sub-tensors, and each sub-tensor and its transpose are matrix multiplied with the prior feature block to obtain M k1ij .

[0139] Select the number of channels of any sub-tensor and the corresponding prior feature block P k2 The number of channels is the same. When performing matrix multiplication, each channel performs matrix multiplication separately, and each channel feature map is added along the channel direction to obtain a matrix tensor, which is specifically expressed by the following expression:

[0140] ;

[0141] Among them, M k1ij is the i-th row and j-th column subtensor of the k1-th channel of the feature map of the input sliding attention part, X c1ij is the i-th row and j-th column subvector in the m×n subtensor of the c1-th channel obtained by block division, P k1c is the matrix tensor of the c1th channel of the k1th prior feature block, i, j, c are all positive integers, and C is a positive integer;

[0142] ;

[0143] Among them, M k2ij is the i-th row and j-th column subtensor of the k2-th channel, X c2ij is the i-th row and j-th column sub-vector in the m×n sub-tensor of the c2-th channel obtained by block division, P k2cis the matrix tensor of the c2th channel of the k2th prior feature block, i, j, c are all positive integers, and C is a positive integer.

[0144] According to an optional implementation manner, the reasoning link is constructed based on the obtained associated entities and proprietary knowledge graph.

[0145] From the proprietary knowledge graph, a first type of associated entity matching the problem to be processed is selected, and a subgraph is constructed using an entity selection algorithm with the first type of associated entity as the center to generate multiple reasoning links.

[0146] From the proprietary knowledge graph, associated entities that match the problem to be processed and whose entity distance is less than a first specific value are selected to form a first set.

[0147] From the proprietary knowledge graph, related entities that match the initial answer of the model and whose entity distance is less than a second specific value are further selected to form a second set, and the first specific value is greater than the second specific value.

[0148] Nodes intersecting with the first set in the first set and the second set are selected to form a subgraph, multiple triples are generated according to the subgraph to construct an inference link, and the constructed inference link is input into the multimodal large model to obtain a corresponding inference text sequence.

[0149] According to an optional implementation, based on the inference text sequence and the user question, text, picture, and audio data related to the user question are retrieved through a multimodal large model to obtain relevant multimodal data.

[0150] Specifically, the embedding model of the multimodal large model is called to embed the inference text sequence, the user question and the multimodal data into the same vector space, and each multimodal data storage block is respectively calculated with the embedding vector of the inference text sequence and the user question to obtain all relevant multimodal data.

[0151] According to an optional implementation, the self-attention mechanism in the transformer layer of the multimodal large model is replaced with a TTT-KAN model, specifically by replacing the hidden state in the traditional RNN with a Kolmogorov Arnold network model, and using the actual gradient descent of the input label to compress the obtained context information.

[0152] It should be noted that the multi-round dialogue processing method based on RAG and knowledge graph implemented in this example is roughly the same as the multi-round dialogue processing method based on RAG and knowledge graph in Example 1. Therefore, the description of the same parts is omitted.

[0153] Compared with the prior art, the present invention performs entity recognition on user questions and generated answer texts to obtain entities associated with user questions. According to the proprietary knowledge graph, the entities associated with the user questions are retrieved and identified to obtain associated entities; according to the obtained associated entities and the proprietary knowledge graph, an inference link is constructed, and the inference link is converted into an inference text sequence, and further related multimodal data is obtained. The related multimodal data, the current inference text sequence, the user question and the generated answer text are integrated as model input, and input into the multimodal large model to obtain a model answer corresponding to the user question. The current user's pending question is received, and the multimodal data related to the pending question is filtered out from the historical conversation list. When the multimodal data related to the pending question is not filtered out, the step of generating the model answer is repeated; when the multimodal data related to the pending question is filtered out, the filtered multimodal data is used as the context and together with the current user's pending question as the model input of the multimodal large model to obtain the answer text corresponding to the pending question. Specifically, the self-attention mechanism in the Transformer layer of the multimodal large model is replaced with TTT-KAN. The core principle of the TTT model is to replace the hidden state in the traditional RNN with a machine learning model, and use the actual gradient descent of the input label (i.e., token) to compress context information, which not only simplifies the model structure, but also achieves significant performance improvement. The present invention accurately parses user intentions while reducing interactions and effectively aligns user intentions with answer texts.

[0154] In addition, the DPE-MNER framework is adopted to improve the accuracy and efficiency of named entity recognition.

[0155] In addition, based on the proprietary knowledge graph, the entities associated with the user questions are retrieved and identified to obtain related entities, so as to construct reasoning links to form reasoning text sequences, and further retrieve relevant multimodal data, so that more accurate model answers can be obtained while diversifying the model input.

[0156] In addition, by combining the parallel feature extraction part of the third-order transformation operator and convolution, the ability to extract high-order information is improved through parallel processing, which can effectively capture higher-order local features in the image data of the user's questions and related multimodal data.

[0157] Figure 7 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

[0158] like Figure 7As shown, the electronic device is presented in the form of a general computing device. The processor may be one or more and work in coordination. The present invention does not exclude distributed processing, that is, the processor may be dispersed in different physical devices. The electronic device of the present invention is not limited to a single entity, but may also be the sum of multiple physical devices.

[0159] The memory stores a computer executable program, which is usually a machine-readable code. The computer-readable program can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least part of the steps in the method.

[0160] The memory includes a volatile memory, such as a random access memory unit (RAM) and / or a cache memory unit, and may also be a non-volatile memory, such as a read-only memory unit (ROM).

[0161] Optionally, in this embodiment, the electronic device further includes an I / O interface, which is used for the electronic device to exchange data with an external device. The I / O interface can represent one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.

[0162] It should be understood that Figure 7 The electronic device shown is only an example of the present invention, and the electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as display screens, and some electronic devices also include human-computer interaction elements such as buttons, keyboards, etc. As long as the electronic device can execute the computer-readable program in the memory to implement the method of the present invention or at least part of the steps of the method, it can be considered as an electronic device covered by the present invention.

[0163] Through the above description of the implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by combining software with necessary hardware. Figure 8 As shown, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of commands to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the embodiment of the present invention.

[0164] The software product may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0165] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, device, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0166] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0167] The computer-readable medium carries one or more programs (eg, computer-executable programs). When the one or more programs are executed by a device, the computer-readable medium implements the method of the present disclosure.

[0168] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.

[0169] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the exemplary embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several commands to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiment of the present invention.

[0170] The exemplary embodiments of the present invention are specifically shown and described above. It should be understood that the present invention is not limited to the detailed structure, configuration or implementation method described herein; on the contrary, the present invention is intended to cover various modifications and equivalent configurations included in the spirit and scope of the appended claims.

Claims

1. A multi-round dialogue processing method based on RAG and knowledge graph, characterized in that: include: Step S101: obtaining a user question of the current user, and based on the user question, the multimodal large model generates an answer text; Step S102: performing entity recognition on the user question and the generated answer text to obtain entities associated with the user question; Step S103: searching and identifying the entities associated with the user's question according to the proprietary knowledge graph to obtain associated entities, or returning re-entered response information to the client; Step S104: constructing a reasoning link based on the obtained associated entities and proprietary knowledge graph, and converting the reasoning link into a reasoning text sequence; Step S105: Based on the inference text sequence and the user question, retrieve text, picture, and audio data related to the user question through a multimodal large model to obtain relevant multimodal data; Step S106: Integrate the relevant multimodal data, the current inference text sequence, the user question and the generated answer text as model input, input the multimodal large model, obtain the model answer corresponding to the user question, and put the user question and its model answer into the historical conversation list; Step S107: receiving the pending question of the current user, filtering the multimodal data related to the pending question from the historical conversation list, and when the multimodal data related to the pending question is not filtered out, repeating steps S101 to S106; when the multimodal data related to the pending question is filtered out, entering step S108; Step S108: using the screened multimodal data as context and the current user's pending question as model input of the multimodal large model, repeatedly executing steps S101 to S103 to obtain an answer text corresponding to the pending question; wherein, Relevant multimodal data, current inference text sequence, user questions and generated answer text are respectively represented by vectors, wherein the relevant multimodal data are mapped to a text vector space using an aligned neural network, specifically including combining a third-order transformation operator with a convolution operation to perform parallel extraction of high-order local features; The alignment neural network includes a third-order transformation convolution processing part, a sliding attention part and a convolution attention part; In the sliding attention part, local features are collected to form a feature map by sliding window and cross attention. Specifically, the following expression is used to make the size of the feature map input to the sliding attention part and the size of the feature map output from the sliding attention part unchanged. Specifically, the feature map input to the sliding attention part is Divide the blocks in the width and height dimensions to get Sub-tensors, each sub-tensor and its transpose are matrix multiplied with the prior feature block to obtain ; Select the number of channels of any sub-tensor and the corresponding prior feature block The number of channels is the same. When performing matrix multiplication, each channel performs matrix multiplication separately, and each channel feature map is added along the channel direction to obtain a matrix tensor, which is specifically expressed by the following expression: ； in, is the i-th row and j-th column subtensor of the k1-th channel of the feature map of the input sliding attention part, The c1th channel obtained by block The i-th row and j-th column sub-vector in the sub-tensor, is the matrix tensor of the c1th channel of the k1th prior feature block, i, j, c are all positive integers, and C is a positive integer; ； in, is the i-th row and j-th column subtensor of the k2-th channel, The first 2 channels The i-th row and j-th column sub-vector in the sub-tensor, is the matrix tensor of the c2th channel of the k2th prior feature block, i, j, c are all positive integers, and C is a positive integer.

2. The multi-round dialogue processing method based on RAG and knowledge graph according to claim 1 is characterized in that: Further including: When the multimodal data related to the problem to be processed is not screened out, the problem to be processed is matched with the preset prompt template, and a matching preset prompt template is selected to guide the multimodal large model to generate an initial answer, and steps S101 to S106 are repeated, that is, multiple rounds of answers are repeatedly generated.

3. The multi-round dialogue processing method based on RAG and knowledge graph according to claim 1 is characterized in that: Further including: A third-order transformation operator is used to perform three transformations on the image data in the related multimodal data. Specifically, three function transformations are performed on each pixel to extract high-order features of the image, wherein the functions corresponding to the three transformations are all learned by the aligned neural network.

4. The multi-round dialogue processing method based on RAG and knowledge graph according to claim 1 is characterized in that: The construction of the reasoning link based on the obtained associated entities and proprietary knowledge graph includes: From the proprietary knowledge graph, a first type of associated entity matching the problem to be processed is selected, and a subgraph is constructed using an entity selection algorithm with the first type of associated entity as the center to generate multiple reasoning links.

5. The multi-round dialogue processing method based on RAG and knowledge graph according to claim 4 is characterized in that: Selecting, from the proprietary knowledge graph, related entities that match the problem to be processed and whose entity distance is less than a first specific value to form a first set; Further selecting, from the proprietary knowledge graph, related entities that match the initial answer of the model and whose entity distance is less than a second specific value to form a second set, wherein the first specific value is greater than the second specific value; Nodes intersecting with the first set in the first set and the second set are selected to form a subgraph, multiple triples are generated according to the subgraph to construct an inference link, and the constructed inference link is input into the multimodal large model to obtain a corresponding inference text sequence.

6. The multi-round dialogue processing method based on RAG and knowledge graph according to claim 1 is characterized in that: Based on the inference text sequence and the user question, the text, picture, and audio data related to the user question are retrieved through a multimodal large model to obtain relevant multimodal data, including: The embedding model of the multimodal large model is called to embed the inference text sequence, the user question and the multimodal data into the same vector space, and each multimodal data storage block is respectively calculated with the embedding vector of the inference text sequence and the user question to obtain all relevant multimodal data.

7. The multi-round dialogue processing method based on RAG and knowledge graph according to claim 6 is characterized in that: The self-attention mechanism in the transformer layer of the multimodal large model is replaced with a TTT-KAN model, specifically by replacing the hidden state in the traditional RNN with a Kolmogorov Arnold network model, and using the actual gradient descent of the input label to compress the obtained context information.

8. A multi-round dialogue processing system based on RAG and knowledge graph, characterized in that: The multi-round dialogue processing method based on RAG and knowledge graph according to claim 1 is adopted, and the multi-round dialogue processing system includes: A text acquisition module is used to acquire the user question of the current user, and based on the user question, the multimodal large model generates an answer text; An entity recognition module is used to perform entity recognition on user questions and generated answer texts to obtain entities associated with user questions; A search and identification module searches and identifies entities associated with the user's question based on the proprietary knowledge graph to obtain associated entities or return re-entered response information to the client; A construction module constructs a reasoning link based on the obtained associated entities and proprietary knowledge graph, and converts the reasoning link into a reasoning text sequence; A data processing module, based on the inference text sequence and the user question, retrieves text, picture, and audio data related to the user question through a multimodal large model to obtain relevant multimodal data; A first model processing module is used to integrate the relevant multimodal data, the current reasoning text sequence, the user question and the generated answer text as model input, input the multimodal large model, obtain the model answer corresponding to the user question, and put the user question and its model answer into the historical conversation list; A screening module, used for receiving the pending questions of the current user, screening the multimodal data related to the pending questions from the historical conversation list, and returning to the step executed by the text acquisition module when the multimodal data related to the pending questions are not screened out; and proceeding to the next step when the multimodal data related to the pending questions are screened out; The second model processing module uses the screened multimodal data as context and the current user's pending question as the model input of the multimodal large model, and returns to the steps performed by the text acquisition module to obtain the answer text corresponding to the pending question; wherein, Relevant multimodal data, current inference text sequence, user questions and generated answer text are respectively represented by vectors, wherein the relevant multimodal data are mapped to a text vector space using an aligned neural network, specifically including combining a third-order transformation operator with a convolution operation to perform parallel extraction of high-order local features; The alignment neural network includes a third-order transformation convolution processing part, a sliding attention part and a convolution attention part; In the sliding attention part, local features are collected to form a feature map by sliding window and cross attention. Specifically, the following expression is used to make the size of the feature map input to the sliding attention part and the size of the feature map output from the sliding attention part unchanged. Specifically, the feature map input to the sliding attention part is Divide the blocks in the width and height dimensions to get Sub-tensors, each sub-tensor and its transpose are matrix multiplied with the prior feature block to obtain ; Select the number of channels of any sub-tensor and the corresponding prior feature block The number of channels is the same. When performing matrix multiplication, each channel performs matrix multiplication separately, and each channel feature map is added along the channel direction to obtain a matrix tensor, which is specifically expressed by the following expression: ； in, is the i-th row and j-th column subtensor of the k1-th channel of the feature map of the input sliding attention part, The c1th channel obtained by block The i-th row and j-th column sub-vector in the sub-tensor, is the matrix tensor of the c1th channel of the k1th prior feature block, i, j, c are all positive integers, and C is a positive integer; ； in, is the i-th row and j-th column subtensor of the k2-th channel, The first 2 channels The i-th row and j-th column sub-vector in the sub-tensor, is the matrix tensor of the c2th channel of the k2th prior feature block, i, j, c are all positive integers, and C is a positive integer.

Images