Knowledge question-answering system and data processing method

Abstract

The embodiment of the invention provides a knowledge question-answering system and a data processing method. The knowledge question-answering system comprises a multi-modal knowledge graph, a question analysis module, a knowledge query module and an answer generation module, the question analysis module can perform semantic analysis on a target question currently asked by a user based on a large language model, extract target entity information and target user intention in the target question, and generate a knowledge query result; the knowledge query module can construct target query logic according to the target entity information and the target user intention and query the multi-modal knowledge graph according to the target query logic to obtain a target prompt information list, and the answer generation module determines a target answer according to all attribute information in the target prompt information list to generate prompt words and sends the prompt words to the user. And based on the large language model, generating a target answer according to the target answer generation cue word and returning the target answer to the user. According to the knowledge question-answering system, the accuracy and timeliness of the system can be improved, and the generalization ability of the system can also be improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of artificial intelligence technology, and in particular, to a knowledge question answering system and a data processing method. Background Art

[0002] Question answering systems are rooted in natural language processing technology in the field of artificial intelligence. They combine advanced machine learning and deep learning algorithms to enable computers to understand and answer questions posed in human language. With the development of technology, question answering systems have evolved from simple rule-based systems to complex systems based on large language models. These systems are able to process and understand large amounts of text data, capture language nuances, and perform well on a variety of language tasks. Existing question answering systems include retrieval-based question answering systems, knowledge graph-based question answering systems, and deep learning-based question answering systems. Among them, retrieval-based question answering systems index documents and retrieve relevant information through search engines, knowledge graph-based question answering systems use graph databases to store entity relationships and provide structured knowledge bases, and deep learning-based question answering systems use pre-trained models such as BERT and GPT to understand the context of questions and generate answers.

[0003] In addition, multimodal question-answering systems integrate multiple data types such as text, images, and videos. Through modal encoders and input mappers, combined with large language models to fuse and process multimodal information, they can provide richer and more accurate answers. However, the existing multimodal question-answering systems based on large models have low timeliness. Summary of the invention

[0004] The embodiments described in this article provide a knowledge question answering system and a data processing method, which can improve the timeliness and accuracy of the system and also improve the generalization ability.

[0005] In a first aspect, the present disclosure provides a knowledge question and answer system, including a multimodal knowledge graph, a question analysis module, a knowledge query module and an answer generation module.

[0006] The multimodal knowledge graph includes multiple first nodes and multiple second nodes, each of the first nodes is connected to at least one of the second nodes, each of the first nodes corresponds to an entity type in a target domain, and each of the second nodes corresponds to an attribute type in the target domain.

[0007] The question analysis module is used to perform semantic analysis on the target question currently asked by the user based on the large language model, and extract the target entity information and target user intention in the target question. The knowledge query module is used to construct a target query logic based on the target entity information and the target user intention, query the multimodal knowledge graph based on the target query logic, and obtain a target prompt information list, wherein the target prompt information list includes at least one attribute information. The answer generation module is used to determine the target answer generation prompt word based on all the attribute information in the target prompt information list, and based on the large language model, generate the target answer based on the target answer generation prompt word and return it to the user.

[0008] In some embodiments of the present disclosure, the question analysis module includes a prompt word construction unit and the large language model.

[0009] The prompt word construction unit is used to construct target entity recognition prompt words and target intent recognition prompt words according to the target question, entity recognition prompt word template and intent recognition prompt word template, and input the target entity recognition prompt words and the target intent recognition prompt words into the large language model. The large language model is used to output the target entity information according to the target entity recognition prompt words, and output the target user intent according to the target intent recognition prompt words.

[0010] In some embodiments of the present disclosure, the knowledge query module is further used to determine the target user's potential intention based on the keywords in the target question, and to construct the target query logic based on the target entity information, the target user's potential intention and the target user's intention.

[0011] In some embodiments of the present disclosure, the knowledge question and answer system also includes a model training module, which is used to use the data of the target field as training samples of the Llama big model, pre-train the Llama big model for masked word prediction tasks and sentence coherence prediction tasks to obtain the pre-trained Llama big model, and fine-tune the pre-trained Llama big model based on LoRA to obtain the large language model.

[0012] In some embodiments of the present disclosure, the knowledge question and answer system also includes a graph construction module, which is used to extract multimodal information from the collected multimodal data of the target field, and associate the multimodal information based on multimodal fusion to obtain the multimodal knowledge graph.

[0013] In a second aspect, the present disclosure provides a data processing method, in which any knowledge question-answering system provided in the first aspect is applied, the data processing method includes:

[0014] Based on the large language model, semantic analysis is performed on the target question currently asked by the user to extract the target entity information and target user intention in the target question;

[0015] Construct a target query logic according to the target entity information and the target user intention; query a multimodal knowledge graph according to the target query logic to obtain a target prompt information list; determine a target answer generation prompt word according to all the attribute information in the target prompt information list; based on the large language model, generate the target answer according to the target answer generation prompt word and return it to the user.

[0016] Among them, the target prompt information list includes at least one attribute information, the multimodal knowledge graph includes multiple first nodes and multiple second nodes, each of the first nodes is connected to at least one of the second nodes, each of the first nodes corresponds to an entity type of the target field, and each of the second nodes corresponds to an attribute type of the target field.

[0017] In some embodiments of the present disclosure, performing semantic analysis on the target question currently asked by the user and extracting target entity information and target user intention in the target question includes:

[0018] Construct target entity recognition prompt words and target intent recognition prompt words according to the target question, entity recognition prompt word template and intent recognition prompt word template; output the target entity information according to the target entity recognition prompt words; output the target user intent according to the target intent recognition prompt words.

[0019] In some embodiments of the present disclosure, constructing a target query logic according to the target entity information and the target user intention includes:

[0020] Determine the target user's potential intention according to the keywords in the target question; and construct the target query logic according to the target entity information, the target user's potential intention and the target user's intention.

[0021] In some embodiments of the present disclosure, before performing semantic analysis on the target question currently asked by the user based on the large language model and extracting the target entity information and the target user intention in the target question, the method further includes:

[0022] The data of the target domain is used as training samples of the Llama large model, and the Llama large model is pre-trained for masked word prediction tasks and sentence coherence prediction tasks to obtain the pre-trained Llama large model; the pre-trained Llama large model is fine-tuned based on LoRA to obtain the large language model.

[0023] In some embodiments of the present disclosure, before querying the multimodal knowledge graph according to the target query logic to obtain a target prompt information list, the method further includes:

[0024] Extracting multimodal information from the collected multimodal data of the target field; associating the multimodal information based on multimodal fusion to obtain the multimodal knowledge graph.

[0025] In the technical solution of the disclosed embodiment, the knowledge question answering system includes a multimodal knowledge graph, a question analysis module, a knowledge query module and an answer generation module. The question analysis module can perform semantic analysis on the target question currently asked by the user based on the large language model, and extract the target entity information and the target user intention in the target question. The knowledge query module can construct a target query logic according to the target entity information and the target user intention, and query the multimodal knowledge graph according to the target query logic to obtain a target prompt information list. The answer generation module determines the target answer generation prompt word according to all attribute information in the target prompt information list, generates the target answer according to the target answer generation prompt word based on the large language model and returns it to the user. The entity information and user intention in the question can be used as the output constraints of the large language model to obtain a more accurate answer, which can improve the accuracy of the system. In addition, the multimodal knowledge graph is used to store the knowledge of the target field, and the large language model answers the question based on the knowledge stored in the multimodal knowledge graph. The knowledge storage and fluent answer can be split, and the multimodal knowledge graph is updated in real time, thereby improving the timeliness of the system, and the hardware resource overhead when the multimodal knowledge graph is updated is small, thereby improving the generalization ability of the system.

[0026] The above description is only an overview of the technical solution of the embodiment of the present application. In order to more clearly understand the technical means of the embodiment of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 A schematic diagram of the structure of a knowledge question-answering system provided in an embodiment of the present disclosure.

[0029] Figure 2 A schematic diagram of the structure of another knowledge question-answering system provided in an embodiment of the present disclosure.

[0030] Figure 3 A flowchart of a data processing method provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work also fall within the scope of protection of the present disclosure.

[0032] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by a person skilled in the art to which the subject matter of the present disclosure belongs. It will be further understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the specification and the relevant art, and will not be interpreted in an idealized or overly formal form unless otherwise explicitly defined herein. As used herein, a statement that two or more parts are "connected" or "coupled" together shall mean that the parts are joined together directly or through one or more intermediate components.

[0033] Reference to "embodiments" in this disclosure means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase "embodiments" in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this disclosure may be combined with other embodiments.

[0034] In addition, the terms "first", "second", etc. in the specification and claims of the present disclosure or the above-mentioned drawings are used to distinguish different objects rather than to describe a specific order, and may explicitly or implicitly include one or more of the features.

[0035] In the description of the present disclosure, unless otherwise clearly stipulated and limited, the terms "connected" and "connected" should be understood in a broad sense. For example, the "connection" or "connection" of a circuit structure can refer to not only a physical connection, but also an electrical connection or a signal connection. For example, it can be a direct connection, that is, a physical connection, or it can be an indirect connection through at least one intermediate element, as long as the circuit is connected, and it can also be the internal connection of two elements; signal connection can refer to signal connection through a circuit as well as through a media medium, such as radio waves.

[0036] The term "and / or" in this disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists, A and B exist at the same time, and B exists. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0037] In the description of the present disclosure, unless otherwise specified, "plurality" and "at least two" mean more than two (including two), and similarly, "multiple groups" and "at least two groups" mean more than two groups (including two).

[0038] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0039] Traditional knowledge question-answering systems based on large models can collect multimodal raw data and clean and format the multimodal raw data to facilitate model training. The multimodal raw data includes raw text, raw images, raw audio, and raw video. Encoders suitable for different modalities are used to convert the corresponding raw data into a format that can be processed by the model, and the features of different modalities are aligned to the text feature space through an input mapper. For example, an image encoder is used to convert the raw image into a format that can be processed by the model, a video encoder is used to convert the raw video into a format that can be processed by the model, and an audio encoder is used to convert the raw audio into a format that can be processed by the model.

[0040] It is also possible to process the aligned features, perform semantic understanding and generate answers based on a powerful pre-trained language model, fuse features from different modalities through technologies such as adapter modules or cross-attention mechanisms to provide rich contextual information, and fine-tune the model using task-specific data based on multimodal pre-training to improve the performance of the system in specific fields, build an efficient retrieval system, split documents or partial data into chunks and encode them into vectors, and store them in a vector database to support fast retrieval, ensure that the generated content meets security and ethical standards, avoid harmful or biased content, and continuously optimize system performance based on user feedback and new technological advances, and update the knowledge base to maintain the timeliness and accuracy of the system.

[0041] First, the performance of traditional large-model-based question-answering systems depends on the quality and diversity of training data. If the training data is biased or incomplete, the system will perform poorly when dealing with unseen problems or domains. At the same time, fine-tuning large models in new domains requires a lot of training data and hardware resources. Traditional large-model-based question-answering systems have problems with data dependence and insufficient generalization capabilities.

[0042] Secondly, due to the complexity and diversity of natural language, the question-answering system needs to have a deep understanding of the user's intention in asking questions and the meaning behind them when performing semantic understanding. However, this is often difficult to achieve when dealing with complex problems or problems with complex language expressions. Therefore, the question-answering system may not be able to fully understand the implicit meaning and context of the language, resulting in the generated answers being inaccurate or irrelevant.

[0043] Third, because traditional question-answering systems based on large models can only answer questions that were input during training or are related to them, the model will speak nonsense or simply return unanswerable questions as answers when answering questions that are not input as training knowledge. This problem can only be solved by re-inputting new data into the large model for fine-tuning, but this will increase the hardware resource overhead and reduce the timeliness of data.

[0044] In order to solve the above technical problems, the knowledge question answering system provided by the present disclosure includes a multimodal knowledge graph, a question analysis module, a knowledge query module and an answer generation module. The question analysis module can perform semantic analysis on the target question currently asked by the user based on the large language model, and extract the target entity information and the target user intention in the target question. The knowledge query module can construct a target query logic based on the target entity information and the target user intention, and query the multimodal knowledge graph according to the target query logic to obtain a target prompt information list. The answer generation module determines the target answer generation prompt word according to all attribute information in the target prompt information list, generates the target answer according to the target answer generation prompt word based on the large language model and returns it to the user. The entity information and user intention in the question can be used as the output constraints of the large language model to obtain a more accurate answer, which can improve the accuracy of the system. In addition, the multimodal knowledge graph is used to store the knowledge of the target field, and the large language model answers the question based on the knowledge stored in the multimodal knowledge graph. The knowledge storage and fluent answer can be split, and the multimodal knowledge graph is updated in real time, thereby improving the timeliness of the system, and the hardware resource overhead when the multimodal knowledge graph is updated is small, thereby improving the generalization ability of the system.

[0045] The technical solution provided by the present disclosure is described in detail below with reference to several specific embodiments.

[0046] Figure 1 A schematic diagram of the structure of a knowledge question answering system provided in an embodiment of the present disclosure is shown in FIG. Figure 1 As shown, the knowledge question answering system includes:

[0047] The multimodal knowledge graph 110 includes a plurality of first nodes and a plurality of second nodes, each first node is connected to at least one second node, each first node corresponds to an entity type in a target domain, and each second node corresponds to an attribute category in a target domain.

[0048] The question analysis module 120 is used to perform semantic analysis on the target question currently asked by the user based on the large language model, and extract the target entity information and target user intention in the target question. The knowledge query module 130 is used to construct the target query logic according to the target entity information and the target user intention, query the multimodal knowledge graph according to the target query logic, and obtain a target prompt information list, which includes at least one attribute information. The answer generation module 140 is used to determine the target answer generation prompt word based on all the attribute information in the target prompt information list, and generate the target answer based on the target answer generation prompt word based on the large language model and return it to the user.

[0049] Exemplarily, the target field is the network security field, and the entity types in the network security field include vulnerabilities, products, manufacturers, and vulnerability patches. The multiple first nodes include a vulnerability class first node, a product class first node, a manufacturer class first node, and a vulnerability patch class first node.

[0050] The first node of each vulnerability class, the first node of each product class, the first node of each vendor class, and the first node of each vulnerability patch class corresponds to an entity information. For example, the entity information corresponding to the first node of the vulnerability class may be the Apache InLong code injection vulnerability (CNVD-2024-35666), the entity information corresponding to the first node of the product class may be Apache InLong, the entity information corresponding to the first node of the vendor class may be the Apache Software Foundation, and the entity information corresponding to the first node of the vulnerability patch class may be the patch for the Apache InLong code injection vulnerability (CNVD-2024-35666).

[0051] The vulnerability class entity includes twelve attribute types, namely vulnerability name, CNVD-ID, CVE-ID, danger level, affected products, vulnerability description, vulnerability cause, vulnerability type, solution, related patches, vulnerability classification and disclosure date. The first node of the vulnerability class is connected to the second node of the vulnerability name class, the second node of the CNVD-ID class, the second node of the CVE-ID class, the second node of the danger level class, the second node of the affected products class, the second node of the vulnerability description class, the second node of the vulnerability cause class, the second node of the vulnerability type class, the second node of the solution class, the second node of the related patch class, the second node of the vulnerability classification class and the second node of the disclosure date class.

[0052] Each second node connected to the first node of the vulnerability class corresponds to an attribute information. For example, the attribute information corresponding to the second node of the vulnerability name class can be Apache InLong code injection vulnerability (CNVD-2024-35666), the attribute information corresponding to the second node of the CNVD-ID class can be CNVD-2024-35666, the attribute information corresponding to the second node of the CVE-ID class can be CVE-2024-36268, the attribute information corresponding to the second node of the danger level class can be high, the attribute information corresponding to the second node of the affected product class can be Apache InLong version 1.10.0 to version 1.12.0, and the attribute information corresponding to the second node B6 of the vulnerability description class can be Apache InLong is a one-stop massive data integration framework of the Apache Foundation of the United States, providing automated, secure, and reliable data transmission capabilities. Apache InLong versions 1.10.0 to 1.12.0 have a code injection vulnerability, which attackers can exploit to cause remote code execution.

[0053] The attribute information corresponding to the second node of the vulnerability cause class may be that Apache InLong versions 1.10.0 to 1.12.0 have a code injection vulnerability, and an attacker can exploit the vulnerability to cause remote code execution. The attribute information corresponding to the second node of the vulnerability type class may be a general vulnerability. The attribute information corresponding to the second node of the solution class may be that the manufacturer has released a vulnerability fix, please pay attention to updates in time https: / / lists.apache.org / thread / 1w1yp1bg5sjvn46dszkf00tz1vfs0frc. The attribute information corresponding to the second node of the related patch class may be the patch for the Apache InLong code injection vulnerability (CNVD-2024-35666). The attribute information corresponding to the second node of the vulnerability classification class may be an application class vulnerability. The attribute information corresponding to the second node of the disclosure date class may be 2024-08-19.

[0054] The product class entity includes five attribute types, namely product name, product model, product type, company and performance and parameters. The first node of the product class connects the second node of the product name class, the second node of the product model class, the second node of the product type class, the second node of the company class and the second node of the performance and parameters class.

[0055] Each second node connected to the first node of the product class corresponds to an attribute information. For example, the attribute information corresponding to the second node of the product name class can be Apache InLong, the attribute information corresponding to the second node of the product model class can be Apache InLong, the attribute information corresponding to the second node of the product type class can be the data integration framework, and the attribute information corresponding to the second node of the company class can be the Apache InLong Software Foundation.

[0056] The manufacturer class entity includes four attribute types, namely company name, establishment time, company profile and company detailed description. The first node of the manufacturer class connects the second node of the company name class, the second node of the establishment time class, the second node of the company profile class and the second node of the company detailed description class.

[0057] Each second node connected to the first node of the manufacturer class corresponds to an attribute information. For example, the attribute information corresponding to the second node of the company name class can be the Apache InLong Software Foundation, the attribute information corresponding to the second node of the establishment time class can be 1999, and the attribute information corresponding to the second node of the company profile class can be the Apache Software Foundation, which is a non-profit organization dedicated to providing support for the Apache group that operates an open source software project. This open source software project is the Apache project.

[0058] The vulnerability patch class entity includes four attribute types, namely patch name, vulnerability number, patch link and patch description. The first node of the vulnerability patch class connects the second node of the patch name class, the second node of the vulnerability number class, the second node of the patch link class and the second node of the patch description class.

[0059] Each second node connected to the first node of the vulnerability patch class corresponds to an attribute information. For example, the attribute information corresponding to the second node of the patch name class can be the patch of the Apache InLong code injection vulnerability (CNVD-2024-35666), the attribute information corresponding to the second node of the vulnerability number class can be CNVD-2024-35666, and the attribute information corresponding to the second node of the patch link can be https: / / lists.apache.org / thread / 1w1yp1bg5sjvn46dszkf00tz1vs0frc.

[0060] In other implementations, the target field may also be a field outside the network security field, and the present disclosure does not impose specific restrictions on this. If the target field is a field outside the network security field, it is necessary to adjust the entity type, attribute type, and the connection relationship between the first node and the second node in the multimodal knowledge graph. The embodiments of the present disclosure are all illustrative using the network security field as an example.

[0061] Exemplarily, the knowledge question answering system has a preset entity recognition prompt template, and the entity recognition prompt template is designed as follows:

[0062] Please identify the four entities (vulnerability, product, vendor, vulnerability patch) from the sentence in {} and return the identification result in JSON format: "{query}".

[0063] The knowledge question and answer system also has a preset intent recognition Prompt template. There are nine types of user intents, namely querying vulnerability information, querying vulnerability causes, querying vulnerability solutions, querying patch information, querying vulnerability-affected products, querying vulnerability product manufacturers, querying similar vulnerabilities, querying manufacturer information, and querying product vulnerabilities. The intent recognition Prompt template stipulates that the intent recognition result of the large language model is at least one of these nine user intents.

[0064] When designing an intent recognition prompt template, first clarify the model's question input and the task to be completed, then specify the category of the returned item, then specify the problem-solving steps and give multiple examples, and finally give the user's question and guide the large language model output. For example, the design of the intent recognition prompt template is:

[0065] Read the following prompts and answer the questions (questions are at the end of the input):

[0066] When you try to identify the query intent in a user's question, you need to carefully analyze the question and judge it one by one in the 9 predefined query categories. For each category, think about whether the user's question contains the intent corresponding to that category. If you judge that the user's question fits a specific category, add that category to the output list. This approach requires you to systematically consider and evaluate every possible query intent to ensure that no possible classification is missed.

[0067] **Query Category**

[0068] "Query vulnerability information", "Query vulnerability causes", "Query vulnerability solutions", "Query patch information", "Query vulnerability-affected products", "Query vulnerability product manufacturers", "Query similar vulnerabilities", "Query manufacturer information" and "Query product vulnerabilities".

[0069] When handling user questions, follow these steps:

[0070] - Read users' questions.

[0071] -Compare the query category list above and consider whether each category is relevant to the user's question.

[0072] -If the user question explicitly or implicitly includes a category of query intent, add a description of that category to the output list.

[0073] - Make sure the final output list contains all category descriptions related to the user's problem.

[0074] Here are some examples of implicit intent, each in an input and output format, with prompts to help you chain your thoughts:

[0075] **Example 1:**

[0076] Input: "What are the similar vulnerabilities to the Windows Remote Desktop Licensing Service Remote Code Execution Vulnerability?"

[0077] Output: ["Query vulnerability information", "Query similar vulnerabilities"]. ......

[0079] **Example 10:**

[0080] Input: "What are the known vulnerabilities of Apache InLong?"

[0081] Output: ["Query product vulnerabilities"].

[0082] Through the above examples, we hope that you can form a systematic thinking process to accurately identify all possible query intents in user questions. Please analyze the user's questions carefully, take into account their possible multiple meanings, and ensure that the output reflects all relevant query intents.

[0083] Now that you know how to solve the problem, please solve the following problem and output the result!

[0084] Question input: "{query}".

[0085] It should be noted that the intent recognition Prompt template in the above embodiment only gives examples 1 and 10 as examples, and omits examples 2 to 9. In actual applications, the query category in the intent recognition Prompt template can be modified according to actual needs.

[0086] The question analysis module 120 includes a prompt word construction unit and a large language model, wherein the prompt word construction unit can determine a target entity recognition Prompt according to an entity recognition Prompt template and a target question currently asked by a user, and input the target entity recognition Prompt into the large language model.

[0087] For example, the target question currently asked by the user is "What are the vulnerabilities similar to or related to the Windows Remote Desktop Licensing Service remote code execution vulnerability?". Replace the query in the entity recognition prompt template with the target question currently asked by the user to obtain the target entity recognition prompt. The target entity recognition prompt is:

[0088] Please identify the four entities (vulnerability, product, vendor, vulnerability patch) from the sentence in {}, and return the identification results in the form of JSON: "{What are the vulnerabilities similar to or related to the Windows Remote Desktop Licensing Service remote code execution vulnerability}".

[0089] The prompt word construction unit can also determine the target intent recognition Prompt according to the target question currently raised by the user and the intent recognition Prompt template, and input the target intent recognition Prompt into the large language model. For example, based on the target question currently raised by the user in the above embodiment, the query in the intent recognition Prompt template is replaced with the target question currently raised by the user to obtain the target intent recognition Prompt. Then, the target intent recognition Prompt is:

[0090] Read the following prompts and answer the questions (questions are at the end of the input):

[0091] When you try to identify the query intent in a user's question, you need to carefully analyze the question and judge it one by one in the 9 predefined query categories. For each category, think about whether the user's question contains the intent corresponding to that category. If you judge that the user's question fits a specific category, add that category to the output list. This approach requires you to systematically consider and evaluate every possible query intent to ensure that no possible classification is missed.

[0092] **Query Category**

[0093] "Query vulnerability information", "Query vulnerability causes", "Query vulnerability solutions", "Query patch information", "Query vulnerability-affected products", "Query vulnerability product manufacturers", "Query similar vulnerabilities", "Query manufacturer information" and "Query product vulnerabilities".

[0094] When handling user questions, follow these steps:

[0095] - Read users' questions.

[0096] -Compare the query category list above and consider whether each category is relevant to the user's question.

[0097] -If the user question explicitly or implicitly includes a category of query intent, add a description of that category to the output list.

[0098] - Make sure the final output list contains all category descriptions related to the user's problem.

[0099] Here are some examples of implicit intent, each in an input and output format, with prompts to help you chain your thoughts:

[0100] **Example 1:**

[0101] Input: "What are the similar vulnerabilities to the Windows Remote Desktop Licensing Service Remote Code Execution Vulnerability?"

[0102] Output: ["Query vulnerability information", "Query similar vulnerabilities"]. ......

[0104] **Example 10:**

[0105] Input: "What are the known vulnerabilities of Apache InLong?"

[0106] Output: ["Query product vulnerabilities"].

[0107] Through the above examples, we hope that you can form a systematic thinking process to accurately identify all possible query intents in user questions. Please analyze the user's questions carefully, take into account their possible multiple meanings, and ensure that the output reflects all relevant query intents.

[0108] Now that you know how to solve the problem, please solve the following problem and output the result!

[0109] Question input: "{What are the vulnerabilities similar or related to the Windows Remote Desktop Licensing Service remote code execution vulnerability}".

[0110] In this way, the prompt word construction unit can construct the target entity recognition Prompt and the target intent recognition Prompt according to the target question, the entity recognition Prompt template and the intent recognition Prompt template, and input the target entity recognition Prompt and the target intent recognition Prompt into the large language model.

[0111] Exemplarily, the large language model is obtained by pre-training a Llama model based on a masked word prediction task and a sentence coherence prediction task, and fine-tuning the pre-trained Llama model based on annotated network security field data.

[0112] The target entity recognition Prompt is used as the input of the large language model. The large language model can identify the target entity information in the target question according to the target entity recognition Prompt, and output the target entity information. For example, based on the target entity recognition Prompt in the above embodiment, the output of the large language model is: {"vulnerability":["Windows Remote Desktop Licensing Service Remote Code Execution Vulnerability"], "product":["Windows Remote Desktop Licensing Service"], "vulnerability patch":[], "manufacturer":[]}.

[0113] The target intent recognition Prompt is used as the input of the large language model. The large language model can identify the target user intent corresponding to the target question according to the target intent recognition Prompt, where the target user intent is a part of all user intents, and output the target user intent. For example, based on the target intent recognition Prompt in the above embodiment, the output of the large language model is: ["Query vulnerability information", "Query similar vulnerabilities"].

[0114] Exemplarily, the knowledge query module 130 includes a logic construction unit and a query unit, wherein the logic construction unit can construct different query logics for different types of user intentions.

[0115] For example, the query logic for querying vulnerability information is: MATCH(a:vulnerability{name:"vulnerability name"}) RETURN a. vulnerability description, that is, find the first node corresponding to the vulnerability name in the first node of the vulnerability class, and return the attribute information corresponding to the second node of the vulnerability description class connected to the first node.

[0116] The query logic for querying the cause of a vulnerability is: MATCH(a:vulnerability{name:"vulnerability name"}) RETURN a. Vulnerability cause, that is, find the first node corresponding to the vulnerability name in the first node of the vulnerability class, and return the attribute information corresponding to the second node of the vulnerability cause class connected to the first node.

[0117] The query logic for querying vulnerability solutions is: MATCH(a:vulnerability{name:"vulnerability name"})RETURN a.Solution, that is, find the first node corresponding to the vulnerability name in the first node of the vulnerability class, and return the attribute information corresponding to the second node of the solution class connected to the first node.

[0118] The query logic for querying patch information is: MATCH(a: vulnerability {name: "vulnerability name"})-[r: related patches]-(b: vulnerability patch) RETURN b, that is, find the first node corresponding to the vulnerability name in the first node of the vulnerability class, find the first node of the vulnerability patch class connected to the first node along the related patch connection edge, and return the attribute information corresponding to all the second nodes connected to the first node of the vulnerability patch class.

[0119] The query logic for querying the products affected by the vulnerability is: MATCH(a:vulnerability{name:"vulnerability name"})-[r:affected product]-(b:product)RETURN b, that is, find the first node corresponding to the vulnerability name in the first node of the vulnerability class, find the first node of the product class connected to the first node along the affected product connection edge, and return the attribute information corresponding to all the second nodes connected to the first node of the product class.

[0120] The query logic for querying the manufacturer of vulnerable products is: MATCH(a:vulnerability{name:"vulnerability name"})-[r1:affected product]-(b1), (b2)-[r2:company]-(c:manufacturer)WHERE b1=b2, RETURN c. The query logic for querying similar vulnerabilities is: MATCH(a:vulnerability{name:"vulnerability name"})-[r:affected product*1..2]-(c:vulnerability)RETURN c. The query logic for finding manufacturer information is: MATCH(a:manufacturer{name:"company name"})RETURN a. The query logic for querying product vulnerabilities is: MATCH(a:vulnerability)-[r:affected product]-(b:product{product name:"product name"})RETURN b.

[0121] In this way, the logic construction unit can construct the corresponding target query logic according to the target entity information and the type of the target user's intention, wherein the target entity information includes the vulnerability name, company name and / or product name in the target query logic.

[0122] The query unit queries the multimodal knowledge graph based on the target query logic, and can obtain attribute information corresponding to multiple second nodes connected to a first node. For example, based on the query logic of querying patch information, querying vulnerability-affected products, querying vulnerability product manufacturers, querying similar vulnerabilities, or querying manufacturer information product vulnerabilities, it can obtain attribute information corresponding to multiple second nodes connected to a first node.

[0123] And / or attribute information corresponding to a second node can be obtained. For example, based on the query logic of querying the cause of a vulnerability, querying the vulnerability information, or querying the vulnerability solution, attribute information corresponding to a second node can be obtained.

[0124] Exemplarily, the knowledge question and answer system has a preset answer prompt template, which stipulates that questions must be answered according to preset prompts and cannot be answered freely. It also stipulates that a large language model answers relevant questions in the field of network security. It stipulates that when there is a deviation in the field to which the identified question belongs, a first preset text is returned to inform the user of the answer restriction. It also stipulates that if the queried information is small, a second text is returned to inform that the current question cannot be answered based on the known information.

[0125] The answer generation module 140 includes a prompt word construction unit and a large language model. The prompt word construction unit generates a Prompt template based on all the queried attribute information and answers, determines the target answer generation Prompt, for example, replaces the query in the answer generation Prompt template with all the queried attribute information, and inputs the target answer generation Prompt into the large language model.

[0126] Prompts are generated based on the target answers. The large language model can answer the target question currently being asked according to preset prompts. When the target question does not belong to the field of network security, the first text is output to prompt the user that the current answer is limited. When the target answer corresponding to the target question is not found, the second text is output to prompt the user that the target question currently being asked cannot be answered based on known information.

[0127] In summary, the knowledge question-answering system provided by the present invention can first identify the target entity information and target user intention of the target question currently raised by the user based on the large language model, and then query the multimodal knowledge graph based on the target entity information and target user intention to obtain attribute information, and finally obtain the target answer to the target question based on the large language model and the attribute information. The entity information and user intention in the question can be used as output constraints of the large language model to obtain a more accurate answer, which can improve the accuracy of the system. In addition, the multimodal knowledge graph is used to store the knowledge of the target field, and the large language model answers questions based on the knowledge stored in the multimodal knowledge graph. The knowledge storage and fluent answers can be split, and the multimodal knowledge graph can be updated in real time, thereby improving the timeliness of the system, and the hardware resource overhead when the multimodal knowledge graph is updated is small, thereby improving the generalization ability of the system.

[0128] In some embodiments, the knowledge query module 130 is further used to determine the target user's potential intention based on the keywords in the target question, and to construct the target query logic based on the target entity information, the target user's potential intention and the target user's intention.

[0129] Exemplarily, while performing entity recognition and user intent recognition on the target question, the question analysis module 120 also extracts keywords from the target question. For example, the keywords may be patch, solution, manufacturer, cause, etc.

[0130] The knowledge query module 130 also includes an intention supplement unit, which can determine the target user's potential intention based on the keywords in the target question, wherein the target user's potential intention is a part of all users' potential intentions.

[0131] The logic construction unit can construct different query logics for different types of user intentions. Specifically, the corresponding target query logic is constructed according to the type of target entity information and target user intention.

[0132] The logic construction unit can also construct different query logics for different types of user potential intentions. Specifically, according to the target entity information and the target user potential intentions, the corresponding target query logic is constructed to supplement the query logic, which can improve the accuracy of the query logic and thus improve the accuracy of the system.

[0133] In some embodiments, Figure 2 A schematic diagram of the structure of another knowledge question answering system provided in an embodiment of the present disclosure is shown in FIG. Figure 2 As shown, the knowledge question and answer system also includes a graph construction module 150, which is used to extract multimodal information from the collected multimodal data of the target field, associate the multimodal information based on multimodal fusion, and obtain a multimodal knowledge graph.

[0134] Exemplarily, the graph construction module 150 includes a data acquisition unit, an information extraction unit, a knowledge fusion unit and a quality assessment unit, wherein the data acquisition unit can collect multimodal data in the field of network open source network security, and send the collected multimodal data to the information extraction unit, the information extraction unit can extract multimodal information from the multimodal data, and can establish multiple first nodes and multiple second nodes.

[0135] The knowledge fusion unit associates multimodal information based on multimodal fusion technology, and can establish a connecting edge between the first node and the second node to obtain a multimodal knowledge graph. The quality assessment unit can perform quality assessment on the multimodal knowledge graph.

[0136] In this way, the accuracy and timeliness of the multimodal knowledge graph can be guaranteed, thereby ensuring the accuracy and timeliness of the system.

[0137] In some embodiments, see Figure 2 The knowledge question answering system also includes a model training module 160, which is used to use the data of the target field as training samples of the Llama large model, pre-train the Llama large model for masked word prediction tasks and sentence coherence prediction tasks, and obtain a pre-trained Llama large model. Based on LoRA, the pre-trained Llama large model is fine-tuned to obtain a large language model.

[0138] Exemplarily, the model training module 160 includes a pre-training unit and a fine-tuning training unit, wherein the pre-training unit uses the network security field text as the pre-training sample of the Llama large model, randomly selects a certain proportion of tokens, replaces the selected tokens with MASK, and then predicts what word MASK actually is, so as to pre-train the Llama large model for the masked word prediction task.

[0139] During the pre-training process of the masked word prediction task, a certain proportion of words can be randomly replaced or kept unchanged to reduce the difference between the pre-training stage and the fine-tuning stage and improve the processing ability of the large language model for text modality.

[0140] The pre-training unit can select two consecutive sentences from the pre-training sample data as positive samples, swap the order of the two consecutive sentences as negative samples, and pre-train the Llama model for the sentence coherence prediction task. If sentence 2 is the next sentence of sentence 1, the output of the Llama model indicates that the two sentences are coherent. If sentence 2 is not the next sentence of sentence 1, the output of the Llama model indicates that the two sentences are incoherent.

[0141] In this way, the pre-training unit can pre-train the Llama large model for the masked word prediction task and the sentence coherence prediction task, and obtain the pre-trained Llama large model after the pre-training is completed.

[0142] The fine-tuning training unit uses LoRA to fine-tune the pre-trained Llama large model, and uses the re-parameterization method to merge the new parameters with the original parameters to obtain a large language model, so that the large language model can better complete entity recognition.

[0143] The disclosed embodiment also provides a data processing method, which is applied to the knowledge question and answer system provided in any of the above embodiments.

[0144] Figure 3 A flowchart of a data processing method provided by an embodiment of the present disclosure is shown in FIG. Figure 3 As shown, the specific steps of the data processing method include:

[0145] S101, based on the large language model, semantic analysis is performed on the target question currently asked by the user to extract the target entity information and target user intention in the target question.

[0146] Exemplarily, the large language model is obtained by pre-training a Llama model based on a masked word prediction task and a sentence coherence prediction task, and fine-tuning the pre-trained Llama model based on annotated network security field data.

[0147] A specific description of a possible implementation method when executing S101 is as follows:

[0148] Build the target entity recognition prompt and target intent recognition prompt according to the target question, entity recognition prompt template and intent recognition prompt template, output the target entity information according to the target entity recognition prompt, and output the target user intent according to the target intent recognition prompt.

[0149] Exemplarily, the target entity recognition Prompt is determined based on the entity recognition Prompt template and the target question currently asked by the user, and the target entity recognition Prompt is input into the large language model. For example, the query in the entity recognition Prompt template is replaced with the target question currently asked by the user to obtain the target entity recognition Prompt.

[0150] According to the target question currently raised by the user and the intent recognition prompt template, the target intent recognition prompt is determined, and the target intent recognition prompt is input into the large language model. For example, the query in the intent recognition prompt template is replaced with the target question currently asked by the user to obtain the target intent recognition prompt.

[0151] When the target entity recognition prompt is used as the input of the large language model, the large language model can identify the target entity information in the target question according to the target entity recognition prompt and output the target entity information. When the target intent recognition prompt is used as the input of the large language model, the large language model can identify the target user intent corresponding to the target question according to the target intent recognition prompt and output the target user intent.

[0152] S102, constructing target query logic according to target entity information and target user intention.

[0153] Exemplarily, different types of user intents correspond to different query logics, and the target user intent is a part of all user intents. The target query logic can be constructed according to the target entity information and the type of the target user intent, wherein the target entity information includes the vulnerability name, company name and / or product name in the target query logic.

[0154] A specific description of a possible implementation method when executing S102 is as follows:

[0155] Determine the target user's potential intention based on the keywords in the target question, and build the target query logic based on the target entity information, target user's potential intention and target user's intention.

[0156] Exemplarily, while entity recognition and user intent recognition are performed on the target question, keywords in the target question are also extracted. For example, the keywords may be patch, solution, manufacturer, reason, etc.

[0157] Different keywords correspond to different potential user intentions. The target user's potential intention can be determined based on the keywords in the target question, where the target user's potential intention is a part of all user's potential intentions.

[0158] Different types of user potential intentions correspond to different query logics. The target user potential intention is a part of all user potential intentions. On the basis of constructing the target query logic according to the target entity information and the type of target user intention, the target query logic can also be constructed according to the target entity information and the type of target user potential intention to supplement the query logic, which can improve the accuracy of the query logic and thus improve the accuracy of the question answering.

[0159] S103, query the multimodal knowledge graph according to the target query logic to obtain a target prompt information list.

[0160] Among them, the target prompt information list includes at least one attribute information, the multimodal knowledge graph includes multiple first nodes and multiple second nodes, each first node is connected to at least one second node, each first node corresponds to an entity type of a target field, and each second node corresponds to an attribute type of a target field.

[0161] Exemplarily, by querying a multimodal knowledge graph based on target query logic, attribute information corresponding to multiple second nodes connected to a first node can be obtained. For example, based on the query logic of querying patch information, querying vulnerability-affected products, querying vulnerability product manufacturers, querying similar vulnerabilities, or querying manufacturer information product vulnerabilities, attribute information corresponding to multiple second nodes connected to a first node can be obtained.

[0162] And / or attribute information corresponding to a second node can be obtained. For example, based on the query logic of querying the cause of a vulnerability, querying the vulnerability information, or querying the vulnerability solution, attribute information corresponding to a second node can be obtained.

[0163] S104, determining a target answer generation prompt word according to all attribute information in the target prompt information list.

[0164] Exemplarily, a Prompt template is generated based on all the queried attribute information and the answer, and a target answer is determined to generate a Prompt. For example, the query in the answer generation Prompt template is replaced with all the queried attribute information.

[0165] Among them, the answer generation prompt template stipulates that questions should be answered according to preset prompts and cannot be answered freely. It also stipulates that the large language model should answer relevant questions in the field of network security. It stipulates that when there is a deviation in the field to which the identified question belongs, the first preset text will be returned to inform the user of the answer restriction. It also stipulates that if the queried information is small, the second text will be returned to inform that the current question cannot be answered based on the known information.

[0166] S105, based on the large language model, generate a target answer according to the target answer prompt words and return it to the user.

[0167] Exemplarily, a prompt is generated based on a target answer. The large language model can answer the target question according to a preset prompt. When the target question does not belong to the field of network security, a first text is output to prompt the user that the current answer is limited. When the target answer corresponding to the target question is not found, a second text is output to prompt the user that the current target question cannot be answered based on known information.

[0168] In the disclosed embodiment, a semantic analysis is performed on the target question currently asked by the user based on a large language model, target entity information and target user intent in the target question are extracted, a target query logic is constructed according to the target entity information and target user intent, a multimodal knowledge graph is queried according to the target query logic, a target prompt information list is obtained, a target answer generation prompt word is determined according to all attribute information in the target prompt information list, a target answer is generated according to the target answer generation prompt word based on the large language model and returned to the user, the entity information and user intent in the question can be used as output constraints of the large language model to obtain a more accurate answer, which can improve the accuracy of the answer to the question, and in addition, a multimodal knowledge graph is used to store knowledge in the target field, and the large language model answers questions based on the knowledge stored in the multimodal knowledge graph, which can split knowledge storage and fluent answers, and update the multimodal knowledge graph in real time, thereby improving the timeliness of the answer to the question.

[0169] In some embodiments, before executing S101, the method further includes:

[0170] The data of the target domain is used as training samples for the Llama large model, and the Llama large model is pre-trained on the masked word prediction task and the sentence coherence prediction task to obtain the pre-trained Llama large model; the pre-trained Llama large model is fine-tuned based on LoRA to obtain the large language model.

[0171] For example, first, the network security field text is used as a pre-training sample of the Llama model, and the Llama model is pre-trained for the masked word prediction task. Specifically, a certain proportion of tokens are randomly selected, the selected tokens are replaced with MASK, and then it is predicted what word MASK actually is.

[0172] Among them, during the pre-training process of the masked word prediction task, a certain proportion of words can be randomly replaced or kept unchanged to reduce the difference between the pre-training stage and the fine-tuning stage and improve the processing ability of the large language model for text modality.

[0173] Then, two consecutive sentences are selected from the pre-training sample data as positive samples, and the order of the two consecutive sentences is swapped as negative samples. The Llama model is pre-trained for the sentence coherence prediction task. If sentence 2 is the next sentence of sentence 1, the output of the Llama model indicates that the two sentences are coherent. If sentence 2 is not the next sentence of sentence 1, the output of the Llama model indicates that the two sentences are incoherent.

[0174] Finally, LoRA is used to fine-tune the pre-trained Llama large model, and the new parameters are merged with the original parameters using the re-parameterization method to obtain a large language model, so that the large language model can better complete entity recognition.

[0175] In some embodiments, before executing S103, the method further includes:

[0176] Extract multimodal information from the collected multimodal data in the target field; associate the multimodal information based on multimodal fusion to obtain a multimodal knowledge graph.

[0177] Exemplarily, multimodal data in the field of network open source network security is collected, and multimodal information in the multimodal data is extracted. Multiple first nodes and multiple second nodes can be established. The multimodal information can be associated based on multimodal fusion technology, and a connecting edge between the first node and the second node can be established to obtain a multimodal knowledge graph, and the quality of the multimodal knowledge graph can be evaluated.

[0178] In this way, the accuracy and timeliness of the multimodal knowledge graph can be guaranteed, thereby ensuring the accuracy and timeliness of the answers to questions.

[0179] Unless the context clearly indicates otherwise, the singular form of the words used herein and in the appended claims includes the plural and vice versa. Thus, when referring to the singular, the plural form of the corresponding term is generally included. Similarly, the words "comprise" and "include" are to be interpreted as inclusive rather than exclusive. Likewise, the terms "include" and "or" should be interpreted as inclusive unless such interpretation is expressly prohibited herein. Where the term "example" is used herein, particularly when it is located after a group of terms, the "example" is merely exemplary and illustrative and should not be considered exclusive or comprehensive.

[0180] Further aspects and scopes of adaptability become apparent from the description provided herein. It should be understood that various aspects of the present application can be implemented individually or in combination with one or more other aspects. It should also be understood that the description and specific embodiments herein are intended for purposes of illustration only and are not intended to limit the scope of the present application.

[0181] Several embodiments of the present disclosure are described in detail above, but it is obvious that those skilled in the art can make various modifications and variations to the embodiments of the present disclosure without departing from the spirit and scope of the present disclosure. The protection scope of the present disclosure is defined by the attached claims.

Claims

1. A knowledge question answering system, characterized in that: include: A multimodal knowledge graph, comprising a plurality of first nodes and a plurality of second nodes, each of the first nodes being connected to at least one of the second nodes, each of the first nodes corresponding to an entity type of a target domain, and each of the second nodes corresponding to an attribute type of the target domain; A question analysis module is used to perform semantic analysis on the target question currently asked by the user based on a large language model, and extract the target entity information and target user intention in the target question; A knowledge query module, configured to construct a target query logic according to the target entity information and the target user intention, query the multimodal knowledge graph according to the target query logic, and obtain a target prompt information list, wherein the target prompt information list includes at least one attribute information; The answer generation module is used to determine a target answer generation prompt word according to all the attribute information in the target prompt information list, and based on the large language model, generate a target answer according to the target answer generation prompt word and return it to the user.

2. The knowledge question answering system according to claim 1, characterized in that: The problem analysis module includes: a prompt word construction unit, used to construct target entity recognition prompt words and target intent recognition prompt words according to the target question, the entity recognition prompt word template and the intent recognition prompt word template, and input the target entity recognition prompt words and the target intent recognition prompt words into the large language model; The large language model is used to output the target entity information according to the target entity recognition prompt word, and to output the target user intent according to the target intent recognition prompt word.

3. The knowledge question answering system according to claim 1, characterized in that: The knowledge query module is further used to determine the target user's potential intention based on the keywords in the target question, and to construct the target query logic based on the target entity information, the target user's potential intention and the target user's intention.

4. The knowledge question answering system according to any one of claims 1 to 3, characterized in that: The knowledge question answering system further includes: The model training module is used to use the data of the target domain as training samples of the Llama large model, pre-train the Llama large model for masked word prediction tasks and sentence coherence prediction tasks to obtain the pre-trained Llama large model, and fine-tune the pre-trained Llama large model based on LoRA to obtain the large language model.

5. The knowledge question answering system according to any one of claims 1 to 3, characterized in that: The knowledge question answering system further includes: The graph construction module is used to extract multimodal information from the collected multimodal data of the target field, associate the multimodal information based on multimodal fusion, and obtain the multimodal knowledge graph.

6. A data processing method, characterized in that: Applied to the knowledge question answering system according to any one of claims 1 to 5, the method comprises: Based on the large language model, semantic analysis is performed on the target question currently asked by the user to extract the target entity information and target user intention in the target question; Constructing target query logic according to the target entity information and the target user intention; According to the target query logic, a multimodal knowledge graph is queried to obtain a target prompt information list; wherein the target prompt information list includes at least one attribute information, the multimodal knowledge graph includes a plurality of first nodes and a plurality of second nodes, each of the first nodes is connected to at least one of the second nodes, each of the first nodes corresponds to an entity type of a target domain, and each of the second nodes corresponds to an attribute type of the target domain; Determine a target answer generation prompt word according to all the attribute information in the target prompt information list; Based on the large language model, the target answer is generated according to the target answer generation prompt word and returned to the user.

7. The method according to claim 6, characterized in that The semantic analysis of the target question currently asked by the user to extract the target entity information and the target user intention in the target question includes: Constructing target entity recognition prompt words and target intent recognition prompt words according to the target question, entity recognition prompt word template and intent recognition prompt word template; Outputting the target entity information according to the target entity recognition prompt word; The target user intention is output according to the target intention recognition prompt word.

8. The method according to claim 6, characterized in that The constructing of the target query logic according to the target entity information and the target user intention includes: Determine the target user's potential intention based on the keywords in the target question; The target query logic is constructed according to the target entity information, the target user potential intention and the target user intention.

9. The method according to claim 6, characterized in that Before performing semantic analysis on the target question currently asked by the user based on the large language model and extracting the target entity information and the target user intention in the target question, the method further includes: Using the data of the target domain as training samples of the Llama large model, pre-training the Llama large model on masked word prediction tasks and sentence coherence prediction tasks, and obtaining the pre-trained Llama large model; The pre-trained Llama large model is fine-tuned based on LoRA to obtain the large language model.

10. The method according to claim 6, characterized in that Before querying the multimodal knowledge graph according to the target query logic to obtain a target prompt information list, the method further includes: Extracting multimodal information from the collected multimodal data of the target domain; The multimodal information is associated based on multimodal fusion to obtain the multimodal knowledge graph.

Images