A method, device, equipment and medium for answering user questions

Abstract

The present invention discloses a method, device, equipment and medium for answering user questions, which relates to the field of artificial intelligence. The method includes: obtaining user input information; using a multi-level associated scenario analysis model to perform application scenario analysis on the input information and understand the semantic intent of the input information, and determine the actual application scenario of the input information; calling a core element template preset for the actual application scenario, and extracting the core element information in the input information according to the core element template; obtaining scenario data, and inputting referenced knowledge, prompt words and scenario data as input data into a trained large language model to obtain the inference result output by the large language model. The present invention solves the data correlation problems of inaccurate and incomplete knowledge referenced by the existing LLM, improves the correlation matching between questions and business scenario data, and thus improves the response effect of LLM to scenario-related questions in various industries.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method, device, equipment and medium for answering user questions. Background Art

[0002] With the development of artificial intelligence technology, various industries have an increasing demand for the construction of large language model (LLM) intelligent agent applications. How to combine existing data in the industry with current large model technology and change the interaction form of traditional applications so that it can both efficiently utilize proprietary knowledge in the industry and protect the asset security of industry data is an important issue that the industry needs to solve urgently.

[0003] Retrieval-Augmented Generation (RAG) is an artificial intelligence technology that combines information retrieval technology with a language generation model. RAG retrieves relevant information from an external knowledge base and inputs it as a prompt to LLM to enhance LLM's ability to handle knowledge-intensive tasks, such as question-answering, text summarization, and content generation, so that LLM can generate more accurate and richer text content. RAG technology enables LLM to use industry data in a correlated manner, solves some of LLM's application problems, and improves the accuracy of LLM's answers. However, there are still problems such as inaccurate and incomplete data matching, which will directly affect the reasoning effect of LLM.

[0004] Therefore, it is necessary to provide an LLM model that can solve the above problems. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a method, apparatus, device and medium for answering user questions, so as to solve the problem that the existing LLM reasoning results are not ideal.

[0006] According to a first aspect, an embodiment of the present invention provides a method for answering a user question, the method comprising:

[0007] Get user input information;

[0008] The input information is analyzed for application scenarios and semantic intent understanding is performed on the input information using a multi-level associated scenario analysis model to determine the actual application scenario of the input information; each level has at least one scenario analysis model, each category of business scenarios has an associated scenario analysis model, and between two adjacent levels of scenario analysis models, the business scenario output by the scenario analysis model of the latter level is a business sub-scenario of the business scenario output by the scenario analysis model of the previous level;

[0009] Retrieving a core element template preset for the actual application scenario, and extracting core element information from the input information according to the core element template;

[0010] Retrieving a preset application programming interface associated with the actual application scenario and / or querying a preset knowledge base to process the input information and obtain scenario data;

[0011] The reference knowledge obtained by semantically matching the input information with the preset knowledge base and the prompt words obtained by semantically retrieving the input information with the preset knowledge base are input into the trained large language model as input data to obtain the inference result output by the large language model.

[0012] In combination with the first aspect, in a first implementation of the first aspect, calling a core element template preset for the actual application scenario, and extracting core element information in the input information according to the core element template specifically includes:

[0013] Retrieving the preset core element template according to the actual application scenario;

[0014] Performing data extraction on the input information according to the core element template to extract the core element information in the input information;

[0015] The core element information is standardized; the standardization includes cleaning, conversion, mapping and verification.

[0016] In combination with the first implementation manner of the first aspect, in the second implementation manner of the first aspect, calling a preset application programming interface associated with the actual application scenario and / or querying a preset knowledge base to process the input information to obtain scenario data specifically includes:

[0017] Converting the standardized core element information into a preset format;

[0018] According to the key-value pairs of the core element information in the preset format, matching processing is performed on the core element information in the preset format and the core rule name of the prompt word;

[0019] According to the core element information after matching, a preset application programming interface associated with the actual application scenario is called and / or a preset knowledge base is queried and the input information is processed to obtain the scenario data.

[0020] In combination with the first aspect, in a third implementation of the first aspect, the large language model is trained based on sample reference knowledge obtained by semantic matching of sample information, sample prompt words obtained by semantic retrieval, and sample scene data of sample information.

[0021] In combination with the first aspect, in a fourth implementation of the first aspect, the use of a multi-level associated scenario analysis model to perform application scenario analysis on the input information and to understand the semantic intent of the input information to determine an actual application scenario of the input information specifically includes:

[0022] Inputting the user's input information into the first-level scenario analysis model to obtain the first-level business scenario output by the first-level scenario analysis model;

[0023] Determine the next-level scenario analysis model associated with the first-level business scenario, and input the user's input information into the next-level scenario analysis model to obtain the business sub-scenario output by the next-level scenario analysis model;

[0024] Determine the next level scenario analysis model associated with the business sub-scenario, and input the user's input information into the next level scenario analysis model to obtain the business sub-scenario output by the next level scenario analysis model, until the user's input information is input into the last level scenario analysis model to obtain the associated business scenario output by the last level scenario analysis model;

[0025] The user's input information is understood, the intention understanding result is obtained, and the actual application scenario of the input information is determined based on the intention understanding result and the associated business scenario.

[0026] In combination with the first aspect, in a fifth implementation of the first aspect, the use of a multi-level associated scenario analysis model to perform application scenario analysis on the input information and to understand the semantic intent of the input information to determine an actual application scenario of the input information specifically includes:

[0027] Inputting the user's input information into the first-level scenario analysis model to obtain the first-level business scenarios output by each of the first-level scenario analysis models;

[0028] Determine the next-level scenario analysis model associated with each first-level business scenario, and input the user's input information into the next-level scenario analysis model to obtain the business sub-scenarios output by the next-level scenario analysis model;

[0029] Determine the next-level scenario analysis model associated with each business sub-scenario, and input the user's input information into the next-level scenario analysis model to obtain the business sub-scenario outputted by the next-level scenario analysis model, until the user's input information is input into the last-level scenario analysis model to obtain the associated business scenarios outputted by the last-level scenario analysis model;

[0030] The user's input information is understood, and the intention understanding results are obtained. All related business scenarios are analyzed based on the intention understanding results to obtain the actual application scenarios of the input information.

[0031] In combination with the first aspect, in the first implementation of the first aspect, the intent understanding is obtained using a trained semantic intent understanding model, and the semantic intent understanding model is trained based on sample information, sample intent understanding results corresponding to the sample information, and sample referenced knowledge description information, and the sample intent understanding results are the labels of the sample information.

[0032] According to a second aspect, an embodiment of the present invention further provides a device for answering user questions, the device comprising:

[0033] Question acquisition module, used to obtain user input information;

[0034] A scenario analysis module is used to use a multi-level associated scenario analysis model to perform application scenario analysis on the input information and understand the semantic intent of the input information to determine the actual application scenario of the input information; each level has at least one scenario analysis model, each category of business scenarios has an associated scenario analysis model, and between two adjacent levels of scenario analysis models, the business scenario output by the scenario analysis model of the latter level is a business sub-scenario of the business scenario output by the scenario analysis model of the previous level;

[0035] An element extraction module is used to call a core element template preset for the actual application scenario, and extract core element information in the input information according to the core element template;

[0036] A data association module is used to call a preset application programming interface associated with an actual application scenario and / or query a preset knowledge base to process the input information and obtain scenario data;

[0037] The question reasoning module is used to use the preset knowledge base to perform semantic matching on the input information to obtain the reference knowledge and use the preset knowledge base to perform semantic retrieval on the input information to obtain the prompt words, and input the reference knowledge, prompt words and scene data as input data into the trained large language model to obtain the reasoning result output by the large language model.

[0038] According to a third aspect, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method for solving user problems as described above are implemented.

[0039] According to a fourth aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described methods for solving user problems.

[0040] According to a fifth aspect, an embodiment of the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method for answering user questions as described in any one of the above items.

[0041] The method, device, equipment and medium for answering user questions of the present invention reduce the occurrence of inaccurate and unstable classification by performing multi-level classification and recognition and understanding of semantic intent of user questions, and then accurately capture the needs of user questions by combining classification and scene element information extraction, improve the interactive experience between the model and the user, and use the preset core element template to extract the core element information in the input information, so as to obtain the problem business scenario data more comprehensively and accurately, effectively improve the trained LLM's answer effect on business problems that are strongly related to industry data, and enable various interfaces related to various traditional applications of the enterprise to the new LLM application, making full use of By introducing data that is strongly related to the business scenario of the problem and conducting reasoning analysis on the company's existing industry data, LLM's response will be more professional and more convincing with data quantitative analysis. Through the above-mentioned classification model to identify intent, understand semantic intent, extract core business elements in combination with business scenario core element templates, and use business scenario-related APIs to supplement the combination of data that is strongly related to the business scenario, LLM's ability to respond to questions based on high-quality data is enhanced, and the data correlation problems such as inaccurate and incomplete knowledge cited by existing LLM are solved, and the correlation matching between questions and business scenario data is improved, thereby improving LLM's response effect on scenario-related issues in various industries. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The features and advantages of the present invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present invention in any way. In the accompanying drawings:

[0043] Figure 1 A schematic diagram of the LLM reasoning process in the prior art is shown;

[0044] Figure 2 A schematic diagram showing a flow chart of a method for answering user questions provided by the present invention;

[0045] Figure 3 A schematic diagram showing the LLM reasoning process in the method for solving user problems provided by the present invention;

[0046] Figure 4A schematic diagram showing the structure of a device for answering user questions provided by the present invention;

[0047] Figure 5 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0049] With the development of artificial intelligence technology, various industries have an increasing demand for the construction of LLM intelligent applications. How to combine the existing data in the industry with the current large model technology and change the interaction form of traditional applications so that it can not only efficiently utilize the proprietary knowledge in the industry field but also protect the asset security of industry data is an important issue that the industry needs to solve urgently.

[0050] RAG is an artificial intelligence technology that combines information retrieval technology with a language generation model. RAG retrieves relevant information from an external knowledge base and inputs it as a prompt to LLM to enhance LLM's ability to handle knowledge-intensive tasks such as question answering, text summarization, content generation, etc., so that LLM can generate more accurate and richer text content.

[0051] like Figure 1 As shown in the figure, in the LLM reasoning process, information related to the question is first retrieved from the pre-established external knowledge base. The purpose of this step is to provide effective context information for the subsequent generation of responses, combine the retrieval results with the question, and hand it over to LLM for answer generation. LLM will learn how to generate accurate and useful answers based on the retrieved information. The prompt words, reference knowledge data, and the reasoning ability of the model itself jointly affect the final output effect of LLM. From industry knowledge to knowledge vector library, a pre-trained text embedding model is usually used to convert queries and documents into vector representations so that similarity calculations can be performed in the vector space. Commonly used vector databases include FAISS, Milvus, etc. Before text embedding, it is usually necessary to segment long texts according to fixed lengths or delimiters or convert them into question-answer pairs before embedding. Then, based on the external knowledge base, question rewriting, result rearrangement, etc. are used to improve and optimize related sub-processes.

[0052] At present, the query effect of conventional text questions is quite significant. However, when the text matching the question semantics is segmented, or the matching answer is not in the text paragraph matching the question semantics, or the answer needs to be combined with multiple text blocks with low similarity for comprehensive response, it is difficult for the existing LLM to obtain high-quality knowledge data for effective response. Although the current methods such as question rewriting, paragraph segmentation of redundant text, and question-answer pair generation can improve the above situation to a certain extent, there are still business problems with strong data quality such as inaccurate and incomplete data matching, which will directly affect the reasoning effect of LLM.

[0053] Therefore, it is necessary to provide an LLM model that can solve the above problems.

[0054] In order to solve the above problems, a method for answering user questions is provided in this specification, which aims to improve the matching and acquisition of scene-related data of user input information, enhance the quality of LLM context reference data, and thus improve the final reasoning response effect of LLM. The method for answering user questions provided in this specification can be applied to electronic devices with data processing capabilities. The electronic device may include a notebook, a desktop computer, a smart phone, a smart wearable device (virtual reality glasses, smart watches, etc.), a tablet computer, etc. Of course, the method for answering user questions provided in this specification can also be applied to applications running in the above-mentioned electronic devices. For example, the method for answering user questions can be applied to a browser with data processing capabilities, or it can be applied to a browser with data processing capabilities. Figure 2 is a flow chart of a method for answering user questions according to an embodiment of the present invention. Figure 2 As shown, the method may include the following steps:

[0055] S10, obtaining user input information. In this embodiment, the input information provided by the user is a question that the LLM is expected to answer.

[0056] Preferably, at least one human-computer interaction method can be provided to the user to allow the user to provide corresponding input information, such as providing the user with an input text box to allow the user to input text-type input information, or multiple human-computer interaction methods such as voice input, gesture input, etc.

[0057] S20. Use a multi-level associated scenario analysis model to perform application scenario analysis on the input information and understand the semantic intent of the input information to determine the application scenario of the input information. In this embodiment, each level has at least one scenario analysis model, and each category of business scenario has an associated scenario analysis model. Moreover, between two adjacent levels of scenario analysis models, the business scenario output by the scenario analysis model of the latter level is a business sub-scenario of the business scenario output by the scenario analysis model of the previous level. For example, the scenario analysis model associated with the business scenario of category A is model A. After inputting the input information into model A, business scenario A1 output by model A is obtained. The scenario analysis model associated with the business scenario of category A1 is model A1. Then, the input information is input into model A1 again. Through such a multi-level associated scenario analysis model, the specific business scenario corresponding to the user's question can be finally obtained.

[0058] Among them, the above-mentioned scene analysis model is constructed by using a large model plus classification prompt words or using traditional machine learning such as neural networks and KMN. There is no restriction on the specific construction form of the scene analysis model.

[0059] By using a multi-level associated scenario analysis model to classify and identify business scenarios for user input information, it is possible to effectively avoid conflicts in sub-classifications in different applications or inaccurate identification due to multiple intent types, thus achieving a multi-level association from user questions to specific business scenarios. In this embodiment, the specific level hierarchy can be further split down according to the needs of segmentation.

[0060] The key to obtaining data related to user problem scenarios is to accurately identify the intent category of the user's problem, and then obtain the corresponding scenario data based on the intent-identified category. When the number of business scenario categories reaches a certain amount, the classification effect will decrease, and classification errors are more likely to occur. In this embodiment, the process of identifying the actual application scenario of the input information is also combined with the semantic intent understanding of the input information. The semantic intent understanding process of the input information is direct intent understanding, and the scenario analysis process of the input information is indirect intent understanding. The use of multi-level classification can better expand and improve the accuracy of classification. The actual application scenario obtained by combining the results of these two intent understandings can better reflect the actual intent of the user's problem, and further improve the accuracy of identifying the problem intent.

[0061] Accordingly, step S20 specifically includes the following steps:

[0062] S21. Input the user's input information into the first-level scenario analysis model to obtain the first-level business scenario output by the first-level scenario analysis model.

[0063] S22. Determine a next-level scenario analysis model associated with the first-level business scenario, and input the user's input information into the business scenario analysis model of that level to obtain a business sub-scenario output by the next-level scenario analysis model.

[0064] S23. Determine the next-level scenario analysis model associated with the business sub-scenario, and input the user's input information into the business scenario analysis model of that level to obtain the business sub-scenario output by the next-level scenario analysis model, until the user's input information is input into the last-level scenario analysis model to obtain the associated business scenario output by the last-level scenario analysis model.

[0065] S24: Understand the intent of the user's input information to obtain an intent understanding result, and determine the actual application scenario of the input information based on the intent understanding result and the associated business scenario.

[0066] In this embodiment, a trained semantic intent understanding model is used to understand the semantic intent of the user's input information to obtain a corresponding intent understanding result.

[0067] Accordingly, the whole process of understanding the semantic intent of user input information includes:

[0068] The user's input information is input into the trained semantic intent understanding model to obtain the intent understanding result output by the semantic intent understanding model, wherein the semantic intent understanding model is trained based on sample information, sample intent understanding results corresponding to the sample information, and referenced knowledge description information. It can be understood that the semantic intent understanding model is able to understand the semantic intent of the input information and extract the user's true question intention. Among them, the sample intent understanding results corresponding to the sample information can be obtained after annotation by professionals in combination with knowledge in related fields, or can be obtained by annotation in combination with expert experience. The sample intent understanding results can be understood as the label information used in the training process of the semantic intent understanding model, and the feature information obtained after semantic retrieval based on referenced knowledge description information.

[0069] More specifically, the training process of the semantic intent understanding model is as follows:

[0070] Sample information is obtained, and a sample intent understanding result of the sample information is determined.

[0071] The sample information is matched with the external preset knowledge base to obtain the sample reference knowledge description information corresponding to the sample information. During the training process, the relevant background knowledge description features are found and matched based on the preset knowledge base to obtain the sample reference knowledge description information. At the same time, the sample reference knowledge description information can also be used to supplement and improve the data of the preset knowledge base.

[0072] The sample information and the sample reference knowledge description information are used as input data for training, and the sample intent understanding results are used as label data. A supervised training method is adopted to train a semantic intent understanding model that is used to output user questions, that is, the intent understanding results of the user's input information.

[0073] To give a specific example, if a two-level associated scenario analysis model is used, the first-level scenario analysis model is an application classification model, and the last-level scenario analysis model is a business sub-classification model. When constructing these scenario analysis models, a certain amount of sample data of commonly used questions is prepared for each business scenario, and the first-level classification and sub-classification features are marked. The user's input information is first input into the first-level scenario analysis model, and the first-level scenario analysis model is used to identify the first-level business scenario associated with the user's input information. After determining the first-level business scenario, the last-level scenario analysis model is called to identify the sub-classification business scenario under the category classification of the first-level business scenario, that is, the second-level business scenario. Since there are only two levels of scenario analysis models, the second-level business scenario is also the associated business scenario finally identified by the user's question. Among them, each level of business scenario has a scenario analysis model.

[0074] Taking into account the complexity of business scenarios and in order to better identify the intention of user questions, in this embodiment, when the business differences are large, they can be horizontally split within the same level of classification to subdivide the business scenarios, so as to support more business scenarios. That is, each level of business scenario has multiple scenario analysis models, so that each level has multiple scenario analysis models.

[0075] Accordingly, step S20 specifically includes the following steps:

[0076] S25. Input the user's input information into the first-level scenario analysis model to obtain the first-level business scenarios output by each of the first-level scenario analysis models.

[0077] S26. Determine the next-level scenario analysis models associated with each first-level business scenario, and input the user's input information into these business scenario analysis models to obtain business sub-scenarios output by the next-level scenario analysis models.

[0078] S27. Determine the next-level scenario analysis models associated with each business sub-scenario, and input the user's input information into these business scenario analysis models to obtain the business sub-scenarios output by the next-level scenario analysis models, until the user's input information is input into the last-level scenario analysis model to obtain the associated business scenarios output by the last-level scenario analysis model.

[0079] S28. Understand the intention of the user's input information to obtain the intention understanding result, and analyze all related business scenarios based on the intention understanding result to obtain the actual application scenario of the input information.

[0080] The process of understanding the intention of the user's input information is shown in step S24, which will not be elaborated in detail here.

[0081] Since there are multiple scenario analysis models at each level, after the input information is input into the scenario analysis models of the same level, each scenario analysis model will output a corresponding business scenario. For example, after the input information is input into the scenario analysis model of the first level, the scenario analysis model of the first level outputs its corresponding first-level business scenario, and several first-level business scenarios are obtained. Based on these first-level business scenarios, the input information is input into the scenario analysis model of the next level, and the scenario analysis model of the next level outputs its corresponding second-level business scenario. After such multi-level association processing, the input information is input into the scenario analysis model of the last level, and the scenario analysis model of the last level outputs its corresponding associated business scenario, and finally several associated business scenarios are obtained.

[0082] In particular, when the input information is input into a certain scenario analysis model and the output result of the scenario analysis model is uncategorized, etc., it means that the input information cannot be classified and identified by the scenario analysis model. Therefore, there is no need to input the input information into each scenario analysis model of the next level of the scenario analysis model. It can be understood that the output information of the last level scenario analysis model that uses the input information as input data is summarized to obtain the required associated business scenario.

[0083] After that, the related business scenarios are analyzed in combination with the intention understanding results to obtain classification and identification that is more in line with the actual business scenarios. At the same time, the intention understanding results can also be used to horizontally split the scenario analysis model to segment the business scenarios.

[0084] In this embodiment, for interactive end products guided by major categories and subcategories, identification information can be added to the major categories and subcategories of the products. In this way, the identification of the business scenario can be directly transmitted, and the actual application scenario of the input information can be directly obtained, thereby more accurately identifying the problem scenario. For example, a scenario identification prefix is ​​added to the user's problem, that is, the input information, and the back end can directly obtain the actual application scenario based on the prefix identification.

[0085] S30. After identifying the business scenario of the input information based on step S20, retrieve the core element template pre-configured for each actual application scenario, and extract the core element information in the input information according to the core element template.

[0086] Specifically, step S30 includes:

[0087] S31. Retrieve the preset core element template according to the actual application scenario.

[0088] S32, extracting data from the input information according to the core element template to extract core element information from the input information;

[0089] S33. Standardize the core element information; standardization includes cleaning, conversion, mapping and verification, so that standardized element information can be obtained.

[0090] After clearly identifying the intention of the user's actual application scenario, the core element template pre-configured for each actual application scenario based on business rules can more accurately obtain the entities that need to be extracted in the user's question. Further verification of the extraction results enhances the interaction with the user, and performs secondary confirmation or improvement of the input of element information. Through a series of conversion, mapping, error correction, standardization and other processing, more accurate matching conditions are provided for the precise acquisition of related business data.

[0091] By establishing element information for different business scenarios and standardizing, regularizing, mapping and converting scenario data, LLM can expand its support scope for different business capabilities, enable various interfaces related to the company's various traditional applications to new large model applications, and make full use of the company's existing industry data. By introducing data that is strongly related to the problem business scenario and conducting reasoning analysis, the model's response will be more professional and the quantitative analysis with data will be more convincing.

[0092] The core element information of each business scenario can be set based on the business attributes of this business scenario. If the necessary core element information items cannot be extracted, feedback can be given to the user to prompt for additional input to increase the interactive experience.

[0093] In this embodiment, the core element information required for the current business is formulated based on the business scenario. For example, the element information formulated based on the industrial economic scenario is: {industry name}, {province}, {city}, {district}, {time}, {amount}, {qualification}, {company name}, etc. At the same time, based on the formulated core element information combination to form different forms of question sample data, the core element information in the user question, i.e., the input information, is extracted based on the LLM plus prompt word method or the dedicated entity extraction model trained based on the sample data.

[0094] S40. After obtaining the core element information of the input information based on the core element template of step S30, further call a series of preset scenario-related application programming interfaces (Application Programming Interface, API) and / or directly query the preset knowledge base of the relevant backend to process the input information, and obtain scenario data directly or indirectly related to the user's question.

[0095] In this embodiment, the extracted core element information is standardized, such as converting the amount (10,000 yuan, 100 million yuan, US dollars, etc.) into a standard amount, formatting the time into a standard format, mapping the enterprise alias into a standard name, correcting obvious typos, and performing synonym expansion matching on core keywords. The standardized data items are then filled into the prefabricated API parameters needed to obtain data in the current scenario and / or directly querying the database of the relevant backend to obtain the required problem scenario data.

[0096] S50, using the preset knowledge base to perform semantic matching on the input information to obtain the reference knowledge and using the preset knowledge base to perform semantic retrieval on the input information to obtain the prompt words, input the reference knowledge, prompt words and scene data as input data into the trained LLM to obtain the inference result output by the LLM.

[0097] Specifically, step S50 includes:

[0098] S51. Convert the standardized core element information into a preset format, which may be JSON.

[0099] S52: According to the key-value pairs of the core element information in the preset format, matching processing is performed on the core element information in the preset format and the core rule name of the prompt word.

[0100] S53: According to the matched core element information, a preset application programming interface associated with the actual application scenario is retrieved and / or a preset knowledge base is queried and the input information is processed to obtain scenario data.

[0101] In order for LLM to better understand and use data, it is necessary to further optimize the core element information. The format of the obtained core element information should be converted into JSON format, and the key name in the key-value pair in JSON and the core rule name in the prompt word should be unified. The relevant scenario data should be displayed in the JSON standard structure, and the relevant key names in the data items should be uniformly matched with the core rule nouns in the prompt word. Even if the understanding of the general parameter amount, such as about 7B, is not particularly strong, the contextual data can be well understood, and the reasoning output based on the prompt word can be more accurate, allowing LLM to better understand the related knowledge and data and enhance the reasoning response effect of LLM.

[0102] Through such processing, it is possible to associate scene data directly related to user questions, set this part of data as the top priority reference data in LLM reasoning, and then combine it with the reference knowledge obtained by semantic matching using the preset knowledge base in the original process and the prompt words obtained by semantic retrieval as the second priority reference data of LLM. Based on its own reasoning ability and the two parts of reference data, LLM can provide more accurate and professional answers than simply using the data associated with the knowledge base.

[0103] It should be noted that, in this embodiment, the knowledge base used for matching to obtain reference knowledge description information, the knowledge base used for matching to obtain scene data, and the knowledge base used for semantic matching and semantic retrieval may be combined into a preset knowledge base.

[0104] In this embodiment, the trained LLM is trained based on the sample reference knowledge obtained by semantic matching of the sample information, the sample prompt words obtained by semantic retrieval, and the sample scene data of the sample information. The method of extracting the sample scene data from the sample information is shown in steps S10 to S40, which will not be elaborated here.

[0105] The method for answering user questions of the present invention reduces the occurrence of inaccurate and unstable classification by performing multi-level classification and semantic intent understanding on user questions. Then, the combination of classification and scene element information extraction is used to accurately capture the needs of user questions, improve the interactive experience between the model and the user, and use the preset core element template to extract the core element information in the input information, so as to obtain the problem business scenario data more comprehensively and accurately, effectively improve the trained LLM's answer effect on business problems that are strongly related to industry data, enable various interfaces related to various traditional applications of the enterprise to the new LLM application, make full use of the existing industry data of the enterprise, introduce data that is strongly related to the problem business scenario, and perform reasoning analysis on it. The LLM's reply will be more professional and more convincing with data quantitative analysis. Through the above-mentioned classification model recognition intent, semantic intent understanding, extraction of business core elements in combination with business scenario core element templates, and use of business scenario-related APIs to supplement business scenario-strongly related data, the ability of LLM to answer questions based on high-quality data is enhanced, the data correlation problem of inaccurate and incomplete knowledge referenced by the existing LLM is solved, and the correlation matching between questions and business scenario data is improved, thereby improving the LLM's response effect on scene-related problems in various industries.

[0106] The following is a description of a device for answering user questions provided by an embodiment of the present invention. The device for answering user questions described below and the method for answering user questions described above can be referenced to each other.

[0107] In order to solve the above problems, a device for answering user questions is provided in this specification, aiming to provide a database performance optimization solution that is efficient, low-cost, highly automated and has good scalability. Figure 4 is a schematic diagram of the structure of a device for answering user questions according to an embodiment of the present invention. Figure 4 As shown, the device may include:

[0108] The question acquisition module 10 is used to acquire the user's input information. In this embodiment, the input information provided by the user is the question that the LLM is expected to answer.

[0109] Preferably, at least one human-computer interaction method can be provided to the user to allow the user to provide corresponding input information, such as providing the user with an input text box to allow the user to input text-type input information, or multiple human-computer interaction methods such as voice input, gesture input, etc.

[0110] The scenario analysis module 20 is used to use a multi-level associated scenario analysis model to perform application scenario analysis on the input information and understand the semantic intent of the input information, and determine the application scenario of the input information. In this embodiment, each level has at least one scenario analysis model, and each category of business scenarios has an associated scenario analysis model. Moreover, between two adjacent levels of scenario analysis models, the business scenario output by the scenario analysis model of the latter level is a business sub-scenario of the business scenario output by the scenario analysis model of the previous level. For example, the scenario analysis model associated with the business scenario of category A is model A. After inputting the input information into model A, the business scenario A1 output by model A is obtained. The scenario analysis model associated with the business scenario of category A1 is model A1. After that, the input information is input into model A1 again. Through such a multi-level associated scenario analysis model, the specific business scenario corresponding to the user's question can be finally obtained.

[0111] Among them, the above-mentioned scene analysis model is constructed by using a large model plus classification prompt words or using traditional machine learning such as neural networks and KMN. There is no restriction on the specific construction form of the scene analysis model.

[0112] By using a multi-level associated scenario analysis model to classify and identify business scenarios for user input information, it is possible to effectively avoid conflicts in sub-classifications in different applications or inaccurate identification due to multiple intent types, thus achieving a multi-level association from user questions to specific business scenarios. In this embodiment, the specific level hierarchy can be further split down according to the needs of segmentation.

[0113] The key to obtaining data related to user problem scenarios is to accurately identify the intent category of the user's problem, and then obtain the corresponding scenario data based on the intent-identified category. When the number of business scenario categories reaches a certain amount, the classification effect will decrease, and classification errors are more likely to occur. In this embodiment, the process of identifying the actual application scenario of the input information is also combined with the semantic intent understanding of the input information. The semantic intent understanding process of the input information is direct intent understanding, and the scenario analysis process of the input information is indirect intent understanding. The use of multi-level classification can better expand and improve the accuracy of classification. The actual application scenario obtained by combining the results of these two intent understandings can better reflect the actual intent of the user's problem, and further improve the accuracy of identifying the problem intent.

[0114] In this embodiment, a trained semantic intent understanding model is used to understand the semantic intent of the user's input information to obtain a corresponding intent understanding result.

[0115] Accordingly, the whole process of understanding the semantic intent of user input information includes:

[0116] The user's input information is input into the trained semantic intent understanding model to obtain the intent understanding result output by the semantic intent understanding model, wherein the semantic intent understanding model is trained based on sample information, sample intent understanding results corresponding to the sample information, and referenced knowledge description information. It can be understood that the semantic intent understanding model is able to understand the semantic intent of the input information and extract the user's true question intention. Among them, the sample intent understanding results corresponding to the sample information can be obtained after annotation by professionals in combination with knowledge in related fields, or can be obtained by annotation in combination with expert experience. The sample intent understanding results can be understood as the label information used in the training process of the semantic intent understanding model, and the feature information obtained after semantic retrieval based on referenced knowledge description information.

[0117] More specifically, the training process of the semantic intent understanding model is as follows:

[0118] Sample information is obtained, and a sample intent understanding result of the sample information is determined.

[0119] The sample information is matched with the external preset knowledge base to obtain the sample reference knowledge description information corresponding to the sample information. During the training process, the relevant background knowledge description features are found and matched based on the preset knowledge base to obtain the sample reference knowledge description information. At the same time, the sample reference knowledge description information can also be used to supplement and improve the data of the preset knowledge base.

[0120] The sample information and the sample reference knowledge description information are used as input data for training, and the sample intent understanding results are used as label data. A supervised training method is adopted to train a semantic intent understanding model that is used to output user questions, that is, the intent understanding results of the user's input information.

[0121] To give a specific example, if a two-level associated scenario analysis model is used, the first-level scenario analysis model is an application classification model, and the last-level scenario analysis model is a business sub-classification model. When constructing these scenario analysis models, a certain amount of sample data of commonly used questions is prepared for each business scenario, and the first-level classification and sub-classification features are marked. The user's input information is first input into the first-level scenario analysis model, and the first-level scenario analysis model is used to identify the first-level business scenario associated with the user's input information. After determining the first-level business scenario, the last-level scenario analysis model is called to identify the sub-classification business scenario under the category classification of the first-level business scenario, that is, the second-level business scenario. Since there are only two levels of scenario analysis models, the second-level business scenario is also the associated business scenario finally identified by the user's question. Among them, each level of business scenario has a scenario analysis model.

[0122] Taking into account the complexity of business scenarios and in order to better identify the intention of user questions, in this embodiment, when the business differences are large, they can be horizontally split within the same level of classification to subdivide the business scenarios, so as to support more business scenarios. That is, each level of business scenario has multiple scenario analysis models, so that each level has multiple scenario analysis models.

[0123] Since there are multiple scenario analysis models at each level, after the input information is input into the scenario analysis models of the same level, each scenario analysis model will output a corresponding business scenario. For example, after the input information is input into the scenario analysis model of the first level, the scenario analysis model of the first level outputs its corresponding first-level business scenario, and several first-level business scenarios are obtained. Based on these first-level business scenarios, the input information is input into the scenario analysis model of the next level, and the scenario analysis model of the next level outputs its corresponding second-level business scenario. After such multi-level association processing, the input information is input into the scenario analysis model of the last level, and the scenario analysis model of the last level outputs its corresponding associated business scenario, and finally several associated business scenarios are obtained.

[0124] In particular, when the input information is input into a certain scenario analysis model and the output result of the scenario analysis model is uncategorized, etc., it means that the input information cannot be classified and identified by the scenario analysis model. Therefore, there is no need to input the input information into each scenario analysis model of the next level of the scenario analysis model. It can be understood that the output information of the last level scenario analysis model that uses the input information as input data is summarized to obtain the required associated business scenario.

[0125] After that, the related business scenarios are analyzed in combination with the intention understanding results to obtain classification and identification that is more in line with the actual business scenarios. At the same time, the intention understanding results can also be used to horizontally split the scenario analysis model to segment the business scenarios.

[0126] In this embodiment, for interactive end products guided by major categories and subcategories, identification information can be added to the major categories and subcategories of the products. In this way, the identification of the business scenario can be directly transmitted, and the actual application scenario of the input information can be directly obtained, thereby more accurately identifying the problem scenario. For example, a scenario identification prefix is ​​added to the user's problem, that is, the input information, and the back end can directly obtain the actual application scenario based on the prefix identification.

[0127] The element extraction module 30 is used to identify the business scenario of the input information, call the core element template pre-configured for each actual application scenario, and extract the core element information in the input information according to the core element template.

[0128] After clearly identifying the intention of the user's actual application scenario, the core element template pre-configured for each actual application scenario based on business rules can more accurately obtain the entities that need to be extracted in the user's question. Further verification of the extraction results enhances the interaction with the user, and performs secondary confirmation or improvement of the input of element information. Through a series of conversion, mapping, error correction, standardization and other processing, more accurate matching conditions are provided for the precise acquisition of related business data.

[0129] The core element information of each business scenario can be set based on the business attributes of this business scenario. If the necessary core element information items cannot be extracted, feedback can be given to the user to prompt for additional input to increase the interactive experience.

[0130] In this embodiment, the core element information required for the current business is formulated based on the business scenario. For example, the element information formulated based on the industrial economic scenario is: {industry name}, {province}, {city}, {district}, {time}, {amount}, {qualification}, {company name}, etc. At the same time, based on the formulated core element information combination to form different forms of question sample data, the core element information in the user question, i.e., the input information, is extracted based on the LLM plus prompt word method or the dedicated entity extraction model trained based on the sample data.

[0131] The data association module 40 is used to obtain the core element information of the input information based on the core element template, and then further call a series of preset scenario-related APIs and / or directly query the preset knowledge base of the relevant backend to process the input information, so as to obtain scenario data directly or indirectly related to the user's question.

[0132] Specifically, in order for LLM to better understand and use data, it is necessary to further optimize the core element information, convert the obtained core element information into JSON format, unify the key name in the key-value pair in JSON and the core rule name in the prompt word, display the relevant scenario data in the JSON standard structure, and unify the relevant key names in the data items with the core rule nouns in the prompt word. Even if the understanding of the general parameter amount, such as about 7B, is not particularly strong, the contextual data can be well understood, and the reasoning output based on the prompt word can be more accurate, allowing LLM to better understand the associated knowledge and data and enhance the reasoning response effect of LLM.

[0133] In this embodiment, the extracted core element information is standardized, such as converting the amount (10,000 yuan, 100 million yuan, US dollars, etc.) into a standard amount, formatting the time into a standard format, mapping the enterprise alias into a standard name, correcting obvious typos, and performing synonym expansion matching on core keywords. The standardized data items are then filled into the prefabricated API parameters needed to obtain data in the current scenario and / or directly querying the database of the relevant backend to obtain the required problem scenario data.

[0134] The question reasoning module 50 is used to obtain reference knowledge by semantically matching the input information with the preset knowledge base and obtain prompt words by semantically searching the input information with the preset knowledge base, and input the reference knowledge, prompt words and scene data as input data into the trained LLM to obtain the reasoning result output by the LLM.

[0135] Through such processing, it is possible to associate scene data directly related to user questions, set this part of data as the top priority reference data in LLM reasoning, and then combine it with the reference knowledge obtained by semantic matching using the preset knowledge base in the original process and the prompt words obtained by semantic retrieval as the second priority reference data of LLM. Based on its own reasoning ability and the two parts of reference data, LLM can provide more accurate and professional answers than simply using the data associated with the knowledge base.

[0136] It should be noted that, in this embodiment, the knowledge base used for matching to obtain reference knowledge description information, the knowledge base used for matching to obtain scene data, and the knowledge base used for semantic matching and semantic retrieval may be combined into a preset knowledge base.

[0137] In this embodiment, the trained LLM is trained based on the sample reference knowledge obtained by semantic matching of the sample information, the sample prompt words obtained by semantic retrieval, and the sample scene data of the sample information. The method of extracting the sample scene data from the sample information is as described above, and will not be elaborated on here.

[0138] The user question answering device of the present invention reduces the occurrence of inaccurate and non-fixed classification by performing multi-level classification and semantic intent understanding on user questions. Then, the classification and scene element information extraction are combined to accurately capture the needs of user questions, improve the interactive experience between the model and the user, and use the preset core element template to extract the core element information in the input information, so as to obtain the problem business scenario data more comprehensively and accurately, effectively improve the trained LLM to answer business questions with strong correlation to industry data, enable various interfaces related to various traditional applications of the enterprise to the new LLM application, make full use of the existing industry data of the enterprise, introduce data with strong correlation to the problem business scenario, and perform reasoning analysis on it. The LLM's reply will be more professional and more convincing with data quantitative analysis. Through the above-mentioned classification model recognition intent, semantic intent understanding, extraction of business core elements in combination with business scenario core element templates, and use of business scenario-related APIs to supplement business scenario-strongly related data, the ability of LLM to answer questions based on high-quality data is enhanced, the data correlation problems such as inaccurate and incomplete knowledge referenced by the existing LLM are solved, and the correlation matching between questions and business scenario data is improved, thereby improving the LLM's reply effect on scene-related questions in various industries.

[0139] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510 (processor), a communication interface 520 (Communications Interface), a memory 530 (memory) and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic command in the memory 530 to execute the method for answering the user's question, and the method includes:

[0140] Get user input information;

[0141] The input information is analyzed for application scenarios and semantic intent understanding is performed on the input information using a multi-level associated scenario analysis model to determine the actual application scenario of the input information; each level has at least one scenario analysis model, each category of business scenarios has an associated scenario analysis model, and between two adjacent levels of scenario analysis models, the business scenario output by the scenario analysis model of the latter level is a business sub-scenario of the business scenario output by the scenario analysis model of the previous level;

[0142] Retrieving a core element template preset for the actual application scenario, and extracting core element information from the input information according to the core element template;

[0143] Retrieving a preset application programming interface associated with the actual application scenario and / or querying a preset knowledge base to process the input information and obtain scenario data;

[0144] The reference knowledge obtained by semantically matching the input information with the preset knowledge base and the prompt words obtained by semantically retrieving the input information with the preset knowledge base are input into the trained large language model as input data to obtain the inference result output by the large language model.

[0145] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memor), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0146] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, when the program instructions are executed by a computer, the computer can execute the method for answering the user question provided by the above methods, the method comprising:

[0147] Get user input information;

[0148] The input information is analyzed for application scenarios and semantic intent understanding is performed on the input information using a multi-level associated scenario analysis model to determine the actual application scenario of the input information; each level has at least one scenario analysis model, each category of business scenarios has an associated scenario analysis model, and between two adjacent levels of scenario analysis models, the business scenario output by the scenario analysis model of the latter level is a business sub-scenario of the business scenario output by the scenario analysis model of the previous level;

[0149] Retrieving a core element template preset for the actual application scenario, and extracting core element information from the input information according to the core element template;

[0150] Retrieving a preset application programming interface associated with the actual application scenario and / or querying a preset knowledge base to process the input information and obtain scenario data;

[0151] The reference knowledge obtained by semantically matching the input information with the preset knowledge base and the prompt words obtained by semantically retrieving the input information with the preset knowledge base are input into the trained large language model as input data to obtain the inference result output by the large language model.

[0152] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the computer program is implemented to perform the above-mentioned methods for answering user questions, the methods comprising:

[0153] Get user input information;

[0154] The input information is analyzed for application scenarios and semantic intent understanding is performed on the input information using a multi-level associated scenario analysis model to determine the actual application scenario of the input information; each level has at least one scenario analysis model, each category of business scenarios has an associated scenario analysis model, and between two adjacent levels of scenario analysis models, the business scenario output by the scenario analysis model of the latter level is a business sub-scenario of the business scenario output by the scenario analysis model of the previous level;

[0155] Retrieving a core element template preset for the actual application scenario, and extracting core element information from the input information according to the core element template;

[0156] Retrieving a preset application programming interface associated with the actual application scenario and / or querying a preset knowledge base to process the input information and obtain scenario data;

[0157] The reference knowledge obtained by semantically matching the input information with the preset knowledge base and the prompt words obtained by semantically retrieving the input information with the preset knowledge base are input into the trained large language model as input data to obtain the inference result output by the large language model.

[0158] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0159] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0160] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for solving user problems, characterized in that: The method comprises: Get user input information; The input information is analyzed for application scenarios and semantic intent understanding is performed on the input information using a multi-level associated scenario analysis model to determine the actual application scenario of the input information; each level has at least one scenario analysis model, each category of business scenarios has an associated scenario analysis model, and between two adjacent levels of scenario analysis models, the business scenario output by the scenario analysis model of the latter level is a business sub-scenario of the business scenario output by the scenario analysis model of the previous level; Retrieving a core element template preset for the actual application scenario, and extracting core element information from the input information according to the core element template; Retrieving a preset application programming interface associated with the actual application scenario and / or querying a preset knowledge base to process the input information and obtain scenario data; The reference knowledge obtained by semantically matching the input information with the preset knowledge base and the prompt words obtained by semantically retrieving the input information with the preset knowledge base are input into the trained large language model as input data to obtain the inference result output by the large language model.

2. The method for answering user questions according to claim 1, characterized in that: The calling of the core element template preset for the actual application scenario and extracting the core element information in the input information according to the core element template specifically includes: Retrieving the preset core element template according to the actual application scenario; Performing data extraction on the input information according to the core element template to extract the core element information in the input information; The core element information is standardized; the standardization includes cleaning, conversion, mapping and verification.

3. The method for answering user questions according to claim 2, characterized in that: The calling of a preset application programming interface associated with the actual application scenario and / or querying a preset knowledge base to process the input information to obtain scenario data specifically includes: Converting the standardized core element information into a preset format; According to the key-value pairs of the core element information in the preset format, matching processing is performed on the core element information in the preset format and the core rule name of the prompt word; According to the core element information after matching, a preset application programming interface associated with the actual application scenario is called and / or a preset knowledge base is queried and the input information is processed to obtain the scenario data.

4. The method for answering user questions according to claim 1, characterized in that: The large language model is trained based on sample reference knowledge obtained by semantic matching of sample information, sample prompt words obtained by semantic retrieval, and sample scene data of sample information.

5. The method for answering user questions according to claim 1, characterized in that: The using of the multi-level associated scenario analysis model to perform application scenario analysis on the input information and to understand the semantic intent of the input information to determine the actual application scenario of the input information specifically includes: Inputting the user's input information into the first-level scenario analysis model to obtain the first-level business scenario output by the first-level scenario analysis model; Determine the next-level scenario analysis model associated with the first-level business scenario, and input the user's input information into the next-level scenario analysis model to obtain the business sub-scenario output by the next-level scenario analysis model; Determine the next level scenario analysis model associated with the business sub-scenario, and input the user's input information into the next level scenario analysis model to obtain the business sub-scenario output by the next level scenario analysis model, until the user's input information is input into the last level scenario analysis model to obtain the associated business scenario output by the last level scenario analysis model; The user's input information is understood, the intention understanding result is obtained, and the actual application scenario of the input information is determined based on the intention understanding result and the associated business scenario.

6. The method for answering user questions according to claim 1, characterized in that: The using of the multi-level associated scenario analysis model to perform application scenario analysis on the input information and to understand the semantic intent of the input information to determine the actual application scenario of the input information specifically includes: Inputting the user's input information into the first-level scenario analysis model to obtain the first-level business scenarios output by each of the first-level scenario analysis models; Determine the next-level scenario analysis model associated with each first-level business scenario, and input the user's input information into the next-level scenario analysis model to obtain the business sub-scenarios output by the next-level scenario analysis model; Determine the next-level scenario analysis model associated with each business sub-scenario, and input the user's input information into the next-level scenario analysis model to obtain the business sub-scenario outputted by the next-level scenario analysis model, until the user's input information is input into the last-level scenario analysis model to obtain the associated business scenarios outputted by the last-level scenario analysis model; The user's input information is understood for its intent, and the intent understanding result is obtained. Based on the intent understanding result, all related business scenarios are analyzed to obtain the actual application scenario of the input information.

7. The method for answering user questions according to claim 1, characterized in that: The intent understanding is obtained using a trained semantic intent understanding model, which is trained based on sample information, sample intent understanding results corresponding to the sample information, and sample referenced knowledge description information, and the sample intent understanding results are labels for the sample information.

8. A device for answering user questions, characterized in that: The device comprises: Question acquisition module, used to obtain user input information; A scenario analysis module is used to use a multi-level associated scenario analysis model to perform application scenario analysis on the input information and understand the semantic intent of the input information to determine the actual application scenario of the input information; each level has at least one scenario analysis model, each category of business scenarios has an associated scenario analysis model, and between two adjacent levels of scenario analysis models, the business scenario output by the scenario analysis model of the latter level is a business sub-scenario of the business scenario output by the scenario analysis model of the previous level; An element extraction module is used to call a core element template preset for the actual application scenario, and extract core element information in the input information according to the core element template; A data association module is used to call a preset application programming interface associated with an actual application scenario and / or query a preset knowledge base to process the input information and obtain scenario data; The question reasoning module is used to use the preset knowledge base to perform semantic matching on the input information to obtain the reference knowledge and the preset knowledge base to perform semantic retrieval on the input information to obtain the prompt words, and input the reference knowledge, prompt words and scene data as input data into the trained large language model to obtain the reasoning result output by the large language model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method for answering user questions according to any one of claims 1 to 7 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for answering user questions as claimed in any one of claims 1 to 7 are implemented.

Images