Intelligent number asking method based on large language model

Abstract

The invention discloses an intelligent number asking method based on a large language model, and relates to the field of artificial intelligence application, a question proposed by a user is analyzed and understood through an LLM, a number asking API matched with the question is searched according to the question of the user, then a system inputs the searched API, query parameters of the API and the question of the user into the LLM together, and the query parameters of the API and the question of the user are input into the LLM; the LLM further analyzes user questions and API query parameters, then generates an API call request body and provides the API call request body to the system; when the number query API cannot be found, performing a query task by using a query SQL; the problem that the length of the cue word exceeds the context length limit of the LLM due to the fact that the number of APIs for query access is too large and API description is too long is effectively solved, and the situation that when the LLM is pre-trained, some knowledge is not learned or expired, and consequently the problem cannot be solved or mistakenly solved when the number asking requirement of a user is processed is avoided by adding a special label into the cue word.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence application technology, and in particular to an intelligent number-asking method based on a large language model. Background Art

[0002] Intelligent query is a data query method that allows users to interact with the system through natural language to query data. The system returns query results in text, charts, and other formats based on the user's data query requirements. In actual applications, user data query requirements are often complex and diverse, requiring combined queries with multiple filter conditions, resulting in a cumbersome and unfriendly query interface. A common implementation method is to classify user query requirements and design corresponding menu items. When querying, users need to select multiple levels of menus to enter the query interface, which is cumbersome to operate. Another implementation method is to provide multiple filter condition options in the query interface, such as date, time, region, category, etc. Users set filter items according to query needs, and the system performs combined queries based on the filter items set by the user to obtain query results. In some large systems, in order to meet the complex and diverse query needs of users, software developers sometimes combine the above two implementation methods to develop systems.

[0003] Large language models (LLMs) have been around for six years since their initial introduction in 2018. They have been widely applied in fields such as natural language processing, text generation, machine translation, text summarization, and question-answering systems, permeating every aspect of our work and lives. LLMs have also been widely used in intelligent data querying scenarios. While LLMs can understand users' data query requirements to a certain extent, integrating them with application systems—instructing them to call appropriate data retrieval interfaces (APIs) or use appropriate SQL statements to retrieve data from the system—remains a challenge. There are currently two approaches to addressing this issue. One is to use a prompt word approach. This involves providing both a prompt word and a user question to the LLM. The prompt word contains descriptions of all APIs that retrieve data from the system. After analyzing the user question, the LLM matches the prompt word with the appropriate API. It then generates the appropriate API request body and provides it to the system. The system then calls the API to retrieve the data, performs further filtering, statistics, or visualization as needed, and returns it to the user. A drawback of the prompt word approach is the LLM's context length limit. LLMs of different sizes have different context length constraints, such as 8KB, 16KB, and 32KB. This requires that the combined length of the prompt word and user question as input to the LLM must not exceed the LLM's permitted context length constraint. In real-world scenarios, queries may involve a large number of data-retrieval APIs, and API descriptions (including API parameter descriptions) may be lengthy. The prompt word length often exceeds the LLM's permitted context length constraint, rendering the system inoperable.Another way is to use the Retrieval Enhanced Generation (RAG) method, that is, the user's query question is first used as the RAG input, and RAG first searches in the pre-prepared vector database to retrieve one or more possible data acquisition APIs. Then RAG provides the found API and the user's query question as input to the LLM, and then the LLM analyzes and understands the user's query question and finally generates the API request body that the system should call. Compared with the previous prompt word method, the RAG method can effectively avoid the context constraint problem of LLM; however, the disadvantages of the RAG method are also obvious, namely: the RAG module directly performs vector retrieval based on the user's question. It is possible that no suitable API is found (or the retrieved API is inappropriate), but there is actually a suitable API. This is because RAG search only matches based on the vector distance of the text, and sometimes this match is not appropriate. Of course, in the implementation process, the user's query question can be handed over to LLM first, and the text generation ability of LLM can be used to first generate some texts similar to the query question, and then these texts are provided to RAG for retrieval to enhance RAG's ability to retrieve APIs. However, this method is not only difficult to guarantee that it is helpful for API retrieval, but more seriously, it will greatly reduce the system's query speed, thereby affecting the user experience. Summary of the Invention

[0004] The purpose of the present invention is to provide an intelligent questioning method based on a large language model to solve the problems raised in the prior art.

[0005] To achieve the above object, the present invention provides the following technical solution: an intelligent number-asking method based on a large language model, comprising the following steps:

[0006] S1. The user enters natural language to query relevant data through the query engine, and the user's question is provided to the LLM by the system;

[0007] S2, LLM analyzes and understands the user's question, and based on the analysis results, queries the API that matches the user's question;

[0008] S3. If the LLM can retrieve an API that matches the question from the pre-set API list, the LLM returns the matching API name to the system. The system obtains the query parameters contained in the API based on the API name, and then inputs the API, API query parameters, and user question into the LLM.

[0009] S4,LLM further analyzes the user’s question and API query parameters, then generates an API call request body and provides it to the system;

[0010] S5. LLM provides the API call request body to the system. The system calls the API, obtains the data, and returns it to the user.

[0011] S6. If LLM cannot retrieve an API that matches the question from the pre-set API list, the system uses natural language generation technology to analyze the user's question, generate query SQL, and provide the generated query SQL to the system;

[0012] S7. LLM provides the generated query SQL to the system, and the system uses the query SQL to obtain data from the data table and returns it to the user.

[0013] Furthermore, in step S2: a prompt word is set in the LLM, and basic information of all data retrieval APIs is provided in the prompt word, such as the names and descriptions of "device list query interface", "device details query interface" and "alarm query interface" are given in the prompt word; and because of the LLM context length limit, the prompt word only provides the most basic API name and description, and other information such as API call path, API parameters, etc. are not reflected. This operation can shorten the length of the prompt word and ensure that the prompt word does not exceed the LLM context length limit; LLM analyzes and understands the user's question, and combines the API description in the prompt word to search for the data retrieval API that matches the user's question in the prompt word; if the data retrieval API is found, step S3 is executed; if the data retrieval API is not found, step S6 is directly executed.

[0014] Furthermore, in step S3: There are two methods for the system to obtain API query parameters. The system here is a data query system. The data query system obtains the user's question through a search engine and provides the user's question to the LLM. Method 1 for obtaining API query parameters is to develop an API search program as a database search tool. The design of the search program includes storing all APIs and their related query parameter information in a database. Each database record represents the information of an API, and each record contains the following fields:

[0015] name: API name;

[0016] description: API interface description;

[0017] path: API call path;

[0018] query_parameter: a list of API query parameters;

[0019] The database fields are designed using this numeric field format. A database search program is then developed based on this database field design. The program's function is to find records whose name field values ​​match the API name returned to the system by the LLM. The query_parameter field content of the matched records is the API query parameter.

[0020] Furthermore, in step S3: Method 2 for obtaining API query parameters is to store all APIs and their API-related query parameters in a text file through RAG technology, store them in a vector database by means of segmentation, embedding, etc., and use the API name returned to the system by LLM as a query to search the vector database, thereby obtaining the API query parameters.

[0021] Furthermore, in step S4: LLM further analyzes the user's question, extracts key information from the question and associates it with the API query parameters. Since the content and expression of questions asked by different users are different, the information in the questions asked by users are divided into two categories. The first category is information that can be analyzed and understood by LLM, and the second category is information that can be understood by LLM only after conversion; for example, the user's question is "Please query the alarm information of Jiangdong Branch this year", where LLM understands "Jiangdong Branch" as the name of the area, so it directly sets the value of the parameter location to "Jiangdong Branch"; on the other hand, "this year" cannot be directly assigned to the parameters start_date and end_date. Since LLM can understand the meaning of "this year" after training, LLM will convert "this year" and set start_date to "2024-01-01" and end_date to the date of the day "2024-11-01".

[0022] Furthermore, for the first type of information that can be directly analyzed and understood by LLM, after extracting such information, LLM can analyze such information through the data collected in the database and associate the information content with the API query parameters.

[0023] Furthermore, for the second type of information that can only be understood by LLM after conversion, special annotations are set in the prompt words. The annotations require LLM to directly call the plug-in formula after extracting the second type of information, and use the plug-in formula to provide analysis and calculation results; due to the limitations of training corpus and model scale, sometimes LLM cannot understand certain information in the user's question, or misunderstands certain information. This information is the second type of information; for example, the user's question is "Please check the alarm information of Jiangdong Branch last week", and LLM misunderstands "last week" as "calculated from the day of the question to the previous week". If the current date is "2024-11-01 (Friday)", LLM sets the start_date to "2024-10-25 (Friday)" and the end_date to "2024-10-31 (Thursday)", which is the date of the previous day. Obviously, LLM has misunderstood the meaning of "last week". Because LLM cannot directly call external formulas, when executing step S4, it first pre-processes the prompt word. If special annotations are found in the prompt word during pre-processing, the external formula is first called to pre-process the second type of information.

[0024] The method of embedding plug-in formulas in prompt words provides a convenient means for LLM, overcoming the disadvantage that LLM can only obtain pre-trained static knowledge, and solving the problem that RAG technology can only provide LLM with preset static knowledge. Furthermore, we derive the plug-in formula method into a plug-in tool to achieve the purpose of providing real-time, dynamically updated information for LLM.

[0025] Furthermore, in step S5: LLM provides the API call request body to the system. After the system calls the API, it obtains data and returns it to the user. The returned content includes single indicator data, multi-indicator data, and tables and graphs generated according to requirements for visual display.

[0026] Furthermore, in step S6: if the LLM cannot retrieve an API that matches the question from the preset API list, a query SQL is generated according to the user's question;

[0027] Generating query SQL based on user questions directly leverages the NL2SQL capabilities of large language models. Existing LLMs gain powerful language understanding and generation capabilities through training on large-scale corpora. In NL2SQL tasks, introducing training data containing both natural language queries and corresponding SQL queries can further improve model performance.

[0028] Leveraging the NL2SQL capabilities of the large language model, the data table structure involved in the query is provided to the LLM, which then analyzes the user's question and generates SQL that matches it. When the LLM encounters information that it cannot directly analyze and understand, it uses external formulas to analyze and calculate the information. For example, if a user asks, "Query the name, model, IP address, management unit, system description, and model description of all devices whose model number contains 'test0-03'," the LLM can generate the query SQL.

[0029] Furthermore, in step S7: LLM provides the generated query SQL to the system, and the system uses the query SQL to obtain data from the data table under the query permission constraints and returns it to the user; the returned content includes single indicator data, multi-indicator data and tables and graphs generated according to needs for visual display.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] The method for optimizing and improving the prompt word mode proposed in the present invention effectively solves the problem that, when existing software systems implement intelligent number querying based on LLM, the number of query and number acquisition APIs is too large and the API descriptions are too long, resulting in the prompt word length exceeding the context length limit of the large language model; and by adding some special annotations to the prompt words, the present invention makes the large language model more intelligent in processing intelligent number querying, avoiding the problem that the large language model cannot solve or incorrectly solves user number query requirements due to some knowledge not learned or learned but expired during pre-training. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 The figure is a flowchart of the steps of an intelligent number-asking method based on a large language model of the present invention. DETAILED DESCRIPTION

[0033] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0034] like Figure 1 As shown, the present invention provides a technical solution, an intelligent number-asking method based on a large language model, comprising the following steps:

[0035] S1. The user enters natural language to query relevant data through the query engine, and the user's question is provided to the LLM by the system;

[0036] S2, LLM analyzes and understands the user's question, and based on the analysis results, queries the API that matches the user's question;

[0037] S3. If the LLM can retrieve an API that matches the question from the pre-set API list, the LLM returns the matching API name to the system. The system obtains the query parameters contained in the API based on the API name, and then inputs the API, API query parameters, and user question into the LLM.

[0038] S4,LLM further analyzes the user’s question and API query parameters, then generates an API call request body and provides it to the system;

[0039] S5. LLM provides the API call request body to the system. The system calls the API, obtains the data, and returns it to the user.

[0040] S6. If LLM cannot retrieve an API that matches the question from the pre-set API list, the system uses natural language generation technology to analyze the user's question, generate query SQL, and provide the generated query SQL to the system;

[0041] S7. LLM provides the generated query SQL to the system, and the system uses the query SQL to obtain data from the data table and returns it to the user.

[0042] In step S2: a prompt word is set in the LLM, and basic information of all data retrieval APIs is provided in the prompt word, such as the names and descriptions of "device list query interface", "device details query interface" and "alarm query interface" are given in the prompt word; and because of the LLM context length limit, the prompt word only provides the most basic API name and description, and other information such as API call path, API parameters, etc. are not reflected. This operation can shorten the length of the prompt word and ensure that the prompt word does not exceed the LLM context length limit; LLM analyzes and understands the user's question, and combines the API description in the prompt word to search for the data retrieval API that matches the user's question in the prompt word; if the data retrieval API is found, execute step S3; if the data retrieval API is not found, execute step S6 directly.

[0043] In step S3, there are two ways for the system to obtain API query parameters. Method 1 is to develop an API search program as a database search tool. The design of the search program includes storing all APIs and their related query parameter information in the database. Each database record represents the information of an API, and each record contains the following fields:

[0044] name: API name;

[0045] description: API interface description;

[0046] path: the API call path;

[0047] query_parameter: a list of API query parameters;

[0048] The specific examples of database field design are as follows:

[0049] "name":"Alarm query interface", / / API name

[0050] "description": "This interface can query alarm information within a certain time period by area name."

[0051] / / API interface description

[0052] "path":" / api / warning / list", / / API call path "query_parameter":[{"start_date":"start time"},{"end_date":"end time"},{"location":"device region"}], / / API query parameter list, each pair of {} describes a parameter;

[0053] The database fields are designed using this numeric field format. A database search program is then developed based on this database field design. The program's function is to find records whose name field values ​​match the API name returned to the system by the LLM. The query_parameter field content of the matched records is the API query parameter.

[0054] In step S3: Method 2 for obtaining API query parameters is to store all APIs and their API-related query parameters in a text file through RAG technology, store them in a vector database by means of segmentation, embedding, etc., and use the API name returned to the system by LLM as a query to search the vector database to obtain the API query parameters.

[0055] In step S4: LLM further analyzes the user's question, extracts key information from the question, and associates it with the API query parameters. Since the content and expression of different users' questions vary, the information in the user's questions is divided into two categories: the first category is information that can be analyzed and understood by LLM, and the second category is information that can only be understood by LLM after conversion. For example, the user's question is "Please query the alarm information of Jiangdong Branch this year", where LLM understands "Jiangdong Branch" as the name of the region, so it directly sets the value of the parameter location to "Jiangdong Branch". On the other hand, "this year" cannot be directly assigned to the parameters start_date and end_date. Since LLM has been trained to understand the meaning of "this year", LLM will convert "this year" and set start_date to "2024-01-01" and end_date to the date of the day "2024-11-01". The API request body call example is as follows:

[0056] data={

[0057] 'start_date':'2024-01-01',

[0058] 'end_date':'2024-11-01',

[0059] 'location':'Jiangdong Office'

[0060] }

[0061] response=requests.post('http: / / api.example.com / api / warning / list',json=data).

[0062] For the first type of information that can be directly analyzed and understood by LLM, after extracting such information, LLM can analyze such information through the data collected in the database and associate the information content with the API query parameters.

[0063] Due to limitations in the training corpus and model size, LLM sometimes fails to understand certain information in user questions or misunderstands it. For example, if a user asks, "Please inquire about the alarm information for Jiangdong Branch last week," LLM misinterprets "last week" as meaning "the week before the question was asked." Since today's date is "2024-11-01 (Friday)," LLM converts the start_date to "2024-10-25 (Friday)" and the end_date to the day before today, "2024-10-31 (Thursday)." Clearly, LLM misunderstands the meaning of "last week." Therefore, for the second type of information that can only be understood by LLM after conversion, special annotations are set in the prompt words. This annotation requires LLM to directly call the plug-in formula after extracting the second type of information and use it to provide analysis and calculation results. For example, if LLM encounters time-related information such as "last week" or "last month," it should annotate the prompt word with "<**" and ">**" as the start and end markers. An example annotation is as follows:

[0064] **Note: If you are asked to provide the start date of "last week", please use the external formula to calculate. The start date and end date will be in the start_date and end_date of the result returned by the external formula. **>

[0065] Then, when a markup is encountered, the plug-in formula is used to calculate the starting date of the previous week. The Python code example of the plug-in formula is as follows:

[0066]

[0067] #Calculate last week's Sunday

[0068] sunday=monday+timedelta(days=6)

[0069] end_date=Sunday

[0070] Because LLM cannot directly call the plug-in formula, when executing step S4, the system first pre-processes the prompt word. When the prompt word is retrieved during pre-processing, if there is a special markup, the plug-in formula is first called to pre-process the second type of information;

[0071] The method of embedding plug-in formulas in prompt words provides a convenient means for LLM, overcoming the disadvantage that LLM can only obtain pre-trained static knowledge, and solving the problem that RAG technology can only provide LLM with preset static knowledge. Furthermore, we derive the plug-in formula method into a plug-in tool to achieve the purpose of providing real-time, dynamically updated information for LLM.

[0072] In step S5: LLM provides the API call request body to the system. After calling the API, the system obtains data and returns it to the user. The returned content includes single indicator data, multi-indicator data, and tables and graphs generated according to requirements for visual display.

[0073] In step S6: if the LLM cannot retrieve an API that matches the question from the preset API list, a query SQL is generated according to the user's question;

[0074] Generating query SQL based on user questions directly leverages the NL2SQL capabilities of large language models. Existing LLMs gain powerful language understanding and generation capabilities through training on large-scale corpora. In NL2SQL tasks, introducing training data containing both natural language queries and corresponding SQL queries can further improve model performance.

[0075] Leveraging the NL2SQL capabilities of the large language model, the data table structure involved in the query is provided to the LLM, which then analyzes the user's question and generates SQL that matches it. When the LLM encounters information that it cannot directly analyze and understand, it uses external formulas to analyze and calculate the information. For example, if a user asks, "Query the name, model, IP address, management unit, system description, and model description of all devices whose model number contains 'test0-03'," the LLM can generate the query SQL.

[0076] In step S7: LLM provides the generated query SQL to the system. The system uses the query SQL to obtain data from the data table under the query permission constraints and returns it to the user; the returned content includes single-indicator data, multi-indicator data, and tables and graphs generated according to requirements for visual display.

[0077] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. An intelligent questioning method based on a large language model, characterized by: The following steps are involved: S1. The user enters natural language to query relevant data through the query engine, and the user's question is provided to the LLM by the system; S2, LLM analyzes and understands the user's question, and queries the API that matches the user's question based on the analysis results; S3. If LLM can retrieve an API that matches the question from the preset API list, LLM returns the matched API name to the system. The system obtains the query parameters contained in the API according to the API name, and then inputs the API, the API query parameters and the user question into LLM. S4, LLM further analyzes the user's question and API query parameters, then generates an API call request body and provides it to the system; S5. LLM provides the API call request body to the system. The system calls the API, obtains the data and returns it to the user. S6. If LLM cannot retrieve an API that matches the question from the preset API list, the system uses natural language to generate database query statements to analyze the user's question, generate query SQL, and provide the generated query SQL to the system; S7. LLM provides the generated query SQL to the system, and the system uses the query SQL to obtain data from the data table and returns it to the user.

2. The intelligent questioning method based on a large language model according to claim 1, characterized in that: In step S2: a prompt word is set in the LLM, and information of all data retrieval APIs is provided in the prompt word. Under the LLM context length limit, the prompt word only provides the API name and description. LLM analyzes and understands the user's question, and searches for the data retrieval API that matches the user's question in the prompt word in combination with the API description in the prompt word; if the data retrieval API is found, execute step S3; if the data retrieval API is not found, directly execute step S6.

3. The intelligent questioning method based on a large language model according to claim 2, characterized in that: In step S3: There are two methods for the system to obtain API query parameters. Method 1 is to store all API and API query parameter information in the database. Each database record represents the information of an API. Each record includes four fields for matching the four basic information of the API, namely name represents the API name, description represents the API interface description, path represents the API call path, and query_parameter represents the API query parameter list; then use the database retrieval tool to find the record whose name field value matches the API name returned to the system by LLM, and the content of the matched query_parameter field is the API query parameter.

4. The intelligent questioning method based on a large language model according to claim 3, characterized in that: In step S3: Method 2 for obtaining API query parameters is to store all API and API query parameter information into a text file through RAG technology, store the text file into a vector database by means of segmentation and embedding, and use the API name returned to the system by LLM as a query to search the vector database, thereby obtaining API query parameters.

5. The intelligent questioning method based on a large language model according to claim 4, characterized in that: In step S4: LLM further analyzes the user's question, extracts information from the question and associates it with the API query parameters, and divides the information in the question raised by the user into two categories, the first category is information that can be directly analyzed and understood by LLM, and the second category is information that can be understood by LLM after conversion.

6. The intelligent questioning method based on a large language model according to claim 5, characterized in that: For the first type of information that can be directly analyzed and understood by LLM, after extracting such information, LLM can analyze such information through the data collected in the database and associate the information content with the API query parameters.

7. The intelligent questioning method based on a large language model according to claim 5, characterized in that: For the second type of information that can be understood by LLM after conversion, special annotations are set in the prompt words. The annotations require LLM to directly call the external formula after extracting the second type of information, and use the external formula to provide analysis and calculation results; and when the system executes step S4, the prompt words are first pre-processed. When the prompt words are retrieved during preprocessing and there are special annotations, the system first calls the external formula to pre-process the second type of information. During preprocessing, the system calls the external formula according to the annotation requirements to analyze and calculate the second type of information, and inputs the calculation results into LLM for further analysis and processing.

8. The intelligent questioning method based on a large language model according to claim 5, characterized in that: In step S5: LLM provides the API call request body to the system. After the system calls the API, it obtains data and returns it to the user. The returned content includes single-indicator data, multi-indicator data, and tables and graphs generated according to requirements.

9. The intelligent questioning method based on a large language model according to claim 2, characterized in that: In step S6: if LLM cannot retrieve an API that matches the question from the preset API list, a query SQL is generated based on the user's question. The NL2SQL capability of the large language model is directly used to provide the data table structure involving the query data to LLM, which analyzes the user's question and generates a query SQL that matches the user's question. When encountering information that LLM cannot directly analyze and understand, the plug-in formula is used to pre-process the information.

10. The intelligent questioning method based on a large language model according to claim 9, characterized in that: In step S7: LLM provides the generated query SQL to the system, and the system uses the query SQL to obtain data from the data table under the query permission constraints and returns it to the user; the returned content includes single-indicator data, multi-indicator data, and tables and graphs generated according to requirements.

Images