LLM-based index query method and device

Abstract

The index query method based on the LLM comprises the following steps: obtaining metadata of an index to be queried, and constructing a training sample set for performing fine tuning training on a basic LLM based on the metadata; the training sample comprises a structured query description information sample constructed based on metadata; the user question sample is matched with the query semantics of the query description information sample; performing fine tuning training on the basic LLM based on the training sample set to obtain a service LLM; the service LLM is used for converting a user question input by a user into the structured query description information matched with the query semantics of the user question; converting a user question input by a user and used for initiating query into the structured query description information matched with the query semantics of the user question on the basis of the service LLM, and querying for a target index stored in a database on the basis of the query description information; wherein the target index is an index specified by the user in the user question.

Description

Technical Field

[0001] The embodiments of this specification belong to the field of artificial intelligence, and more particularly to an LLM-based indicator query method and device. Background Art

[0002] Indicators are usually core data in a business system. Business personnel can usually monitor business progress or business health based on indicators. For example, taking the above business system as an online business system, in actual applications, online services, revenue, sales, user volume, etc. can be monitored in combination with the indicators of the online business system.

[0003] However, in traditional indicator-based BI (Business Intelligence) analysis, it is usually necessary to manually build business reports first, and then define the target indicators that need to be paid attention to. This process is very cumbersome and requires operators to have both business and technical backgrounds, resulting in a sharp increase in the cost for business personnel to view indicators. Summary of the Invention

[0004] This specification proposes an indicator query method based on LLM, including:

[0005] Obtain metadata of the indicator to be queried, and construct a training sample set for fine-tuning the basic LLM based on the obtained metadata; wherein the training sample includes a structured query description information sample constructed based on the metadata; and a user question sample that matches the query semantics of the query description information sample;

[0006] Fine-tuning the basic LLM based on the training sample set to obtain a service LLM; wherein the service LLM is used to convert the user question input by the user into the structured query description information that matches the query semantics of the user question;

[0007] Based on the service LLM, the user question input by the user for initiating a query is converted into the structured query description information that matches the query semantics of the user question, and based on the query description information, a query is performed on the target indicator stored in the database; wherein, the target indicator is the indicator specified by the user in the user question.

[0008] Optionally, the query description information includes query description information based on a domain-specific language.

[0009] Optionally, the domain-specific language includes a domain-specific language based on the JSON data format.

[0010] Optionally, constructing a training sample set for fine-tuning the basic LLM based on the acquired metadata includes:

[0011] Obtaining a preset query description information template set; wherein the query description information templates in the query description information template set correspond to different query semantics; and the query description information templates include a number of query parameters;

[0012] A state machine for automatically generating training samples is constructed based on the query description information template set; wherein the states in the state set maintained in the state machine correspond one-to-one to the query description information templates in the query description information template set; and the state parameters of the states are consistent with the query parameters contained in the query description information template corresponding to the states;

[0013] Filling the values of the state parameters of the states in the state set based on the metadata of the indicator to be queried to construct a query description information sample corresponding to the state, and generating a user question sample that matches the query semantics of the query description information sample;

[0014] A training sample is generated based on the query description information sample and the user question sample, and a training sample set is constructed based on the generated training sample.

[0015] Optionally, filling the values of the state parameters of the states in the state set based on the metadata of the indicator to be queried includes:

[0016] randomly selecting an indicator from the indicators to be queried, and further randomly selecting metadata from the metadata of the randomly selected indicator;

[0017] The values of the state parameters of the states in the state set are filled based on the randomly selected metadata.

[0018] Optionally, at least some of the states in the state set maintained in the state machine are defined with an execution action set; wherein the execution action in the execution action set is used to modify the value of a specified state parameter in the state parameters of the state;

[0019] Before constructing a training sample set based on the generated training samples, it also includes:

[0020] Triggering the execution of an action in the execution action set corresponding to the state, modifying the value of a specified state parameter in the state parameters of the state to construct an extended sample corresponding to the query description information sample, and generating a user follow-up question sample that matches the query semantics of the extended sample; wherein the user follow-up question sample is a question sample generated by the user asking a follow-up question based on the user question sample;

[0021] A training sample is generated based on the extended sample and the user question sample.

[0022] Optionally, the actions in the execution action set each have a preset probability of triggering execution;

[0023] Triggering the execution of actions in the execution action set corresponding to the state, including:

[0024] Based on the probabilities of triggering execution respectively preset for the actions in the execution action set, the actions in the execution action set corresponding to the state are triggered to execute.

[0025] Optionally, the user question sample and the user follow-up question sample are samples generated based on a preset user question template;

[0026] Before fine-tuning the basic LLM based on the training sample set, the method further includes:

[0027] Constructing prompt words corresponding to training samples in the training sample set; wherein the prompt words are used to instruct the basic LLM to convert the user question text or user follow-up question sample contained in the training sample into a colloquial user question sample or user follow-up question sample that matches the query semantics of the query description information sample contained in the training sample;

[0028] The prompt word is input into the basic LLM, so that the basic LLM performs inference calculation based on the prompt word, and converts the user question text or user follow-up question sample contained in the training sample into a colloquial user question sample or user follow-up question sample that matches the query semantics of the query description information sample contained in the training sample.

[0029] Optionally, before querying the target indicator stored in the database based on the query description information, the method further includes:

[0030] Obtaining the entity name contained in the query description information;

[0031] Retrieving a standardized entity ID corresponding to the entity name from a database based on the entity name;

[0032] If a standardized entity ID corresponding to the entity name is retrieved from the database, the entity name included in the query description information is replaced with the entity ID.

[0033] Optionally, before querying the target indicator stored in the database based on the query description information, the method further includes:

[0034] Parsing the query description information to determine whether the query semantics of the query description information include external call semantics;

[0035] If the query semantics of the query description information include external call semantics, an external call related to the external call semantics is executed, and the external call result is added to the query description information.

[0036] Optionally, before querying the target indicator stored in the database based on the query description information, the method further includes:

[0037] Performing executable verification on the query semantics of the query description information;

[0038] If the query semantics of the query description information passes the executable verification, a query for the target indicator stored in the database is further initiated based on the query description information.

[0039] Optionally, performing executable verification on the query semantics of the query description information includes:

[0040] Determining whether there are multiple indicators in the query semantics of the query description information that require simultaneous query initiation;

[0041] If the query semantics of the query description information include multiple indicators that need to be queried simultaneously, verify whether the multiple indicators have the same query dimension; if so, determine whether the query semantics of the query description information has passed the executable verification.

[0042] Optionally, the query description information includes query parameters for initiating the query and visualization parameters for visually displaying the queried parameters;

[0043] Querying the target indicator stored in the database based on the query description information includes:

[0044] A query is performed on the target indicator stored in the database based on the query parameters included in the query description information, and a visual display is performed on the queried target indicator based on the visual display parameters.

[0045] This specification also proposes an indicator query device based on LLM, including:

[0046] An acquisition module acquires metadata of the indicator to be queried and constructs a training sample set for fine-tuning the basic LLM based on the acquired metadata; wherein the training sample includes a structured query description information sample constructed based on the metadata; and a user question sample that matches the query semantics of the query description information sample;

[0047] A training module, which fine-tunes the basic LLM based on the training sample set to obtain a service LLM; wherein the service LLM is used to convert the user question input by the user into the structured query description information that matches the query semantics of the user question;

[0048] The query module converts the user question input by the user for initiating the query into the structured query description information that matches the query semantics of the user question based on the service LLM, and queries the target indicator stored in the database based on the query description information; wherein the target indicator is the indicator specified by the user in the user question.

[0049] In the above embodiment, on the one hand, a service LLM is obtained by fine-tuning the basic LLM, which is capable of converting user questions input by the user into structured query description information that matches the query semantics of the user question. The service LLM is used to convert the user questions input by the user into structured query description information that matches the query semantics of the user question, so that the user can quickly and conveniently query the indicator through natural language, thereby simplifying the query process for the indicator and improving the query efficiency of the indicator.

[0050] On the other hand, in the process of fine-tuning the basic LLM, it is no longer necessary to use general corpus for fine-tuning training. Instead, it is possible to use metadata of the indicators to be queried that contain rich business semantics for fine-tuning training. In this way, the service LLM that has completed fine-tuning training can accurately capture the business semantics contained in the metadata of the indicators to be queried involved in the user's questions, thereby optimizing the query effect for the indicators and allowing ordinary users without business background to conduct in-depth business-level queries on the indicators to be queried through natural language. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0052] Figure 1 This is a flow chart of an LLM-based indicator query method shown in an embodiment of this specification;

[0053] Figure 2 This is a flow chart of fine-tuning training of a basic LLM shown in an embodiment of this specification;

[0054] Figure 3This is a schematic diagram of a query description information template constructed using a DSL based on the JSON data format, as shown in an embodiment of this specification;

[0055] Figure 4 This is a schematic diagram of a query description information sample constructed using a DSL based on the JSON data format, as shown in an embodiment of this specification;

[0056] Figure 5 This is a flowchart of an embodiment of this specification in which a user queries an indicator with business semantics through natural language;

[0057] Figure 6 This is a schematic diagram of another query description information sample constructed using a DSL based on the JSON data format shown in an embodiment of this specification;

[0058] Figure 7 is a schematic structural diagram of an electronic device shown in an embodiment of this specification;

[0059] Figure 8 This is a block diagram of an LLM-based indicator query device shown in an embodiment of this specification. DETAILED DESCRIPTION

[0060] To help those skilled in the art better understand the technical solutions in this specification, the following will provide a clear and complete description of the technical solutions in the embodiments of this specification, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. All other embodiments derived by those skilled in the art based on the embodiments in this specification without creative effort shall fall within the scope of protection of this specification.

[0061] This specification aims to propose a technical solution that introduces LLM (Large Language Model) to enable ordinary users without business background to conduct in-depth queries on indicators with business semantics through natural language.

[0062] Based on this technical solution, metadata of the indicator to be queried can be first obtained, and a training sample set for fine-tuning the basic LLM can be constructed based on the obtained metadata. The training sample includes a structured query description information sample constructed based on the metadata; and a user question sample that matches the query semantics of the query description information sample.

[0063] Secondly, the basic LLM can be fine-tuned based on the training sample set to obtain a service LLM; wherein the service LLM is used to convert the user question input by the user into the structured query description information that matches the query semantics of the user question;

[0064] Finally, based on the service LLM, the user question input by the user for initiating a query can be converted into the structured query description information that matches the query semantics of the user question, and based on the query description information, a query can be performed on the target indicator stored in the database; wherein, the target indicator is the indicator specified by the user in the user question.

[0065] In the above technical solution, on the one hand, a service LLM is obtained by fine-tuning the basic LLM, which can convert the user questions input by the user into structured query description information that matches the query semantics of the user questions. The service LLM is used to convert the user questions input by the user into structured query description information that matches the query semantics of the user questions, so that the user can quickly and conveniently query the indicators through natural language, thereby simplifying the query process for the indicators and improving the query efficiency of the indicators.

[0066] On the other hand, in the process of fine-tuning the basic LLM, it is no longer necessary to use general corpus for fine-tuning training. Instead, it is possible to use metadata of the indicators to be queried that contain rich business semantics for fine-tuning training. In this way, the service LLM that has completed fine-tuning training can accurately capture the business semantics contained in the metadata of the indicators to be queried involved in the user's questions, thereby optimizing the query effect for the indicators and allowing ordinary users without business background to conduct in-depth business-level queries on the indicators to be queried through natural language.

[0067] See Figure 1 , Figure 1 This is a flowchart of an LLM-based indicator query method shown in this specification; the method includes the following execution process:

[0068] Step 102: Obtain metadata of the indicator to be queried, and construct a training sample set for fine-tuning the basic LLM based on the obtained metadata; wherein the training sample includes a structured query description information sample constructed based on the metadata; and a user question sample that matches the query semantics of the query description information sample;

[0069] In this specification, based on the basic LLM, the basic LLM can be fine-tuned by constructing training samples, and then a service LLM can be obtained that can convert user questions input by users into structured query description information that matches the query semantics of the user questions.

[0070] The above-mentioned basic LLM (also referred to as base LLM) can specifically adopt any form of LLM with reasoning capabilities; for example, in practical applications, an open source general LLM can be adopted as the basic LLM.

[0071] To allow ordinary users without a business background to query indicators with business semantics through natural language, when fine-tuning the basic LLM, general corpus can no longer be used as training samples. Instead, metadata of the indicators to be queried that contain rich business semantics can be used as training samples.

[0072] See Figure 2 , Figure 2 This is a flowchart of fine-tuning training for a basic LLM as shown in this specification.

[0073] like Figure 2 As shown, when training the basic LLM, the metadata of the indicator to be queried can be obtained first; for example, in actual applications, the indicator to be queried can be pre-calculated in an offline manner, and then the calculated indicator and the metadata related to the indicator can be stored in a database. When the metadata of the indicator to be queried is needed, the metadata of the indicator to be queried can be obtained by accessing the database. Of course, in addition to using an offline calculation method to calculate indicators, in actual applications, a method of calculating indicators in real time can also be used. In this case, the indicator can be calculated in real time when a certain indicator needs to be queried.

[0074] It should be noted that the aforementioned indicators to be queried may include any form of indicators with query requirements. For example, in actual applications, several key indicators that users need to access frequently can be screened from all indicators in the business system as the indicators to be queried.

[0075] The metadata of the indicator to be queried may include any form of data describing the indicator; for example, the metadata of the indicator to be queried may specifically include the name of the indicator, the associated dimensions of the indicator and the corresponding dimension values, the period corresponding to the indicator, and so on.

[0076] After obtaining the metadata of the indicator to be queried, a training sample set for fine-tuning the basic LLM can be constructed based on the obtained metadata.

[0077] Please continue to see Figure 2 In the scenario described in this specification where a user queries an indicator using natural language, the training samples in the training sample set may specifically include a query description information sample constructed based on the metadata of the indicator to be queried, and a user question sample that matches the query semantics of the query description information sample. That is, each training sample in the training sample set may specifically be a training sample pair consisting of the query description information sample and the user question sample.

[0078] The query description information may be structured. For example, in practical applications, the query description information may be constructed using a structured programming language. The query description information may include several predefined query parameters that describe all necessary information for an indicator query process.

[0079] In an embodiment shown, the query description information may specifically include query description information based on DSL (domain-specific language).

[0080] The data format used by the DSL is not particularly limited in this specification. In practical applications, any structured data format may be used. For example, in one example, the DSL may specifically include a DSL based on the JSON data format.

[0081] By using structured programming languages and data formats to construct the above query description information, it is possible to facilitate permission control over query permissions and avoid the problem of being unable to accurately control query permissions when using unstructured programming languages.

[0082] When constructing a training sample set for fine-tuning the basic LLM based on the acquired metadata, in order to improve the efficiency of constructing the training sample set, an execution framework that can support automation and large-scale construction can be used.

[0083] In one embodiment shown, see Figure 2 Specifically, a state machine framework can be used to automatically construct a training sample set.

[0084] In this case, first, all query semantics that may be involved when the user queries the indicator to be queried can be listed, and query description information templates can be customized based on these query semantics. Then, a query description information template set can be generated based on the generated query description information templates.

[0085] For example, in one example, developers or business personnel can manually edit these query semantics into query description information templates based on DSL. Each query description information template corresponds to a query syntax for querying the indicator to be queried.

[0086] Secondly, when constructing a training sample set for fine-tuning the basic LLM based on the acquired metadata, a pre-generated set of query description information templates can be obtained. The query description information templates in this set can each correspond to a different query semantics. Specifically, each query description information template can include several pre-defined query parameters that describe all necessary information for a single indicator query process. Initially, the query parameters included in the query description information templates may not have values set, or may only have initial default values set.

[0087] Then, a state machine for automatically generating training samples can be constructed based on the acquired query description information template set. The states in the state set maintained by the generated state machine can correspond one-to-one with the query description information templates in the query description information template set. That is, each query description information template can correspond to a state maintained by the state machine. The state parameters of the states maintained in the state machine can be consistent with the query parameters contained in the query description information template corresponding to the state.

[0088] Furthermore, after the above-mentioned state machine is constructed, the values of the state parameters of the states in the state set maintained by the state machine can be automatically filled in based on the acquired metadata to construct a query description information sample corresponding to each state. After constructing the query description information sample corresponding to each state maintained in the state machine, user question samples that match the query semantics of the above-mentioned query description information samples can be generated based on the query semantics of these query description information samples.

[0089] For example, see Figure 2 In practical applications, some user question templates can be preset for each query semantics, and then these user question templates can be referred to to generate user question samples that match the query semantics of the above query description information samples.

[0090] It should be noted that in actual applications, multiple user question templates can be set for each query semantics. In this way, the same query semantics can be expressed from different perspectives, thereby enriching the generated training samples and improving the diversity of the generated training samples.

[0091] The following describes in detail the specific process of constructing query description information samples and user question samples corresponding to the states maintained by the state machine through a specific example.

[0092] For example, assuming that the indicator to be queried is the GDP of each province or city, the query description information template corresponding to a certain state maintained by the above state machine can be described as follows using a DSL based on the JSON data format: Figure 3 The code is shown in the form.

[0093] in, Figure 3 In the code shown, metric_filter defines filtering conditions for numerical metrics; date_filter filters data based on events or date ranges, allowing users to specify a specific time period; and dimension_filter defines the dimension of a metric, allowing users to specify a specific dimension to describe the metric. drill_down is a data analysis technique that allows users to drill down from high-level summary data to more detailed, granular data.

[0094] Assuming that the metadata corresponding to the GDP indicator is obtained, including the province corresponding to GDP (assuming it is Zhejiang Province), population (assuming it is 5 million), per capita GDP (assuming it is 100,000 yuan), city, population, etc., then based on these metadata, after randomly filling the query parameters in the query description information template corresponding to the state (that is, the state parameters of the state), the query description information sample obtained can be as follows: Figure 4 shown. Figure 4 The query semantics of the query description information sample shown is to find cities in Zhejiang Province with a population of more than 5 million and a per capita GDP of more than 100,000 yuan last year. A user question sample generated based on the preset question template that matches the query semantics of the query description information sample can be specifically "Which cities in Zhejiang Province had a population of more than 5 million and a per capita GDP of more than 100,000 yuan last year? Rank them from largest to smallest by population size?"

[0095] After automatically constructing query description information samples corresponding to each state in the state set maintained by the state machine, as well as user question samples corresponding to the query description information samples, a corresponding training sample can be generated for each state based on the query description information samples corresponding to each state and the user question samples corresponding to the query description information samples. In other words, for any state in the state set maintained by the state machine, the sample pair formed by the automatically constructed query description information sample corresponding to the state and the user question sample corresponding to the query description information sample is called the training sample corresponding to the state.

[0096] It should be noted that the filling method used when filling the values of the state parameters of the state in the state set maintained by the state machine based on the acquired metadata is not specifically limited in this specification. In actual application, it can be flexibly selected based on specific needs.

[0097] For example, in one embodiment shown, when filling in the values of the state parameters of the states in the state set maintained by the state machine based on the acquired metadata, an indicator can be randomly selected from the indicators to be queried, and metadata can be further randomly selected from the metadata of the randomly selected indicator, and then the values of the state parameters of the states in the state set maintained by the state machine can be filled in based on the randomly selected metadata.

[0098] In this way, the randomness of the metadata of the indicators contained in the final generated training samples can be ensured, so that the metadata of the indicators contained in the training samples in the generated training sample set can be evenly distributed in each training sample to ensure the availability of the training samples.

[0099] In order to ensure that the training samples in the training sample set finally constructed can fully simulate the scenario of multiple rounds of dialogue in the process of users querying indicators with business semantics through natural language, in actual applications, it is also possible to simulate the scenario of multiple rounds of dialogue, and expand the query description information sample corresponding to each state in the state set maintained by the state machine and the user question sample corresponding to the query description information sample to generate extended samples corresponding to the above query description information samples and user follow-up samples corresponding to the above user question samples.

[0100] In one embodiment shown, see Figure 2 , an action set may also be defined for at least some of the states in the state set maintained in the state machine; wherein the actions in the action set are used to modify the values of specified state parameters in the state corresponding to the action, thereby transitioning the state to another state. In practical applications, the actions in the action set defined for a state may include pre-specified state parameters to be modified and the values of the modified state parameters.

[0101] In this case, after automatically constructing the query description information samples corresponding to each state in the state set maintained by the state machine and the user question samples corresponding to the query description information samples, it is also possible to trigger the execution of the actions in the execution action set corresponding to each state, modify the values of the specified state parameters in the state parameters of the state, and construct the extended samples corresponding to the query description information samples. After constructing the extended samples corresponding to the query description information samples, it is also possible to generate user follow-up question samples that match the query semantics of the extended samples based on the query semantics of these extended samples. The user follow-up question samples specifically refer to question samples generated by users asking follow-up questions based on the above user question samples.

[0102] For example, see Figure 3 and Figure 4 , assuming that the query description template corresponding to this state is Figure 3 As shown, an execution action corresponding to this state is used to modify the dimension parameter under dimension_filter in this state, and it is pre-specified that the dimension parameter is changed from "Zhejiang Province" to "Jiangsu Province". After triggering the execution of this action, Figure 4 The value of the dimension parameter under dimension_filter in the query description information sample shown is changed from "Zhejiang Province" to "Jiangsu Province" to generate Figure 4 The extended sample corresponding to the query description information sample shown. At this time, based on the query semantics of the query extended sample, the generated user question sample that matches the query semantics of the above extended sample can be specifically "Which cities in Jiangsu Province had a population of more than 5 million and a per capita GDP of more than 100,000 yuan last year? Rank them from largest to smallest by population?"

[0103] After triggering the execution of an action in the execution action set corresponding to each state and modifying the value of the specified state parameter in the state parameter of the state to construct an extended sample corresponding to the query description information sample and a user question sample that matches the query semantics of the extended sample, a corresponding training sample can be generated for each state based on the query description information sample corresponding to each state and the user question sample corresponding to the query description information sample, as well as the extended sample corresponding to the query description information sample and the user question sample that matches the query semantics of the extended sample.

[0104] At this time, for any state, the training sample corresponding to the state can be specifically a sample sequence composed of several training samples; wherein, in this sample sequence, it can specifically include a training sample composed of a query description information sample corresponding to the state and a user question sample corresponding to the query description information sample, as well as several extended training samples composed of an extended sample corresponding to the query description information sample and a user follow-up question sample that matches the query semantics of the extended sample.

[0105] It should be noted that the specific method of triggering the execution of actions in the execution action set corresponding to each state is no longer specifically limited in this specification. In actual applications, the triggering method can be flexibly selected based on specific needs.

[0106] For example, in one example, each state in the state machine can be traversed. Whenever a state is traversed, an execution action can be randomly selected from the execution action set corresponding to the state to trigger execution, and then the next state can be traversed. And so on, N rounds of traversal can be looped to continuously generate new samples.

[0107] In one embodiment shown, the execution actions in the execution action set defined for at least some of the states in the state set maintained in the above-mentioned state machine can also respectively preset the probability of triggering execution; the probability of triggering execution can specifically be used to represent the transition probability between each state in the state machine.

[0108] In this case, the actions in the execution action set may be triggered to execute based on the probability of triggering execution preset for each action in the execution action set.

[0109] Since actions with a higher probability of being triggered are usually triggered more times than actions with a lower probability of being triggered, this will result in the number of extended training samples generated corresponding to actions with a higher probability of being triggered being greater than the number of extended training samples corresponding to actions with a lower probability of being triggered. Therefore, this method can be used to control the proportion of the number of extended training samples generated corresponding to specific actions in the training sample set, thereby ensuring that the final generated training sample set can truly simulate the user's daily query process.

[0110] For example, a higher trigger execution probability can be set for the execution actions corresponding to some common user questions, thereby ensuring that the extended training samples corresponding to these common user questions can account for a larger proportion in the final generated training sample set.

[0111] In actual applications, the constructed user question samples and user follow-up question samples are usually samples generated based on preset templated user question templates; therefore, this may result in the constructed user question samples and user follow-up question samples not conforming to the user's daily spoken habits, which in turn affects the fine-tuning effect of the basic LLM, making it impossible for the service LLM obtained by fine-tuning training to accurately extract the user's true intention.

[0112] Based on this, please continue to see Figure 2 After constructing a training sample set for fine-tuning the basic LLM, data enhancement processing can be performed on the user question texts or user follow-up question samples contained in the training samples in the training sample set to convert the user question texts or user follow-up question samples contained in these training samples into colloquial user question texts or user follow-up question samples that are more in line with the user's spoken language habits.

[0113] In one embodiment shown, before fine-tuning the basic LLM based on the training sample set, a prompt word corresponding to the training sample in the training sample set can also be constructed; wherein, the prompt word is specifically used to instruct the basic LLM to convert the user question text or user follow-up question sample contained in the training sample into a colloquial user question sample or user follow-up question sample that matches the query semantics of the query description information sample contained in the training sample.

[0114] It should be noted that the specific content of the constructed prompt word will not be described in detail in this specification. In actual application, the prompt word can be flexibly edited based on the above requirements.

[0115] After the above-mentioned prompt words are constructed, the prompt words can be input into the basic LLM, so that the basic LLM can perform inference calculations based on the prompt words and convert the user question text or user follow-up question sample contained in the training sample into a colloquial user question sample or user follow-up question sample that matches the query semantics of the query description information sample contained in the training sample.

[0116] Step 104: fine-tune the basic LLM based on the training sample set to obtain a service LLM; wherein the service LLM is used to convert the user question input by the user into the structured query description information that matches the query semantics of the user question;

[0117] Please continue to see Figure 2 After constructing a training sample set for fine-tuning the basic LLM, the basic LLM can be fine-tuned based on the training sample set to obtain a service LLM that has the ability to convert user questions input by users into structured query description information that matches the query semantics of the user questions.

[0118] Among them, the training method used for fine-tuning the basic LLM will not be described in detail in this specification. When those skilled in the art implement the technical solution disclosed in this specification, they can refer to the records in the relevant technology.

[0119] Step 106: Based on the service LLM, the user question input by the user for initiating the query is converted into the structured query description information that matches the query semantics of the user question, and based on the query description information, a query is performed on the target indicator stored in the database; wherein, the target indicator is the indicator specified by the user in the user question.

[0120] After fine-tuning the basic LLM, the service LLM obtained through fine-tuning can be deployed in the business system. Based on the model capabilities of the service LLM, query services for indicators can be provided to users.

[0121] See Figure 5 , Figure 5 This is a flowchart of a user querying an indicator with business semantics using natural language, as shown in this specification.

[0122] like Figure 5 As shown in the figure, when a user needs to query a specific target indicator, they can specifically enter a question in natural language, using a conversational interaction method to query the target. The business system deployed with the service LLM can obtain the user's question, construct a prompt word corresponding to the query based on the context of the user's question, and then input the constructed prompt word into the service LLM. The service LLM performs inference calculation based on the prompt word, converting the user's colloquial question into structured query description information that matches the query semantics of the user's question, and then outputs the structured query description information.

[0123] It should be noted that the specific content of the constructed prompt word is no longer specifically limited in this specification. In actual application, the prompt word can be flexibly edited based on specific needs.

[0124] For example, a complete prompt word can usually include the following three parts:

[0125] Instructions: These are usually overall instructions for the service LLM; for example, requiring the model to output answers that conform to the JSON format;

[0126] Context: usually some necessary contextual information; for example, the source information of the user's question;

[0127] Conversation history: Questions the user asked to previous users.

[0128] It should be noted that since the multi-round dialogue scenario of users is simulated in the training samples during the fine-tuning training phase, the service LLM can refer to the dialogue history when generating structured query description information to determine whether the current user question is a follow-up question, and use different methods to generate structured query description information for the first question and the user follow-up question.

[0129] For example, when the service LLM makes inferences based on the prompt word, it can refer to the conversation history part in the prompt word to determine whether the current user question is the first user question;

[0130] On the one hand, if the current user question is the first time the user has asked the question, then it is not possible to refer to the structured query description information generated for the previous historical user question. Instead, the user's colloquial question can be directly converted into structured query description information that matches the query semantics of the user's question.

[0131] On the other hand, if the user question this time is not the first user question, but an additional question based on the previous historical user question, the service LLM can first obtain the structured query description information that has been generated corresponding to the previous historical user question, and compare the current user question contained in the prompt word with the content of the previous historical user question to clarify the query parameters that need to be modified, and then modify the corresponding query parameters contained in the structured query description information that has been generated corresponding to the previous historical user question to generate corresponding structured query description information for this user question.

[0132] Please continue to see Figure 5 After the service LLM converts the colloquial user question input by the user into structured query description information that matches the query semantics of the user question, in order to ensure that the structured query description information can be executed normally, the business system can further convert the structured query information into executable query description information before initiating a query for the target indicator stored in the database based on the structured query description information.

[0133] It should be explained that executable query description information may generally refer to query description information that can comply with the database's specifications regarding data queries.

[0134] For example, in actual applications, since the above-mentioned structured query description information is directly converted from spoken user questions, it may contain some information that does not comply with the database's specifications for data queries. Therefore, in order to ensure that the structured query description information complies with the database's specifications for data queries, the business system can convert the structured query information into query description information that complies with the database's data query specifications based on the database's data query specifications.

[0135] In one embodiment shown, the structured query description information may generally include some non-standard colloquial entity names. These non-standard colloquial entity names may cause the structured query description information to be unable to be directly executed in the database.

[0136] In this case, when converting the structured query information into executable query description information, the business system can obtain the entity name contained in the query description information and retrieve the standardized entity ID corresponding to the entity name from the database based on the entity name. If the standardized entity ID corresponding to the entity name is retrieved from the database, the entity name contained in the query description information can be replaced with the standardized entity ID.

[0137] For example, in practical applications, the entity name contained in the above structured query description information can be converted into a vector based on the embedding model, and then matched with the query index built for the metadata of the indicator in the database through a vector retrieval engine to find the standardized entity ID that matches the entity name. Then, the non-standard colloquial entity name can be converted into a standardized entity ID in the database.

[0138] The specific process of replacing the entity name contained in the query description information with the standardized entity ID is described in detail below through a specific example.

[0139] See Figure 6 , assuming that the above structured query description information is as follows Figure 4 As shown, Figure 4 The entity names "population", "per capita GDP", "last year", "province", "Zhejiang", and "city" are usually extracted from the user's colloquial questions and may not be the standardized entity IDs in the database. Assuming that after searching, it is determined that the standardized entity IDs corresponding to the above entity names in the database are "population", "per_capita_gdp", "year", "province", "zhejiang", and "city", the business system can Figure 4The entity names "population", "per capita GDP", "last year", "province", "Zhejiang", and "city" are replaced with "population", "per_capita_gdp", "year", "province", "zhejiang", and "city" respectively. Figure 4 The structured query description information shown is converted into query description information that conforms to the data query specification of the database. The query description information after conversion can be Figure 6 shown.

[0140] In one embodiment shown, the above-mentioned structured query description information may generally also include some external call semantics that have not yet been executed. Since these external call semantics are external call semantics that have not yet been executed, the structured query description information may not be directly executed in the database.

[0141] In this case, the business system can parse the query description information in the process of converting the structured query information into executable query description information to determine whether the query semantics of the query description information contain external call semantics; if the query semantics of the query description information contain external call semantics, the business system can execute the external call related to the external call semantics, and then add the external call result to the query description information.

[0142] The form of the external call corresponding to the above-mentioned external call semantics is not specifically limited in this specification. In practical applications, it can cover any form of call to external tools or external services.

[0143] For example, in one example, the above-mentioned external call semantics can specifically be the call of a function of a tool class outside the business system; in this case, if the query semantics of the above-mentioned structured query description information include the call semantics of a function of a tool class outside the business system, the business system can parse the string related to the function call, extract the function name and input parameters, and then execute the call to the function based on the function name and input parameters, and then add the call result to the query description information.

[0144] Since the service LLM is usually unable to directly execute the external call semantics contained in the query semantics of the converted structured query description information, this method is equivalent to externally enhancing the model capabilities of the service LLM.

[0145] In one embodiment shown, the structured query description information may generally include some query semantics that cannot be directly executed, which may cause the structured query description information to be unable to be directly executed in the database.

[0146] In this case, the business system can pre-perform executable verification on the query semantics contained in the query description information during the process of converting the structured query information into executable query description information; if the query semantics contained in the query description information have passed the executable verification, a query can be further initiated for the target indicators stored in the database based on the query description information.

[0147] The query semantics that cannot be directly executed contained in the structured query description information may generally include any form of query semantics that do not conform to the database data query specification, and will not be specifically limited in this specification.

[0148] In one embodiment, the query semantics contained in the structured query description information may involve initiating queries for multiple indicators simultaneously. Under such query semantics, if the multiple indicators do not have the same query dimension, the structured query description information may not be directly executed in the database.

[0149] In this case, in the process of converting the structured query information into executable query description information, the business system can determine whether there are multiple indicators in the query semantics of the query description information that need to be queried simultaneously; if there are multiple indicators in the query semantics of the query description information that need to be queried simultaneously, it can be further verified whether the multiple indicators have the same query dimension; if so, it can be further determined that the query semantics of the query description information have passed the executable verification.

[0150] For example, if the metrics to be queried simultaneously are the GDP and population height indicators of a province, the business system can verify whether the GDP and population height indicators have the same query dimension "Zhejiang Province". If so, the two queries for Zhejiang Province's GDP and population height can be executed simultaneously and completed in one go. Conversely, if the query dimension for the GDP indicator is "Zhejiang Province" and the query dimension for the population height indicator is "Jiangsu Province", the two queries for Zhejiang Province's GDP and Jiangsu Province's population height cannot usually be executed in one go and need to be run in two separate queries.

[0151] Please continue to see Figure 5 After the business system converts the structured query description information into executable query description information, in order to avoid the problem of missing information in the query description information, the structured query description information can also be supplemented.

[0152] Completing the structured query description information refers to completing other necessary information related to the indicator query.

[0153] In one embodiment shown, the parameters included in the structured query description information may specifically include query parameters for initiating a query, and visual display parameters for visually displaying the queried parameters; in this case, the missing parameters in the query parameters and the visual display parameters can be completed separately.

[0154] For example, the missing parameters in the query parameters above may be query conditions (such as the query time range). In this case, the query conditions can be completed manually or based on the default query conditions of the business system. The missing parameters in the visualization display parameters above may be parameters related to visualization rendering, such as the specific meaning of the horizontal and vertical axes of an icon. In this case, these parameters can be completed manually or based on the default visualization rendering parameters of the business system.

[0155] Please continue to see Figure 5 When the business system completes the above-mentioned structured query description information and obtains complete parameters related to the indicator query, it can now query the target indicator stored in the database based on the above-mentioned structured query description information; wherein, the target indicator can specifically refer to the indicator specified by the user in the user question input.

[0156] For example, suppose a user enters a question like, "Which cities in Zhejiang Province had a population of over 5 million and a per capita GDP of over 100,000 yuan last year? Sort by population from largest to smallest?" The business system, through semantic analysis, can determine that the user's target indicators are "population" and "per capita GDP." Both "population" and "per capita GDP" share the same query dimension, "Cities in Zhejiang Province," and can be queried simultaneously.

[0157] In one embodiment shown, the parameters included in the structured query description information may specifically include query parameters for initiating a query, and visualization parameters for visually displaying the queried parameters; in this case, when the business system queries the target indicators stored in the database based on the above-mentioned structured query description information, on the one hand, it can query the target indicators stored in the database based on the query parameters included in the query description information; on the other hand, it can further perform a visualization display of the queried target indicators based on the visualization parameters included in the query description information.

[0158] For example, assuming the user inputs a question like "Which cities in Zhejiang Province had a population of more than 5 million and a per capita GDP of more than 100,000 yuan last year? Sorted by population from largest to smallest?", the above business system can then execute Figure 4 The structured query description information shown takes "cities in Zhejiang Province" as the query dimension (i.e. Figure 4 ), set "Zhejiang Province has a population of more than 5 million and a per capita GDP" as the query condition to initiate a query, and then use the "sort by population from large to small" visualization method to display the cities that meet the query condition. Among them, it needs to be explained that Figure 4 In the structured query description information shown, gt means greater than, eq means equal to, and desc means sorting in reverse order, that is, from largest to smallest.

[0159] Corresponding to the embodiments of the aforementioned method, this specification also provides embodiments of an apparatus, an electronic device, and a storage medium.

[0160] Figure 7 This is a schematic structural diagram of an electronic device provided by an exemplary embodiment. Figure 7 At the hardware level, the device includes a processor 702, an internal bus 704, a network interface 706, a memory 708, and a non-volatile memory 710, and may also include other required hardware. One or more embodiments of this specification can be implemented based on software, such as the processor 702 reading the corresponding computer program from the non-volatile memory 710 into the memory 708 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0161] like Figure 8 As shown, Figure 8 This is a block diagram of an LLM-based index query device according to an exemplary embodiment of the present specification. The device can be operated in Figure 7 In the electronic device shown in FIG. 1 , the technical solution of this specification is implemented. The device 80 includes:

[0162] Acquisition module 801 acquires metadata of the indicator to be queried and constructs a training sample set for fine-tuning the basic LLM based on the acquired metadata; wherein the training sample includes a structured query description information sample constructed based on the metadata; and a user question sample that matches the query semantics of the query description information sample;

[0163] A training module 802 fine-tunes the basic LLM based on the training sample set to obtain a service LLM; wherein the service LLM is used to convert the user question input by the user into the structured query description information that matches the query semantics of the user question;

[0164] The query module 803 converts the user question input by the user for initiating the query into the structured query description information that matches the query semantics of the user question based on the service LLM, and queries the target indicator stored in the database based on the query description information; wherein, the target indicator is the indicator specified by the user in the user question.

[0165] Accordingly, this specification also provides an electronic device, which includes a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the steps in all the method flows described above.

[0166] Accordingly, this specification also provides a computer-readable storage medium on which executable computer program instructions are stored; wherein, when the instructions are executed by a processor, the steps in all the method flows described above are implemented.

[0167] Accordingly, this specification also provides a computer program product having executable computer program instructions stored thereon; wherein, when the computer program instructions are executed by a processor, the steps in all the method flows described above are implemented.

[0168] In the 1990s, technological improvements could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures like diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD through their own programming, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0169] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules that implement the method and structures within the hardware component.

[0170] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a server system. Of course, this application does not exclude that with the future development of computer technology, the computer that implements the functions of the above embodiments may be, for example, a personal computer, a laptop computer, an in-vehicle human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0171] Although one or more embodiments of this specification provide method operation steps as described in the embodiments or flow charts, more or fewer operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the device or terminal product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The term "comprise", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or equipment including a series of elements includes not only those elements, but also includes other elements that are not clearly listed, or also includes elements inherent to such process, method, product or equipment. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or equipment including the elements. For example, if the words first, second, etc. are used to represent the name, they do not represent any particular order.

[0172] For the convenience of description, the above devices are described in terms of functions divided into various modules. Of course, when implementing one or more of the present specifications, the functions of each module can be implemented in the same or multiple software and / or hardware, or the module that implements the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0173] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0174] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0175] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0176] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0177] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0178] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage, graphene storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0179] Those skilled in the art will appreciate that one or more embodiments of this specification may be provided as a method, system, or computer program product. Thus, one or more embodiments of this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0180] One or more embodiments of this specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. One or more embodiments of this specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0181] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between the various embodiments can be referenced across them. Each embodiment focuses on the differences from the other embodiments. In particular, since the system embodiments are generally similar to the method embodiments, their description is relatively simple. For relevant parts, reference can be made to the description of the method embodiments. Throughout this specification, reference to the terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and integrate the different embodiments or examples, and features of different embodiments or examples, described in this specification, without conflict.

[0182] The foregoing is merely an example of one or more embodiments of this specification and is not intended to limit the one or more embodiments of this specification. It will be apparent to those skilled in the art that various modifications and variations may be made to one or more embodiments of this specification. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this specification shall be included within the scope of the claims.

Claims

1. An LLM-based indicator query method, comprising: Obtain metadata of the indicator to be queried, and construct a training sample set for fine-tuning the basic LLM based on the obtained metadata; wherein the training sample includes a structured query description information sample constructed based on the metadata; and a user question sample that matches the query semantics of the query description information sample; Fine-tuning the basic LLM based on the training sample set to obtain a service LLM; wherein the service LLM is used to convert the user question input by the user into the structured query description information that matches the query semantics of the user question; Based on the service LLM, the user question input by the user for initiating a query is converted into the structured query description information that matches the query semantics of the user question, and based on the query description information, a query is performed on the target indicator stored in the database; wherein, the target indicator is the indicator specified by the user in the user question. 2 . The method according to claim 1 , wherein the query description information comprises query description information based on a domain-specific language.

3. The method of claim 1, constructing a training sample set for fine-tuning the basic LLM based on the acquired metadata, comprising: Obtaining a preset query description information template set; wherein the query description information templates in the query description information template set correspond to different query semantics; and the query description information templates include a number of query parameters; A state machine for automatically generating training samples is constructed based on the query description information template set; wherein the states in the state set maintained in the state machine correspond one-to-one to the query description information templates in the query description information template set; and the state parameters of the states are consistent with the query parameters contained in the query description information template corresponding to the states; Filling the values of the state parameters of the states in the state set based on the metadata of the indicator to be queried to construct a query description information sample corresponding to the state, and generating a user question sample that matches the query semantics of the query description information sample; A training sample is generated based on the query description information sample and the user question sample, and a training sample set is constructed based on the generated training sample.

4. The method according to claim 3, wherein filling the values of the state parameters of the states in the state set based on the metadata of the indicator to be queried comprises: randomly selecting an indicator from the indicators to be queried, and further randomly selecting metadata from the metadata of the randomly selected indicator; The values of the state parameters of the states in the state set are filled based on the randomly selected metadata.

5. The method according to claim 4, wherein at least part of the states in the state set maintained in the state machine are defined with an execution action set; wherein, The execution action in the execution action set is used to modify the value of a specified state parameter in the state parameters of the state; Before constructing a training sample set based on the generated training samples, it also includes: Triggering the execution of an action in the execution action set corresponding to the state, modifying the value of a specified state parameter in the state parameters of the state to construct an extended sample corresponding to the query description information sample, and generating a user follow-up question sample that matches the query semantics of the extended sample; wherein the user follow-up question sample is a question sample generated by the user asking a follow-up question based on the user question sample; A training sample is generated based on the extended sample and the user question sample.

6. The method according to claim 5, wherein the actions in the execution action set each have a preset probability of triggering execution; Triggering the execution of actions in the execution action set corresponding to the state, including: Based on the probabilities of triggering execution respectively preset for the actions in the execution action set, the actions in the execution action set corresponding to the state are triggered to execute.

7. The method according to claim 5, wherein the user question sample and the user follow-up question sample are samples generated based on a preset user question template; Before fine-tuning the basic LLM based on the training sample set, the method further includes: Constructing prompt words corresponding to training samples in the training sample set; wherein the prompt words are used to instruct the basic LLM to convert the user question text or user follow-up question sample contained in the training sample into a colloquial user question sample or user follow-up question sample that matches the query semantics of the query description information sample contained in the training sample; The prompt word is input into the basic LLM, so that the basic LLM performs inference calculation based on the prompt word, and converts the user question text or user follow-up question sample contained in the training sample into a colloquial user question sample or user follow-up question sample that matches the query semantics of the query description information sample contained in the training sample.

8. The method according to claim 1, before querying the target indicator stored in the database based on the query description information, further comprising: Obtaining the entity name contained in the query description information; Retrieving a standardized entity ID corresponding to the entity name from a database based on the entity name; If a standardized entity ID corresponding to the entity name is retrieved from the database, the entity name included in the query description information is replaced with the entity ID.

9. The method according to claim 8, before querying the target indicator stored in the database based on the query description information, further comprising: Parsing the query description information to determine whether the query semantics of the query description information include external call semantics; If the query semantics of the query description information include external call semantics, an external call related to the external call semantics is executed, and the external call result is added to the query description information.

10. The method according to claim 9, before querying the target indicator stored in the database based on the query description information, further comprising: Performing executable verification on the query semantics of the query description information; If the query semantics of the query description information passes the executable verification, a query for the target indicator stored in the database is further initiated based on the query description information.

11. The method according to claim 10, wherein the step of performing executable verification on the query semantics of the query description information comprises: Determining whether there are multiple indicators in the query semantics of the query description information that require simultaneous query initiation; If the query semantics of the query description information include multiple indicators that need to be queried simultaneously, verify whether the multiple indicators have the same query dimension; If yes, it is determined that the query semantics of the query description information passes the executable verification.

12. The method according to claim 1, wherein the query description information includes query parameters for initiating the query and visualization parameters for visually displaying the queried parameters; Querying the target indicator stored in the database based on the query description information includes: A query is performed on the target indicator stored in the database based on the query parameters included in the query description information, and a visual display is performed on the queried target indicator based on the visual display parameters.

13. An indicator query device based on LLM, comprising: An acquisition module acquires metadata of the indicator to be queried and constructs a training sample set for fine-tuning the basic LLM based on the acquired metadata; wherein the training sample includes a structured query description information sample constructed based on the metadata; and a user question sample that matches the query semantics of the query description information sample; A training module, which fine-tunes the basic LLM based on the training sample set to obtain a service LLM; wherein the service LLM is used to convert the user question input by the user into the structured query description information that matches the query semantics of the user question; The query module converts the user question input by the user for initiating the query into the structured query description information that matches the query semantics of the user question based on the service LLM, and queries the target indicator stored in the database based on the query description information; wherein the target indicator is the indicator specified by the user in the user question.

14. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 12 are implemented.

15. A computer program product comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

Images