Large model training method and data query method based on large model

Abstract

The present application relates to a large model training method and a data query method based on the large model, which aims to enhance the understanding and recognition ability of the large language model for complex, uncommon or user-infrequently queried basic fields in a specific data warehouse environment. By combining the constellation data model of the data warehouse, this method can accurately locate and strengthen the basic fields of the training target, ensuring that the model has a higher understanding ability in these key areas. During the training process, by generating a variety of second-natural query statements, the training data set is enriched, so that the model can learn more about the correlation between query intent and fields, and enhance the model's reasoning and generalization capabilities. When faced with diverse query requirements, the model can maintain a high degree of accuracy, and can also quickly respond and accurately identify complex column names that are difficult to handle with traditional methods, thereby providing users with a more efficient and convenient data query experience.

Description

Technical Field

[0001] The present application relates to the field of data query technology, and in particular to a large model training method and a data query method based on the large model. Background Art

[0002] With the advent of the big data era, data warehouses have become an important tool for enterprises to manage and analyze data. However, how to efficiently and accurately extract the information required by users from massive amounts of data has become an urgent problem to be solved. In the prior art, with the help of the natural language understanding ability of large language models, the function of converting users' natural language query sentences into data query sentences is realized, which greatly facilitates users' data query operations. However, this solution still faces many challenges in practical applications, especially when dealing with complex and diverse column names in a specific data warehouse.

[0003] Specifically, the column names in the data warehouse may include uncommon column names, column names that are not frequently queried by users, column names with diverse expressions, and specific column names that can be flexibly customized and created in specific fields. The diversity and complexity of these column names make it difficult for large language models to identify and understand them, resulting in inaccurate generated query statements, which in turn affects query results. In addition, since the constellation data models of different data warehouses are different, and the data tables and basic fields they contain are also different, existing models are often difficult to directly apply to all scenarios, lacking pertinence and flexibility, and often resulting in various recognition errors in different scenarios. Summary of the invention

[0004] Based on this, the purpose of this application is to provide a data query large model training method, which aims to improve the large language model's understanding and recognition capabilities of specific basic fields in the data warehouse through targeted intensive training.

[0005] The data query large model training method provided in the embodiment of the present application includes the following steps:

[0006] Obtaining a number of first natural query statements;

[0007] If any of the first natural query statements contains a target basic field, generate a plurality of second natural query statements having the same query intent according to the first natural query statement; wherein the second natural query statement does not contain the target basic field in the first natural query statement; the target basic field is determined from a constellation data model corresponding to a target data warehouse; the constellation data model records a plurality of data tables and basic fields contained in each data table;

[0008] Input each of the second natural query statements, the preset first task execution text, and the first query label information corresponding to each of the second natural query statements into a text processing model for training; wherein the first query label information includes the target basic field contained in the first natural query statement corresponding to the second natural query statement; the first task execution text is used to prompt the text processing model to output query information that meets the query intent of the second natural query statement, and the query information includes the basic field;

[0009] The trained text processing model is determined as the data query model corresponding to the target data warehouse.

[0010] The present application also provides a method for querying a large data model based on the model training method of any one of the embodiments of the present application, comprising the following steps:

[0011] Obtaining natural query statements input by users in the query interface of the target data warehouse;

[0012] Acquire a constellation data model corresponding to the target data warehouse; determine several relevant basic fields from the constellation data model as candidate basic fields according to the natural query statement;

[0013] Constructing a task processing instruction according to the candidate basic fields, the natural query statement and a preset query task execution text; wherein the query task execution text is used to prompt the data query big model to output query information that meets the query intent of the natural query statement, and the query information is determined according to the candidate basic fields;

[0014] Inputting the task processing instruction into the data query big model corresponding to the target data warehouse to obtain output query information;

[0015] According to the query information and the constellation data model, corresponding business data is queried from the target data warehouse.

[0016] The data query large model training method proposed in this application significantly improves the large language model's ability to understand and recognize complex, uncommon, or infrequently queried basic fields in a specific data warehouse environment through an innovative reinforcement training strategy. This method can accurately locate and strengthen the basic fields of the training target by combining the constellation data model of the data warehouse, ensuring that the model has a higher level of understanding in these key areas. During the training process, the training data set is enriched by generating a variety of second-natural query statements, allowing the model to learn more about the correlation between query intent and fields.

[0017] During the training process, the model is guided to focus on inferring missing target basic fields from natural query statements through preset task execution text and query label information. This targeted training method greatly enhances the model's reasoning and generalization capabilities. After intensive training, the resulting data query model can not only maintain a high degree of accuracy when facing diverse query requirements, but also quickly respond and accurately identify complex column names that are difficult to handle with traditional methods, thereby providing users with a more efficient and convenient data query experience.

[0018] In general, the technical solution of this application optimizes the training process and improves model performance. The trained data query model is highly customized and designed for specific data warehouses. It can efficiently and accurately understand users' natural query sentences, and achieves precise understanding and identification of target basic fields in the data warehouse, which significantly improves the accuracy and efficiency of data query and provides strong support for enterprise decision support and data analysis.

[0019] For better understanding and implementation, the present application is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flowchart of a data query large model training method according to an embodiment of the present application;

[0021] Figure 2 A schematic diagram of the steps for determining the target basic fields of the target data warehouse in an embodiment of the present application;

[0022] Figure 3 A schematic diagram of the steps of generating a second natural query statement by using an intelligent language model in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of the steps of constructing training samples to train a text processing model in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of the steps of updating and training a text processing model according to an embodiment of the present application;

[0025] Figure 6 A flowchart of a data query method based on a data query big model according to another embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present application clearer, the embodiments of the present application will be further described in detail below in conjunction with the accompanying drawings. Wherein, when the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements.

[0027] It should be clear that the implementation methods described in the embodiments described below do not represent all implementation methods consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0028] The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application are also intended to include plural forms, unless the context clearly indicates other meanings. In addition, in the description of this application, unless otherwise specified, "multiple" refers to two or more. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items, for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone; the character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0029] It should be understood that, although the terms first, second, third, etc. may be used in the present application to describe various information, such information should not be limited to these terms, and these terms are only used to distinguish similar objects, and do not have to be used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. Depending on the context, the words "if" / "if" used in the present application can be interpreted as "at the time of" or "when" or "in response to determination".

[0030] With the advent of the big data era, data warehouses have become an important tool for enterprises to manage and analyze data. However, how to efficiently and accurately extract the information required by users from massive amounts of data has become an urgent problem to be solved. In the prior art, with the help of the natural language understanding ability of large language models, the function of converting users' natural language query sentences into data query sentences is realized, which greatly facilitates users' data query operations. However, this solution still faces many challenges in practical applications, especially when dealing with complex and diverse column names in a specific data warehouse.

[0031] Specifically, the column names in the data warehouse may include uncommon column names, column names that are not frequently queried by users, column names with diverse expressions, and specific column names that can be flexibly customized and created in specific fields. The diversity and complexity of these column names make it difficult for large language models to identify and understand them, resulting in inaccurate generated query statements, which in turn affects query results. In addition, since the constellation data models of different data warehouses are different, and the data tables and basic fields they contain are also different, existing models are often difficult to directly apply to all scenarios, lacking pertinence and flexibility, and often resulting in various recognition errors in different scenarios.

[0032] In this regard, an embodiment of the present application provides a data query large model training method, which aims to improve the large language model's understanding and recognition capabilities of specific basic fields in the data warehouse through targeted intensive training.

[0033] Please refer to Figure 1 The data query large model training method provided in the embodiment of the present application includes the following steps:

[0034] S101: Obtaining a number of first natural query statements;

[0035] S102: if any of the first natural query statements contains a target basic field, generate a plurality of second natural query statements having the same query intent according to the first natural query statement; wherein the second natural query statement does not contain the target basic field in the first natural query statement; the target basic field is determined from a constellation data model corresponding to a target data warehouse; the constellation data model records a plurality of data tables and basic fields contained in each data table;

[0036] S103: Input each of the second natural query statements, the preset first task execution text, and the first query label information corresponding to each of the second natural query statements into a text processing model for training; wherein the first query label information includes the target basic field contained in the first natural query statement corresponding to the second natural query statement; the first task execution text is used to prompt the text processing model to output query information that meets the query intent of the second natural query statement, and the query information includes the basic field;

[0037] S104: Determine the trained text processing model as the data query macro model corresponding to the target data warehouse.

[0038] The data query large model training method proposed in this application significantly improves the understanding and recognition ability of the large language model for complex, uncommon or user-infrequently queried basic fields in a specific data warehouse environment through an innovative reinforcement training strategy. This method can accurately locate and strengthen the basic fields of the training target by combining the constellation data model of the data warehouse, ensuring that the model has a higher understanding ability in these key areas. During the training process, the training data set is enriched by generating a variety of second natural query statements, so that the model can learn more about the correlation between query intent and fields. During the training process, through the preset task execution text and query label information, the model is guided to focus on inferring the missing target basic fields from the natural query statements. This targeted training method greatly enhances the reasoning and generalization capabilities of the model. After intensive training, the obtained data query large model can not only maintain a high degree of accuracy in the face of diverse query requirements, but also quickly respond and accurately identify those complex column names that are difficult to handle with traditional methods, thereby providing users with a more efficient and convenient data query experience. In general, the technical solution of this application optimizes the training process and improves model performance. The trained data query model is highly customized and designed for specific data warehouses. It can efficiently and accurately understand users' natural query sentences, and achieves precise understanding and identification of target basic fields in the data warehouse, which significantly improves the accuracy and efficiency of data query and provides strong support for enterprise decision support and data analysis.

[0039] It should be noted that the data query big model training method of the embodiment of the present application mainly trains the improved model's understanding and recognition capabilities of basic fields, but it does not mean that the data query big model can only be used to understand and process natural query statements that only query basic fields. In fact, the improvement of the recognition ability of basic fields will have a significant effect on the model's understanding and processing of various queries, including queries of complex multi-dimensional operation fields, as long as the basic fields are involved, and can improve the accuracy of the data query big model in understanding query intent, as well as improve the comprehensiveness and accuracy of the query information generated by the model for various query intents.

[0040] The data query big model training method proposed in the embodiment of the present application is executed by a computer. In the embodiment of the present application, the data query big model is specially customized for the "constellation data model" (i.e., the data model system of the data warehouse built based on a specific field or business logic) to better adapt to the complex query requirements in different fields and scenarios, and improve the accuracy and efficiency of the query.

[0041] The following first explains the text processing model and task execution text, and then explains each step in detail.

[0042] In the embodiments of the present application, the text processing model is a natural language processing (NLP) model that uses machine learning or deep learning technology to process, understand and generate text data. The model can learn the grammatical rules, semantic relationships and contextual dependencies of the language through training with a large amount of text data, thereby achieving various tasks such as understanding, extracting, classifying and / or generating text information. The text processing model can be a pre-trained language model, or it can be a model obtained by fine-tuning a pre-trained model (such as BERT, GPT, etc.) to adapt to the specific tasks in this application (i.e., query intent understanding and query information generation) (of course, the latter can still be considered as a pre-trained model in the scenario of this application). In short, the text processing model in this application has been preliminarily trained on text data, so it has a certain language understanding and generation ability, and can quickly adapt to and improve performance on new tasks through a small amount of annotated data.

[0043] In a data query scenario of an embodiment of the present application, a text processing model is a natural language processing model that can parse natural query statements input by users, understand their query intent, and generate corresponding query information based on these intents. The text processing model can be based on one or more pre-trained language models, which are fine-tuned to adapt to the specific tasks in this application. This usually involves further training the pre-trained language model on a labeled data set containing query statements and corresponding query information so that it can better understand the query intent and generate accurate query information. Of course, the text processing model can also be an existing large language model, as long as it has the processing capability of the specific tasks of the data query scenario of the embodiment of the present application, it can be applied in the technical solution of the present application, and the embodiment of the present application is not limited to this.

[0044] It is understandable that in the embodiments of the present application, the text processing model needs to be further optimized. It should be said that the existing natural language models are all trained using text data, but due to the limited amount of text data, the ambiguity and inaccuracy of the text corpus, and the large model hallucination problem, the existing models always have certain limitations. Although they have certain recognition capabilities, they cannot achieve universal adaptability to various fields, various scenarios, and various natural language expressions. The accuracy of their understanding is insufficient in certain fields or scenarios. Therefore, the embodiments of the present application are intended to specifically train a dedicated model for the target data warehouse, train the corresponding data query large model for data warehouses in different fields and scenarios, and realize the specialization and precision of the data query large model. Especially in the embodiments of the present application, although the text processing model itself has undergone certain data training and has the ability to understand the query intent of natural query statements and generate query information, it is obviously insufficient in recognizing column names when understanding query statements involving uncommon column names, column names that are not often queried by users, column names with diverse expressions, and specific column names that can be flexibly customized and created in specific fields. Therefore, the data query large model training method provided by the present application is mainly used to further enhance the model's understanding and recognition capabilities of special column names. Of course, the technical concept of the present application is not limited to this, and please refer to the description of each embodiment of the present application for details.

[0045] Furthermore, a task execution text is involved in the application of the text processing model. The task execution text is used to require or prompt the task requirements of the text processing model, including specific requirements and formatting instructions for the model output, and may also include task-related scene information or context information for the model to learn and understand the task to be processed, etc. In an embodiment of the present application, different sub-texts can be set in the task execution text according to the needs of the query, respectively used to require or prompt the corresponding task requirements of the text processing model. In one embodiment, the task execution text may include a first task execution text, a second task execution text, a third task execution text... and so on, and different sub-texts contain specific requirements for model processing and output according to the task requirements, etc.

[0046] In one embodiment of the present application, the task execution text includes a first task execution text; the first task execution text is used to prompt the text processing model to output query information that meets the query intent of the second natural query statement, and the query information includes a query field. Among them, the first task execution text is generally a command text used to require or prompt the text processing model what task to perform, which details the specific rules that the text processing model should follow when generating query information. Alternatively, the first task execution text can also be an instruction set in a specific format, which also details the specific rules that the text processing model should follow when generating query information. The first task execution text can specifically determine the form to be adopted based on the requirements and specifications of the text processing model for task processing instructions. In addition, other task execution texts such as the second task execution text and the third task execution text are understood by analogy based on this principle.

[0047] It should be noted that those skilled in the art understand that what the task execution text prompt text processing model should do does not mean what the text processing model must do. For example, the task execution text prompt text processing model outputs query information that meets the query intent, wherein the query information includes a multidimensional operation field, but after the text processing model understands the natural query statement, it believes that this query does not involve a multidimensional operation field, then it may not output the multidimensional operation field to avoid violating the core task of the task execution text prompt, that is, outputting query information that meets the query intent. Of course, if the training sample involves a query of a multidimensional operation field, but the text processing model does not output the multidimensional operation field, it means that the task processing capability of the model needs to be improved, and this is also the significance of the technical solution of the embodiment of the present application.

[0048] In step S101, a number of first natural query statements are obtained.

[0049] In this step, a certain number of natural query statements can be collected from actual application scenarios or user logs. These statements are query requests to the data warehouse made by users in natural language. For example, they can be query statements collected from the enterprise's internal data analysis platform, customer support system, or user feedback, such as "query the sales of all products last month" or "list the names and departments of new employees who joined this year."

[0050] In one embodiment, the step of obtaining a plurality of first natural query statements in step S101 includes:

[0051] Step S1011, obtaining a historical natural query statement of the data warehouse, which is a first natural query statement;

[0052] And / or, step S1012, obtain each basic field from the constellation data model corresponding to the target data warehouse, and generate a task command according to each basic field and preset task prompt instructions; send the task command to the intelligent language model to obtain a number of natural query statements corresponding to each basic field, which is a first natural query statement; wherein the task prompt instruction shown is used to prompt the intelligent language model to generate a natural query statement related to the basic field.

[0053] Among them, the intelligent language model, in the embodiment of the present application, is different from the text processing model. It is a large language model with more powerful processing capabilities, ultra-large-scale parameters and trained with ultra-large-scale data. It is usually the most powerful large language model recognized in the field of natural language processing in the current period, such as the current ChatGPT. It has very powerful natural language understanding and text processing capabilities, and can support the implementation of the technical solution of the present application-that is, this embodiment uses an external intelligent language model that is mature in text task execution to perform related text generation tasks, thereby obtaining the required text, including generating a first natural query statement based on the basic field, and generating a second natural query statement based on the first natural query statement, etc. The ability of the local text processing model (data query large model) is improved by training with the help of an external stronger large language model.

[0054] This embodiment provides two methods for obtaining the first natural query statement for step S101, aiming to cover query scenarios with different sources and requirements. Specifically, either method or both methods may be used.

[0055] For step S1011, by directly accessing the historical records of the data warehouse, natural query statements previously executed by users or the system can be obtained. These query statements are generated based on actual data retrieval requirements and therefore have a high reference value. Using them as part of the first natural query statement can ensure that the query optimization process of the model by the technical solution of this application can cover the common query patterns and requirements of users.

[0056] For step S1012, considering that the data warehouse may contain complex data structures and a large number of basic fields, in order to dig out more potential query needs, this step innovatively combines the constellation data model (a data modeling method for describing complex data relationships, see the subsequent content for details) and the intelligent language model. Specifically, first extract the various basic fields from the constellation data model, which represent the most basic query units in the data warehouse. Subsequently, based on these basic fields and preset task prompt instructions, construct and send task commands to an intelligent language model, such as ChatGPT. Based on its powerful natural language understanding and generation capabilities, the intelligent language model can generate multiple natural query statements related to each basic field. This approach not only expands the source of query statements, but also improves the diversity and pertinence of query statements, which helps to discover more potential and valuable query patterns.

[0057] By combining the above two methods, this embodiment shows significant advantages in obtaining the first natural query statement. On the one hand, by utilizing the historical query records of the data warehouse, the valuable information of the user's actual query is retained, providing a solid foundation for subsequent query optimization. On the other hand, by introducing the constellation data model and the intelligent language model, a large number of natural query statements with potential value can be automatically generated based on the structure and semantics of the data warehouse, which greatly enriches the diversity of query statements and helps to discover new query patterns and needs. This comprehensive approach not only improves the efficiency and accuracy of the query optimization process.

[0058] For step S102, if any of the first natural query statements contains a target basic field, several second natural query statements with the same query intent are generated according to the first natural query statement; wherein the second natural query statement does not contain the target basic field in the first natural query statement; the target basic field is determined from the constellation data model corresponding to the target data warehouse; the constellation data model records several data tables and the basic fields contained in each data table.

[0059] In this step, the target basic fields that need to be intensively trained are determined according to the constellation data model of the target data warehouse. Then, the collected first natural query statements are checked to identify the statements containing the target basic fields. This process can be determined by parsing the query statements and the field recognition algorithm, or by character matching based on the preset target basic fields. For the first natural query statement containing the target basic field, multiple second natural query statements with the same query intent but not containing the target basic field can be generated using natural language processing technology or manual editing. For example, if the first query statement is "query the sales of all products last month", and "sales" is the target basic field, then a second query statement such as "view last month's sales data" or "list the sales of each product last month" can be generated.

[0060] The target data warehouse is a data warehouse selected by the user. In the embodiment of the present application, the user can select one or more data warehouses in the data warehouse list, so as to train the data query model of the target data warehouse in a targeted manner based on the constellation data model of the selected target data warehouse.

[0061] Among them, the constellation data model is an abstract representation used to describe the database structure. It is a data logic layer established in advance based on various data tables stored in the data warehouse. It records several data tables, the basic fields contained in each data table, and the graph relationship of each data table. In addition, it also records the binding relationship between at least some basic fields and multi-dimensional operation fields. In other words, the constellation data model does not record the specific business data of the fields of each data table, but only records the relevant information used to generate query statements. This information mainly includes the table name of the data table, the field information of the data table, and the multi-dimensional operation field information bound to the basic field. In addition, the graph relationship between each data table is established in the constellation data model, and the graph relationship at least includes a directed graph relationship.

[0062] Among them, the basic field is generally a field that records the original business data in the database. More specifically, the basic field is generally divided into a metric field and a dimension field. The metric field generally refers to the field in the fact table, and the dimension field generally refers to the field in the dimension table. Among them, the fact table and the dimension table are concepts in data modeling in the field of data warehouses. At present, the main data models involved in the field of data warehouses are star data models, snowflake data models generated on the basis of star data models (which can actually be considered as a type of star data model) and constellation data models. In the star data model, it includes a data table located in the center and other data tables connected to the center data table, wherein the center data table is the fact table, and the other data tables connected to it are dimension tables.

[0063] Among them, the multidimensional operation field, also known as the MDX operation field, is different from the basic field. It is not a native field recorded in the database, but a field obtained by definition. The data queried by it is obtained by performing complex operations (such as complex calculations) based on the data of the basic field in the database. Specifically, multidimensional operation fields for various complex operations or various data analysis indicators can be defined according to the needs of data analysis. In one embodiment, the multidimensional operation field can be "same period", "previous period", "year-on-year", "month-on-month", "global proportion", "group proportion", "global ranking" and "group ranking", etc. Obviously, the query of the multidimensional operation field involves the calculation or comparison of the data of the basic field in multiple dimensions. Unlike the query of the basic field, it only needs to obtain the data corresponding to the basic field from the database, but requires more complex analysis and processing involving multiple dimensions and / or multiple basic data. In addition, according to the needs of data analysis, users can also customize more multidimensional operation fields and corresponding calculation formulas. The query of these multidimensional operation fields often contains important information of business logic and data analysis, and is an indispensable part of deep data exploration, but it is also flexible and changeable in different industries and application scenarios, and is often difficult to be directly understood and recognized by existing large language models.

[0064] In the embodiment of the present application, at least some of the basic fields of the constellation data model are bound to multidimensional operation fields. This binding is generally established in advance by the user according to the query requirements, so that when the user enters natural language to query, the required data analysis results can be accurately and quickly queried. This binding mechanism allows any basic field to be associated with one or more multidimensional operation fields. Different multidimensional operation fields define different specific operations that can be performed on the business data represented by the basic fields to which they are bound, greatly enhancing the flexibility and dynamism of data processing.

[0065] Among them, the target basic fields are the basic fields (column names) in the constellation data model of the target data warehouse to be subjected to intensive training. Specifically, they may include uncommon column names, column names that are not frequently queried by users, column names with diverse expressions, and specific column names that can be flexibly customized and created in specific fields.

[0066] Please refer to Figure 2 In one embodiment, the data query large model training method further includes the steps of:

[0067] Step S201, obtaining historical query statement data of the target data warehouse;

[0068] Step S202, determining the basic field whose number of historical queries is greater than a preset number according to the historical query statement data;

[0069] Step S203, determining other basic fields in the constellation data model corresponding to the target data warehouse, except for the basic fields whose historical query times are greater than a preset number, as target basic fields.

[0070] This embodiment introduces a data screening strategy based on historical query data. The strategy aims to identify basic fields with high actual query frequency by users in the target data warehouse, distinguish these high-frequency fields from relatively low-frequency or insufficiently queried fields, and determine the insufficiently queried basic fields as target basic fields. Obviously, the target basic fields determined by this screening strategy may include basic fields that existed in the constellation data model in the past but were not frequently queried, and may also include basic fields newly added to the target data warehouse or newly created / named by the user (which will also appear in the constellation data model of the target data warehouse at the same time). First, step 201 obtains a series of historical query statement data by accessing the log system or query record database of the target data warehouse. These data record the query behavior of users on the data warehouse in the past period of time, and are an important basis for analyzing user query habits and needs. Next, step 202 uses data processing and analysis technology to parse and count the collected historical query statement data. Specifically, the basic fields involved in each query statement are identified, and the number of times each basic field appears in the historical query is counted. By comparing these times with a preset threshold (i.e., the preset number of times), it is possible to determine which basic fields are frequently queried by users, i.e., basic fields whose historical query times are greater than the preset number of times. After determining the basic fields for high-frequency queries, step 203 turns the focus to the constellation data model corresponding to the target data warehouse. According to the result of step 202, those basic fields that have not been determined as high-frequency queries, i.e., basic fields whose historical query times are not greater than the preset number of times, are screened out from the constellation data model and marked as target basic fields. These target basic fields represent data areas that users may not have fully explored or queried, and are the parts that require special attention in the subsequent training of the large data query model.

[0071] In summary, this embodiment provides more targeted training data for the data query big model by identifying and distinguishing the basic fields of high-frequency queries and low-frequency queries. Specifically, for low-frequency or insufficiently queried target basic fields, a special training process is performed in the model to improve the accuracy and efficiency of the model in future queries. This training strategy not only helps to improve the overall performance of the data query big model, but also provides more comprehensive and in-depth support for enterprise decision-making and data analysis.

[0072] In one embodiment, before the step of obtaining the historical query statement data of the target data warehouse in step 201, the following steps are also included:

[0073] Step 200: obtaining the user's selection operation information for the data warehouse list, and determining the target data warehouse according to the selection operation information.

[0074] This embodiment first captures the user's selection operation information for the data warehouse list. This list may be presented to the user in the form of a graphical user interface, which contains an overview or name of all data warehouses that the user has access to, and each data warehouse corresponds to a different data source, data structure or business scenario. The user selects one or more target data warehouses that he wants to operate or query from the list by clicking, checking or other interactive methods. These warehouses have been organized and configured according to the constellation data model. Once the user completes the selection operation, the system will receive the corresponding selection operation information. This information usually contains a unique identifier (such as ID, name, etc.) of the data warehouse selected by the user, so that the system can accurately identify the user's target data warehouse. Subsequently, the system will determine and record the target data warehouse specified by the user based on these selection operation information. This step is the basis for all subsequent operations (such as obtaining historical query statement data, training data query large models, etc.), because it ensures that all operations are performed on the data warehouse that the user really cares about.

[0075] In summary, this embodiment introduces additional steps to ensure that the user can accurately specify the target data warehouse and perform subsequent operations based on the specific constellation data model of the warehouse. This not only enhances the pertinence and effectiveness of model training, but also improves the flexibility and convenience of users in the actual query process.

[0076] Please refer to Figure 3 In one embodiment, if any of the first natural query statements contains a target basic field, the step of generating a plurality of second natural query statements having the same query intent according to the first natural query statement in step S102 includes:

[0077] Step S1021: if any of the first natural query statements contains a target basic field, construct a second task processing instruction according to the first natural query statement and a preset second task execution text; wherein the second task execution text is used to prompt the intelligent language model to replace the target basic field in the first natural query statement with other semantically matching words, so as to obtain a plurality of second natural query statements having the same query intent as the first natural query statement;

[0078] Step S1022: input the second task processing instruction into an intelligent language model to obtain a plurality of second natural query statements corresponding to the first natural query statement.

[0079] This embodiment proposes a method for generating a second natural query statement with the same query intent for a first natural query statement containing a target basic field. In the process of data query optimization or data query large model training, in order to enhance the generalization ability and robustness of the model, it is usually necessary to generate multiple natural query statements with the same query intent but different expressions as the original natural query statement. This helps the model better understand the core requirements of the query and establish connections between different natural query statements, thereby improving the accuracy and efficiency of the query.

[0080] In step S1021, when it is detected that a first natural query statement contains a target basic field, a second task processing instruction is first constructed according to the query statement and the preset second task execution text. The second task execution text here is a key guiding information, which tells the intelligent language model the specific task to be performed, that is, to replace the target basic field in the first natural query statement with other semantically matching words. This replacement needs to keep the overall intention of the query statement unchanged, but only changes the way or angle of expression. In this way, multiple second natural query statements with the same query intention as the original query statement but with different expressions can be obtained. After the second task processing instruction is constructed, it is input into the intelligent language model in step S1022. Based on its powerful natural language understanding and generation capabilities, the intelligent language model will parse the requirements in the instruction, and intelligently generate multiple second natural query statements that meet the requirements based on the context information of the first natural query statement and the target basic field. These generated query statements are not only semantically consistent with the original query statement, but also more diverse and flexible in expression.

[0081] In summary, through the technical solution of this embodiment, a large number of second natural query statements with the same query intent but different expressions as the original query statement can be effectively generated, especially avoiding the original target basic field from appearing in the second natural query statement, which not only enriches the diversity of query statements, improves the fuzziness of training samples, improves the training effect of the model, but also helps to improve the generalization ability and robustness of the model in practical applications. Specifically, when the model faces a new query statement, even if the expression of the statement is different from that in the training data, especially for the basic fields of the query, various different expressions are used, but because the model has learned a variety of expressions under the same query intent, it can still accurately understand and identify the basic fields and accurately understand the query intent. This is of great significance for improving the accuracy and efficiency of data queries, and also provides users with a more flexible and convenient query experience.

[0082] In one embodiment, the second task execution text is also used to prompt the intelligent language model, and the target basic field of the first natural query sentence is determined from the candidate basic fields;

[0083] The step of constructing a second task processing instruction according to the first natural query statement and the preset second task execution text in step S1021 includes:

[0084] Step S10211, acquiring each target basic field in the constellation data model corresponding to the target data warehouse to obtain candidate basic fields;

[0085] Step S10212: construct a second task processing instruction according to the first natural query statement, the candidate basic fields and the preset second task execution text.

[0086] Among them, step S10211, in order to determine which basic fields may be used as the target basic fields in the first natural query statement, this step obtains each target basic field from the constellation data model corresponding to the target data warehouse, and uses these basic fields as candidate basic fields. After obtaining the candidate basic fields, step S10212 combines the first natural query statement, the candidate basic fields and the preset second task execution text to jointly construct the second task processing instruction. In this process, the second task execution text not only plays a role in guiding the intelligent language model to replace the field, but also clearly points out that the target basic field should be determined from the candidate basic fields. This means that when the intelligent language model performs the replacement task, it will only consider those basic fields that are clearly listed as candidates, thereby reducing the blindness and uncertainty of the replacement; of course, this also gives the intelligent language model certain hints on the other hand, helping it to accurately lock the target basic field based on the candidate basic fields.

[0087] In summary, this embodiment introduces the concept of candidate basic fields and combines it with the second task execution text and the first natural query statement to construct the second task processing instruction, which significantly improves the efficiency and accuracy of the intelligent language model in processing query statements containing target basic fields. On the one hand, the introduction of candidate basic fields provides a clear replacement range for the intelligent language model, reducing unnecessary calculations and error rates; on the other hand, combined with the precise guidance of the second task execution text, the intelligent language model can more accurately understand the query intent and generate a second natural query statement that meets the requirements. This optimization not only helps to improve the accuracy and efficiency of data queries, but also helps to enhance the generalization ability and robustness of the large data query model, providing more reliable and efficient data support for practical applications.

[0088] In one embodiment, if any of the first natural query statements contains a target basic field, the step of generating a plurality of second natural query statements having the same query intent according to the first natural query statement in step S102 includes:

[0089] Step S1023: If any of the first natural query statements contains a target basic field, obtain a number of words that semantically match the target basic field, replace the target basic field in the first natural query statement with the semantically matching words, and obtain a number of second natural query statements that have the same query intent and correspond to the first natural query statement.

[0090] In this embodiment, for the first natural query statement containing the target basic field, another processing method is adopted to generate multiple second natural query statements with the same query intent. This method can replace the other method provided above; of course, it can also be combined with the above method to generate a richer second natural query statement. The core of this method is to use semantic matching technology to find words that are semantically close or related to the target basic field, and replace them one by one into the original query statement, thereby generating a series of second natural query statements that are consistent in query intent but have different expressions.

[0091] For step S1023, first, the system needs to identify the target basic field contained in the first natural query statement. Once the target basic field is determined, the system will start the semantic matching mechanism to search for words that are semantically matched or related to the target basic field from a preset vocabulary library or dynamically acquired semantic resources. These words may include synonyms, near-synonyms, related terms, etc., which may be slightly different in expression, but can convey the same query intent in the context of the query. Next, the system replaces these semantically matching words one by one to the target basic field position in the first natural query statement. Each replacement generates a new query statement, which is consistent with the original query statement in grammatical structure and query intent, but changes in specific expression. In this way, the system can quickly generate multiple second natural query statements with the same query intent but different expressions.

[0092] This embodiment introduces semantic matching and vocabulary replacement technology to generate multiple second natural query statements with the same query intent but different expressions, which not only improves the coverage of the query statement, but also enhances the robustness and fault tolerance of the query. Even if the basic fields in the original query statement are misspelled, unclearly expressed, or expressed in various ways, the trained data query model can still accurately identify them.

[0093] In one embodiment, the step of obtaining a plurality of words that semantically match the target basic field in step S1023 includes:

[0094] Step S10231, calculating a vector of the target basic field according to the first natural query statement and the target basic field;

[0095] Step S10232, obtaining words whose vector similarity with the target basic field is greater than a preset similarity threshold, as words that semantically match the target basic field.

[0096] This embodiment adopts a method based on vector similarity to obtain words that semantically match the target basic field. This method calculates the vector representation of the target basic field and compares it with a preset vocabulary library or dynamically obtained vocabulary vectors to find words that are semantically close or related to the target basic field.

[0097] Specifically, in step S10231, the first natural query statement is parsed and the target basic field is accurately identified therein, and the target basic field is converted into a vector representation in a high-dimensional space using word embedding technology (such as Word2Vec, BERT, etc.) in natural language processing (NLP). This vector not only contains the vocabulary information of the target basic field, but also implies its semantic information in a specific context. After obtaining the vector of the target basic field, step S10232 compares it with each vocabulary vector in a preset vocabulary library or a dynamically acquired vocabulary vector library to calculate the similarity between them. The calculation of similarity is usually based on measurement methods such as cosine similarity and Euclidean distance. The system sets a preset similarity threshold. Only when the similarity between the vocabulary vector and the target basic field vector is greater than this threshold, the vocabulary is considered to be semantically matched with the target basic field. Finally, all words whose similarity with the target basic field vector is greater than the preset threshold are screened out as vocabulary outputs that match the semantics of the target basic field. These words will be used in the subsequent query statement generation step to replace the target basic fields in the original query statement and generate multiple second natural query statements with the same query intent but different expressions.

[0098] In summary, this embodiment achieves accurate acquisition of semantic matching vocabulary for the target base field by introducing a semantic matching method based on vector similarity. This method not only takes into account the literal similarity between words, but also fully considers their semantic similarity in a specific context, thereby improving the accuracy and effectiveness of the matching. By generating multiple query statements based on semantic matching, the system can more comprehensively cover the user's query intent and improve the recall rate and accuracy of the query. At the same time, this method also enhances the robustness and fault tolerance of the system. Even if some words in the original query statement are ambiguous or vague, the system can still find related words through semantic matching and generate valid query statements.

[0099] In one embodiment, after the step of inputting the second task processing instruction into the intelligent language model to obtain a plurality of second natural query statements corresponding to the first natural query statement in step S1022, the following steps are further included:

[0100] If any of the second natural query statements contains the target basic field in the corresponding first natural query statement, obtain several words that semantically match the target basic field, replace the target basic field in the second natural query statement with the semantically matching words, and obtain several second natural query statements corresponding to the first natural query statement.

[0101] After step S1022 of this embodiment, the system will first traverse all the second natural query statements generated by the intelligent language model. For each second natural query statement, it can be identified by character matching whether it contains the same target basic field in the first natural query statement. Once it is confirmed that a second natural query statement contains the target basic field, the system will then perform a semantic matching operation. As in the above embodiment, in this step, the system will use a pre-trained semantic similarity model or dictionary to search for several words that are highly semantically matched with the target basic field. These words can be synonyms, near-synonyms, or phrases describing similar concepts, aiming to enrich the diversity and accuracy of the query statement. Subsequently, the system will replace the original target basic field in the second natural query statement with one of the semantically matched words found above (which can be randomly selected or based on a certain strategy such as the one with the highest relevance score). The replacement operation will maintain the grammatical correctness of the query statement and try to keep the query intent of the original sentence unchanged. After the above replacement operation, the system will update all second natural query statements to form a new set of second natural query statements. Each statement in this set is based on the first natural query statement, expanded through an intelligent language model, or replaced with semantically matching vocabulary. Therefore, they retain the core intent of the original query while increasing the flexibility and diversity of the query.

[0102] This embodiment makes up for the possible deficiencies of the intelligent language model, ensures the effectiveness of the second natural query sentence as a training sample, and can be used to train the model's recognition ability for basic fields.

[0103] For step S103, each of the second natural query statements, the preset first task execution text, and the first query label information corresponding to each of the second natural query statements are input into the text processing model for training; wherein the first query label information includes the target basic field contained in the first natural query statement corresponding to the second natural query statement; the first task execution text is used to prompt the text processing model to output query information that meets the query intent of the second natural query statement, and the query information includes basic fields.

[0104] In this step, the generated second natural query statement, the preset first task execution text, and the first query label information corresponding to each second query statement are combined and input into the text processing model as training data. The first task execution text is a prompting text used to guide the model to understand and output query information that meets the query intent during the training process. For example, it can be "Please output query information that meets the query intent of the following natural query statement, which contains relevant basic fields." The first query label information includes the target basic fields in the first natural query statement corresponding to each second query statement, which is used as label data during the training process to help the model learn how to identify basic fields from natural language queries.

[0105] Please refer to Figure 4 In one embodiment, the step of inputting each of the second natural query statements, the preset first task execution text, and the first query tag information corresponding to each of the second natural query statements into the text processing model for training in step S103 includes:

[0106] Step S1031, constructing a first training sample corresponding to each of the second natural query statements according to each of the second natural query statements, the preset first task execution text, and the first query label information corresponding to the second natural query statement;

[0107] Step S1032, obtaining a training data set according to each of the first training samples; inputting the training data set into a text processing model for training.

[0108] In this embodiment, the training process of step S103 is further described. In step S1031, the system first collects all the second natural query statements generated by the previous steps, and then constructs a training sample by combining each second natural query statement with the preset first task execution text. In addition, the training sample also includes query label information corresponding to the second natural query statement, and the query label information is the label data of the training sample. In other words, the training sample includes at least three parts: a query statement (i.e., a second natural query statement), a task execution text (i.e., a preset first task execution text), and query label information. Such a structure enables the model to simultaneously learn the semantics of the query statement, the specific requirements of the task execution, and the intention or classification of the query during training. After completing the construction of all the first training samples, they are aggregated into a training data set in step S1032. Subsequently, the training data set is input into the text processing model for training. Through continuous iterative training, the model will gradually learn the intrinsic association between the query statement, the task execution text, and the query label, and comprehensively improve the generalization ability of the model to understand the basic fields of the natural query statement, so that when a new query statement is input, the corresponding query information can be accurately generated.

[0109] In summary, this embodiment provides rich and effective learning materials for the training of text processing models by constructing first training samples and organizing them into training data sets. This method not only improves the efficiency and accuracy of model training, but also enables the model to better understand the semantics and task requirements of query statements, thereby generating more accurate and expected query information. Through continuous training and optimization, the model can gradually adapt to various complex query scenarios and task requirements, providing users with more intelligent and convenient text processing services.

[0110] In one embodiment, the text processing model includes an encoder and / or a decoder, wherein the encoder and the decoder respectively include an attention layer and a fully connected layer connected to each other;

[0111] The step of inputting the training data set into the text processing model for training in step S1032 includes:

[0112] Step S10321, obtaining an initialized low-rank matrix, embedding the low-rank matrix in a weight matrix of a fully connected layer or an attention layer of an encoder of the text processing model, or embedding the low-rank matrix in a weight matrix of a fully connected layer or an attention layer of a decoder of the text processing model;

[0113] Step S10322, input the training data set into the text processing model, obtain the query information corresponding to each generated training sample; calculate the loss value between the query information of each training sample and the corresponding query label information; according to the loss value, adjust the weight of the low-rank matrix until the loss value is less than a preset loss threshold or the total number of weight adjustments reaches a preset number of adjustments, and obtain a trained text processing model; or, input the training data set into the text processing model, and obtain the query information corresponding to each training sample generated within the preset time at preset intervals; calculate the loss value between the query information of each training sample generated within the preset time and the corresponding query label information, and cache the current text processing model and the corresponding loss value; according to the loss value, adjust the weight of the low-rank matrix until the total number of weight adjustments reaches the preset number of adjustments, and determine the text processing model with the smallest loss value as the trained text processing model from among the cached text processing models and the corresponding loss values.

[0114] The text processing model includes an encoder and / or a decoder, that is, the text processing model of this embodiment can be an encoder-only model, a decoder-only model, or an encoder-decoder model, and all three types of models can be applied in the technical solution of this application. In the encoder or decoder, an attention layer and a fully connected layer are included respectively, and the attention layer and the fully connected layer are interconnected, which can effectively process text data and realize the conversion from input to output.

[0115] In step S10321, the weight matrix of the model is optimized by embedding a low-rank matrix to reduce the number of parameters and computational complexity of the model, and at the same time may improve the generalization ability of the model. Specifically, the initialized low-rank matrix is ​​embedded in the weight matrix of the fully connected layer or attention layer of the encoder or decoder. This embedding method can be flexibly selected and adjusted according to the specific structure and performance requirements of the model. Regarding the specific structure and principle of the low-rank matrix, it can be roughly understood that it is based on the original parameter weight matrix of the model, and two low-rank column matrices A and row matrices B are determined, and these two matrices are embedded in the model as low-rank matrices in the form of products, so that in the data training process, without changing the original structure and original weight matrix of the model, the weight of the low-rank matrix can be adjusted to achieve the adjustment of the overall superimposed weight of the model (the weight of the low-rank matrix is ​​superimposed). After the training is completed, the weight of the low-rank matrix is ​​merged into the model to obtain a fine-tuned model. Please refer to the LoRA fine-tuning principle in this field for details. If there are any inaccuracies in the description here, please refer to the LoRA fine-tuning principle in this field, and no further description will be given here.

[0116] In step S10322, the training data set obtained in the previous step is obtained. The training data set should contain rich training samples, each training sample is a different second natural query statement, and in addition, it also includes data such as task execution text, which will construct the task processing instruction and input it into the text processing model for training. In addition, each training sample also includes query label information corresponding to the second natural query statement, and these query label information are the target query information of the second natural query statement, that is, the accurate query information expected to be generated, so as to be used as a label during model training, and used for the model to calculate the loss value and adjust the parameters according to the loss value. In one embodiment, the query label information as a label is the target query information of the second natural query statement in a preset format, so that when fine-tuning the training model, the training model can organize the query information in a preset format and then output it. Correspondingly, the preset loss function comprehensively evaluates the accuracy of the query information and the accuracy of the format, calculates the loss value, and more accurately adjusts the weight of the low-rank matrix, so that the fine-tuned text processing model meets the requirements of query information accuracy and format accuracy.

[0117] In step S10322, two training strategies are proposed, one of which is to input the training data set into the text processing model embedded in the low-rank matrix. The model outputs the corresponding query information based on the input second natural query sentence and the first task execution text. Then, a preset loss function is used to calculate the loss value between the query information and the query label information of each training sample. According to the loss value, the weight of the low-rank matrix is ​​adjusted by the back propagation algorithm until the loss value is less than the preset loss threshold or the total number of weight adjustments reaches the preset number of adjustments. This process continuously adjusts the weight of the low-rank matrix and iterates continuously until the model converges, and the generated query information is as close as possible to the actual query label information, and a trained text processing model is obtained.

[0118] In addition, another training strategy can be adopted. Compared with the previous training strategy, it sets a preset time interval during the training process. At the end of each time interval, the system collects and calculates the loss value between the query information and the query label information of all training samples in the period. At the same time, the state of the current model (including its parameters) and the corresponding loss value are cached. As the training progresses, when the total number of weight adjustments reaches the preset limit, the system selects the one with the smallest loss value from the cached multiple model versions as the final training completion model. This method allows the model to conduct more exploration during the training process while ensuring that the final selected model has the best performance. Specifically, in the step of inputting the training data set into the text processing model embedded in the low-rank matrix, the training data set can be divided into multiple small batches of data sets and input into the text processing model one after another; or it can be divided into multiple batches of data sets, and each time a number of training samples are randomly obtained from the training data set (self-help resampling method or sampling method without replacement) and input into the text processing model. After receiving the input training sample, the text processing model starts to calculate and output the query information corresponding to the training sample. This embodiment obtains query information corresponding to each training sample generated within the preset time at preset intervals, uses a preset loss function to calculate the loss value between the query information of each training sample generated within the preset time and the corresponding query label information, and caches the current text processing model and the corresponding loss value; according to the loss value, adjusts the weight of the low-rank matrix to update the first text processing model; continues to input the training samples into the updated text processing model, and again obtains query information corresponding to each training sample generated within the preset time at preset intervals, calculates the loss value, and caches the current text processing model and the corresponding loss value; until the total number of weight adjustments reaches the preset number of adjustments, the text processing model with the smallest loss value is determined as the trained text processing model from among the cached text processing models and the corresponding loss values.

[0119] The trained text processing model described in this embodiment is fine-tuned on the basis of the pre-trained text processing model. Specifically, the LoRA fine-tuning method can be adopted. By embedding a low-rank matrix in the weight matrix of the model, the number of parameters and computational complexity that need to be adjusted in the model training process are reduced. This embodiment is based on a rich second natural query statement, and its rich training data set is constructed for training, so that the model can accurately understand the basic fields and other information of the natural query statement, and generate query information that meets expectations. The trained text processing model finally obtained can be used as a large data query model for the target data warehouse. When processing query tasks for the target data warehouse, it can show higher accuracy and efficiency, and improve the performance of the overall system and user experience.

[0120] For step S104, the trained text processing model is determined as the data query macro model corresponding to the target data warehouse.

[0121] After sufficient training, the trained text processing model is determined as the data query model corresponding to the target data warehouse. This model has learned how to accurately understand the user's natural query sentences, and can identify the relevant basic fields from the constellation data model of the target data warehouse, generate accurate query information (to be further generated in a certain format) or directly generate database query statements. Users can now use this data query model to submit natural query sentences, and the model will automatically generate query information or data query statements, so that business data can be queried in the target data warehouse, thereby providing users with fast and accurate data query services.

[0122] Please refer to Figure 5 In one embodiment, the data query large model training method further includes the steps of:

[0123] Step S301, in response to a user's update operation on the target data warehouse, obtaining update information of a constellation data model corresponding to the target data warehouse;

[0124] Step S302: if the update information indicates that a new basic field is added to the constellation data model, the new basic field is determined as a new target basic field;

[0125] Step S303, obtaining a plurality of third natural query statements; if any of the third natural query statements contains the newly added target basic field, generating a plurality of fourth natural query statements having the same query intent according to the third natural query statement; wherein the fourth natural query statement does not contain the newly added target basic field in the third natural query statement;

[0126] Step S304, constructing a second training sample corresponding to each of the fourth natural query statements according to each of the fourth natural query statements, the first task execution text, and the second query label information corresponding to the fourth natural query statement; wherein the second query label information includes the newly added target basic field contained in the third natural query statement corresponding to the fourth natural query statement;

[0127] Step S305, updating each of the second training samples into the training data set, and retraining the text processing model with the updated training data set; determining the trained text processing model as the data query macro model corresponding to the target data warehouse.

[0128] This embodiment further expands the training method of the data query big model to support dynamic updating and adaptive learning of the target data warehouse, thereby ensuring that the data query big model can be automatically adjusted and optimized as the data warehouse is updated, thereby maintaining the accuracy and timeliness of its query capabilities.

[0129] In step S301, when the user performs update operations on the target data warehouse, the system will actively monitor and respond to these operations. Specifically, the system will capture all information related to the update operation, especially the update information related to the constellation data model. This information is crucial to understand the latest status of the data warehouse.

[0130] After obtaining the updated information in step S302, the system will further analyze the information to determine whether new basic fields have been added to the constellation data model. Basic fields are elements used to store key information in the data warehouse, such as user ID, order number, etc. If the system detects that new basic fields have been added, then these fields will be regarded as new target basic fields because they may have an important impact on subsequent query operations.

[0131] Step S303: In order to cope with the changes brought about by the newly added target basic fields, the system collects a series of third natural query statements containing these newly added fields. The collection of these query statements specifically refers to the above embodiment, and can be submitted by the user, or automatically generated by the system, etc. For each third natural query statement containing the newly added target basic field, the system generates multiple fourth natural query statements with the same query intent but without the newly added field.

[0132] Next, step S304 constructs a second training sample based on the generated fourth natural query, the preset first task execution text, and the second query label information corresponding to the fourth natural query (specifically referring to the label related to the newly added target basic field). These samples will be used to guide the text processing model to learn how to process query statements that do not contain the newly added fields but have the same intent.

[0133] Finally, step S305 updates the constructed second training sample to the existing training data set, and uses the updated data set to train the text processing model again. The purpose of this training is to enable the model to adapt to the latest structural changes in the data warehouse, especially to understand and process query statements containing newly added target basic fields. After the training is completed, the obtained model will be determined as the data query model corresponding to the target data warehouse, ready for actual use.

[0134] In summary, the data query large model training method proposed in this embodiment realizes the adaptive learning of the model to the changes in the data warehouse structure by dynamically monitoring and responding to the update operation of the target data warehouse. This method not only improves the flexibility and scalability of the model, but also ensures that the model can maintain the accuracy and efficiency of query capabilities when facing structural changes such as the addition of new basic fields. By continuously generating and training new training samples, the model can gradually accumulate the ability to understand and process various query statements, thereby providing users with more intelligent and convenient data query services.

[0135] Please refer to Figure 6 The present application also provides a method for querying a large data model based on the method of any one of the above embodiments, comprising the following steps:

[0136] Step S401, obtaining a natural query statement input by a user in a query interface of a target data warehouse;

[0137] Step S402, obtaining a constellation data model corresponding to the target data warehouse; determining a number of related basic fields from the constellation data model as candidate basic fields according to the natural query statement;

[0138] Step S403, constructing a task processing instruction according to the candidate basic fields, the natural query statement, and a preset query task execution text; wherein the query task execution text is used to prompt the data query big model to output query information that meets the query intent of the natural query statement, and the query information is determined according to the candidate basic fields;

[0139] Step S404, inputting the task processing instruction into the data query model corresponding to the target data warehouse to obtain output query information;

[0140] Step S405: query and obtain corresponding business data from the target data warehouse according to the query information and the constellation data model.

[0141] The data query method proposed in this embodiment achieves an in-depth understanding of the user's natural query statement and outputs accurate query information by integrating the constellation data model of the target data warehouse and its trained dedicated data query big model. Among them, the constellation data model, as the cornerstone of structured data, carries detailed information on the data table and basic fields of the target data warehouse. When the user enters a query statement in natural language to query the target data warehouse, the system can quickly filter out relevant candidate basic fields from the constellation data model of the target data warehouse. Subsequently, combining these candidate basic fields, natural query statements and preset query task execution texts, the system constructs an accurate task processing instruction. This instruction not only guides the data query big model on how to interpret the user's query intention, but also ensures that the query requirements involving the basic fields can be effectively identified and processed, and the query information that meets the query requirements can be accurately output, thereby accurately extracting business data from the target database, supporting users to conduct deeper data analysis and business insights. Through the method of this embodiment, the recognition accuracy of the basic fields of the target data warehouse is improved, not only supporting direct query of the basic fields of the database, but also significantly enhancing the processing capabilities of complex operation fields (such as MDX fields). This innovative technology not only simplifies the data query process and lowers the user operation threshold, but also greatly improves the accuracy and flexibility of the query, provides users with more comprehensive and in-depth data analysis support, and significantly optimizes the overall user experience.

[0142] The data query method of this embodiment is executed by a computer. Hereinafter, the data query model of the embodiment of the present application is first described, and then each step is described in detail.

[0143] In this embodiment, the data query model is obtained by training the constellation data model of the target data warehouse through the model training method of the above embodiment. It can parse the natural query sentence input by the user, understand the query intention, and generate corresponding query information according to these intentions, so as to retrieve data from the database later.

[0144] For step S401, a natural query statement input by a user in a query interface of a target data warehouse is obtained.

[0145] In this embodiment, the user can input the query requirements in the form of natural language through the user query interface provided by the system (such as a web page, mobile application or command line interface). For example, the user may input "query the names and sales of the top 10 products with the highest sales in the first quarter of 2023", or for another example, the user may input "query the total sales and year-on-year growth rate of each sales region in the first quarter of this year". This step is the starting point of the data query process and provides the necessary input for subsequent steps.

[0146] For step S402, a constellation data model corresponding to the target data warehouse is obtained; and several relevant basic fields are determined from the constellation data model as candidate basic fields according to the natural query statement.

[0147] This step first loads or accesses the constellation data model, and then selects basic fields related to the query content, such as "product name" and "sales", as candidate basic fields based on the natural query statement entered by the user. These candidate basic fields are the column names that may be needed when constructing the query statement later.

[0148] In one embodiment, the step of determining, according to the natural query statement, a number of relevant basic fields from the constellation data model as candidate basic fields in step S402 includes:

[0149] Step S4021, performing keyword recognition on the natural query sentence to obtain a number of keywords;

[0150] Step S4022, vectorizing each of the keywords and each basic field in the constellation data model; calculating the vector similarity between any of the keywords and any of the basic fields;

[0151] Step S4023, for any of the keywords, determining a first preset number of basic fields that are ranked high in similarity to the keyword as relevant basic fields of the keyword;

[0152] Step S4024, ranking the relevant basic fields of each keyword by similarity according to the relevant basic fields of each keyword and their corresponding similarities;

[0153] Step S4025: Determine a second preset number of related basic fields with top similarity rankings as candidate basic fields.

[0154] The first preset number and the second preset number can be set according to specific circumstances. In this embodiment, the first preset number is set to 8, and the second preset number can be set to 40.

[0155] This embodiment describes in detail how to filter out the basic fields related to the natural query statement input by the user from the constellation data model as the candidate basic fields. First, in step S4021, the natural query statement is identified by keywords, and a number of keywords that can represent the user's query intention are extracted using natural language processing technology (such as word segmentation, part-of-speech tagging, etc.). These keywords are the basis for subsequent analysis and are crucial for accurately understanding user needs. Then, in step S4022, the identified keywords and each basic field in the constellation data model are converted into a vector representation in a high-dimensional space using vectorization technology. Vectorization is a common method for processing text data. It can convert text information into a numerical form that is easy for a computer to process, which is convenient for subsequent similarity calculations. Then, the vector similarity between each keyword and each basic field is calculated. Similarity calculation is a key step in evaluating the proximity of two vectors in a high-dimensional space, and its result can reflect the correlation between the keyword and the basic field. Furthermore, in step S4023, for each keyword, the first preset number of basic fields with a high similarity ranking are selected as the relevant basic fields of the keyword. Through this step, the basic fields most relevant to each keyword can be screened out, reducing the amount of data to be processed subsequently and improving the efficiency and accuracy of the query. Step S4024 performs a global similarity ranking on these basic fields according to the relevant basic fields of each keyword and their corresponding similarities. Through the global similarity ranking, the query requirements of multiple keywords can be comprehensively considered to ensure that the candidate basic fields finally selected can fully reflect the user's query intention, thereby improving the comprehensiveness and accuracy of the query results. Finally, in step S4025, the second preset number of relevant basic fields with the highest similarity ranking are determined as candidate basic fields. Through this step, a limited number of candidate basic fields are finally determined, which are most likely to meet the user's query requirements. This not only reduces the complexity and computational complexity of subsequent query processing, but also improves the response speed and user satisfaction of queries.

[0156] In summary, this embodiment achieves efficient and accurate conversion from natural query statements to candidate basic fields through steps such as keyword recognition, vectorization processing, similarity calculation, relevant basic field screening and global ranking, providing strong support for subsequent query processing.

[0157] In one embodiment, the step of ranking the related basic fields of each keyword by similarity according to the related basic fields of each keyword and their corresponding similarities in step S4024 includes:

[0158] Step S40241: If any two of the keywords have the same related basic fields and correspond to different similarities, the related basic field with the highest similarity among the same related basic fields is retained, and the remaining related basic fields are removed.

[0159] This embodiment further refines the specific implementation method of step S4024. Specifically, when it is found during the processing that any two or more keywords have the same relevant basic fields, but these same basic fields correspond to different similarities, a specific processing strategy is adopted. First, the system needs to identify which basic fields are considered relevant by multiple keywords, that is, there are repeated basic fields. For each basic field shared by multiple keywords, the system needs to compare the similarity between these keywords and the basic field. After comparing the similarities, the system will choose to retain the basic field with the highest similarity (with the corresponding keyword), and remove other (same) basic fields (that is, they will not participate in the global ranking).

[0160] For example, the natural query sentence input by the user is "What is the contract amount of the financial industry this year?" Assume that the determined keywords include "this year", "finance", "industry" and "contract amount". Among them, the related basic fields determined by the keyword "finance" and the corresponding similarities include "industry category" (0.84), "industry subcategory" (0.82), "contract amount" (0.77), etc.; and the related basic fields determined by the keyword "contract amount" and the corresponding similarities include "contract amount" (0.98), "average contract amount" (0.88), "number of contracts" (0.75), etc. Then, it can be found that the keywords "finance" and "contract amount" have the same related basic field "contract amount", but the corresponding similarities of the two are 0.77 and 0.98 respectively. Therefore, when all related basic fields are ranked globally, the related basic field with the greatest similarity is retained, that is, the "contract amount" with a similarity of 0.98 is included in the global similarity ranking, while the "contract amount" with a similarity of 0.77 is removed and does not participate in the global similarity ranking.

[0161] Finally, in the global similarity ranking of related basic fields, for two or more identical related basic fields (corresponding to different keywords), only the related basic field with the highest similarity will be retained to participate in the ranking. Therefore, two identical basic fields (these two identical basic fields correspond to different keywords with different similarities) will not appear in the global ranking.

[0162] This embodiment limits the sorting rules for the similarity of the relevant basic fields of keywords. For the case where different keywords correspond to the same relevant basic fields, by retaining the same relevant basic fields with the highest similarity and removing other same relevant basic fields with lower similarities, it is ensured that the second preset number of candidate basic fields finally determined contain as many different basic fields as possible, effectively reducing the redundant information in the candidate basic fields and expanding the scope of the candidate basic fields selected. On the whole, the candidate basic fields cover as many basic fields as possible, which is as comprehensive and relatively accurate as possible, laying a better foundation for the subsequent processing of the large data query model and improving the accuracy, efficiency and user experience of the final query.

[0163] In one embodiment, the step of performing keyword recognition on the natural query statement to obtain a plurality of keywords in step S4021 includes:

[0164] Step S40211, segmenting the natural query sentence based on a preset segmenter to obtain a segmentation result;

[0165] Step S40212: Based on the pre-trained entity recognition model, entity recognition is performed on the natural query sentence and the word segmentation result to obtain a first entity recognition result and a second entity recognition result respectively;

[0166] Step S40213, obtaining a number of keywords according to the first entity recognition result and the second entity recognition result.

[0167] Among them, the word segmenter is a basic tool in natural language processing, and its main function is to split a continuous text string into independent vocabulary units (tokens). There are many ways to implement the word segmenter, including rule-based word segmentation, statistical-based word segmentation, and hybrid methods. In practical applications, the choice of word segmenter depends on the specific language characteristics and task requirements. In this embodiment, a suitable word segmenter can be selected according to the specific scenario.

[0168] The entity recognition model is an important component in natural language processing, which is used to identify entities with specific meanings from text. These entities can be named entities (such as names of people, places, names of institutions, etc.), or other entities with specific meanings (such as time, date, numbers, etc.). Entity recognition models are usually built based on machine learning or deep learning technology, and learn the characteristics and patterns of entities in text by training a large amount of labeled data. This embodiment accurately identifies entities in text through the entity recognition model, and can better determine candidate basic fields related to natural query statements.

[0169] This embodiment describes in detail how to perform keyword recognition on natural query statements. This process aims to extract keywords that are crucial to subsequent query processing from natural language texts to more accurately determine relevant candidate basic fields. First, in step S40211, a preset word segmenter is used to perform word segmentation on the natural query statement. Through word segmentation, complex query statements can be decomposed into more easily processed vocabulary sequences. Then, in step S40212, a pre-trained entity recognition model is used to perform dual entity recognition on the natural query statement and its word segmentation results. By performing entity recognition on the original query statement and the word segmentation result respectively, the key information in the query statement can be captured more comprehensively and the accuracy of keyword recognition can be improved. The first entity recognition result is based on the overall understanding of the original query statement, while the second entity recognition result focuses more on the vocabulary unit after word segmentation. The two complement each other and together constitute an in-depth understanding of the query statement. Finally, in step S40213, several keywords are extracted based on the comprehensive analysis of the first entity recognition result and the second entity recognition result. These keywords include not only the explicit named entities in the query, but also some words with specific meanings or importance, which together constitute the core content of the query and provide key guidance for subsequent data retrieval and processing. In one embodiment, the entities in the first entity recognition result and the second entity recognition result are combined to obtain all the keywords of the natural query.

[0170] In summary, this embodiment achieves accurate recognition of keywords in natural query sentences by introducing word segmentation and dual entity recognition. This process not only improves the accuracy and comprehensiveness of keyword recognition, thereby ensuring accurate and comprehensive determination of candidate basic fields, but also provides strong support for subsequent query information generation and data retrieval.

[0171] For step S403, a task processing instruction is constructed according to the candidate basic fields, the candidate multidimensional operation fields, the natural query statement and the preset query task execution text; wherein the query task execution text is used to prompt the data query big model to output query information that conforms to the query intent of the natural query statement, and the query information is determined according to the candidate basic fields.

[0172] In this step, a task processing instruction is constructed using the candidate basic fields, the original natural query statement, and the preset query task execution text. This task processing instruction is designed to guide the data query big model to select accurate basic fields from the candidate basic fields according to the intent of the natural query statement and generate query information that meets the query intent.

[0173] As described in the previous embodiments, the query task execution text is also used to require or prompt the task requirements of the data query large model, which includes specific requirements and format instructions for the model output, and may also include task-related scene information or context information for the model to learn and understand the tasks to be processed, etc. In the embodiment of the present application, different sub-texts can be set in the query task execution text according to the needs of the query, which are respectively used to require or prompt the corresponding task requirements of the data query large model. For example, the query task execution text may include a first query task execution text, a second query task execution text, a third query task execution text... and so on, and different sub-texts contain specific requirements for model processing and output according to task requirements, etc.

[0174] In one embodiment, the query task execution text in step S403 includes a first query task execution text; the first query task execution text is used to prompt that the query information output by the large model includes a first basic field and a query condition, and prompts that the first basic field is determined from the candidate basic fields and the query condition is determined based on the candidate basic fields.

[0175] The first basic field is used to query corresponding business data, and the query condition is used to limit the conditions that the business data queried by the first basic field needs to meet.

[0176] The first query task execution text is generally a command text used to request or prompt the data query big model what task to perform, which details the specific rules that the data query big model should follow when generating query information. Alternatively, the first query task execution text can also be an instruction set in a specific format, which also details the specific rules that the data query big model should follow when generating query information. The first query task execution text can be specifically determined in what form according to the requirements and specifications of the data query big model for task processing instructions.

[0177] The first basic field is the core part of the query information and directly corresponds to the field used to retrieve business data in the database. With the guidance of the first query task execution text, the data query model can accurately identify and select the most appropriate field as the basis for the query.

[0178] The query condition is a condition that further limits the query results of the first basic field and is used to filter out business data that meets specific requirements. It can include multiple types such as value range, dimension restriction, text matching mode or time window, depending on business needs and data characteristics.

[0179] In this embodiment, the first query task execution text prompts the data query big model, and the output query information contains two key elements: the first basic field and the query condition. It also instructs the big model to determine the first basic field from the candidate basic fields. At the same time, it also stipulates the method for determining the query condition. The query condition should be constructed based on the candidate basic fields to further limit the scope of the query results and ensure that the final query results only return business data that meets specific conditions.

[0180] In one embodiment, the constellation data model further records the multidimensional operation field bound to at least part of the basic fields; the query task execution text includes a second query task execution text; the second query task execution text is used to prompt the query information output by the data query large model to include a multidimensional operation field and a second basic field bound to the multidimensional operation field, the second basic field is determined from the candidate basic fields, and the multidimensional operation field is determined from the candidate multidimensional operation fields; the multidimensional operation field is used to specify the multidimensional operation on the business data of the second basic field;

[0181] After the step of determining a number of related basic fields as candidate basic fields from the constellation data model according to the natural query statement in step S402, the following steps are further included:

[0182] Step S4021, determining the multidimensional operation fields bound to each of the candidate basic fields as candidate multidimensional operation fields;

[0183] The step of constructing a task processing instruction according to the candidate basic fields, the natural query statement and the preset query task execution text in step S403 includes:

[0184] Step S4031, constructing a task processing instruction according to the candidate basic fields, the candidate multi-dimensional operation fields, the natural query statement and the preset query task execution text.

[0185] The form or content specification of the second query task execution text is the same as that of the first query task execution text, the main difference being that different task requirements are suggested for the data query big model.

[0186] In this embodiment, the second query task execution text prompts the data query big model, and the output query information includes a multidimensional operation field and a second basic field bound to the multidimensional operation field. This will guide the data query big model to deeply understand whether the natural query statement involves a query of the multidimensional operation field, and identify the multidimensional operation field and its corresponding basic field. Among them, it specifically instructs the data query big model to determine the second basic field from the candidate basic fields, and to determine the multidimensional operation field from the candidate multidimensional operation fields, guiding the data query big model to focus on these candidate fields. Since these candidate fields are all fields related to the user's query intention determined according to the constellation data model, therefore, it actually gives the data query big model enough information to enable it to accurately lock the query intention of the natural query statement and accurately determine the basic fields and / or multidimensional operation fields involved in this query.

[0187] It should be noted that, those skilled in the art understand that the query task execution text prompts the data query big model what to do, which does not mean that the data query big model must do it. For example, the query task execution text prompts the data query big model to output query information that meets the query intent, where the query information includes multidimensional operation fields, but after the data query big model understands the natural query statement, it believes that the query does not involve multidimensional operation fields. In this case, it may not output the multidimensional operation fields to avoid violating the core task of the query task execution text prompt, that is, to output query information that meets the query intent.

[0188] In one embodiment, the second query task execution text is further used to prompt that the second basic field bound to the multidimensional operation field is determined from the candidate basic fields bound to the multidimensional operation field. In the embodiment of the present application, any basic field in the constellation data model can be bound to one or more multidimensional operation fields. Similarly, the same multidimensional operation field may be bound to multiple basic fields. Therefore, when determining the candidate basic fields and determining the candidate multidimensional operation fields based on the candidate basic fields, the binding relationship between the basic fields and the multidimensional operation fields can also be recorded, and these binding relationships can be used to generate task processing instructions. When the data query large model recognizes that the query intent involves a multidimensional operation field, the second basic field can be determined based on the candidate basic fields bound to this multidimensional operation field.

[0189] In one embodiment, the query task execution text also includes a third query task execution text; the third query task execution text is used to prompt that the query information output by the data query big model includes query conditions, and the query conditions are determined according to the candidate basic fields; the query conditions are used to limit the conditions that the business data queried by the first basic field and / or the second basic field needs to meet.

[0190] In this embodiment, the third query task execution text prompts the data query big model to include query conditions in the output query information, and is constructed based on the candidate basic fields to further limit the scope of the query results and ensure that the final query results meet the specified conditions.

[0191] For step S404, the task processing instruction is input into the data query big model corresponding to the target data warehouse to obtain output query information.

[0192] In this step, the constructed task processing instructions are sent to the data query model of the target data warehouse. The model analyzes the intent of the natural query statement according to the requirements in the instructions, combines the candidate basic fields, and generates query information that meets the query requirements / intent. The query information is usually expressed in a structured format (such as JSON), which contains the specific field names and possible query logic used for database query.

[0193] For step S405, the corresponding business data is queried from the target data warehouse according to the query information and the constellation data model.

[0194] This step builds a complete database query statement based on the query information output by the data query model and the constellation data model. This query statement is sent to the target data warehouse for execution to retrieve business data that meets the user's query intent. This business data may include data from basic fields, calculation results of multi-dimensional operation fields, etc. This data can then be returned to the user or used for further analysis and processing.

[0195] In one embodiment, the query task execution text includes a fourth query task execution text, and the fourth query task execution text is used to prompt the data query big model to output the query information according to a preset report query request format;

[0196] The step of querying and obtaining corresponding business data from the target data warehouse according to the query information and the constellation data model in step S405 includes:

[0197] Step S4051, using the query information as a report query request, and inputting the report query request and the constellation data model into an MDX query engine; obtaining corresponding business data from the target data warehouse through the MDX query engine; wherein the MDX query engine is an engine for parsing the report query request and performing data query based on the parsing result and the constellation data model.

[0198] Among them, the MDX (Multidimensional Expressions) query engine is a component specially designed to process complex report queries. It can understand and parse query requests written in the MDX query language, and retrieve corresponding business data from the database based on these requests and the definition of the constellation data model (multidimensional data structure model, commonly used in OLAP systems).

[0199] In this embodiment, for the understanding of the MDX query engine, specific reference can be made to the working principle of the Mondrian engine (an open source online analytical processing engine developed based on Java). In the MDX query engine, it is necessary to combine the data model description information and the MDX statement for querying. The MDX statement is a query statement with a similar grammatical structure to the SQL statement. In the SQL statement or the MDX statement, the general format is: select query field from query data source; when there is a field that needs to be defined, the query statement also includes "with field operation definition" before the select field; when there is a query restriction condition that needs to be limited, the query statement also includes "where query restriction condition" after the from field. Among them, the query data source generally specifies the target data table of the target database that the user wants to query, so it generally fills in the table name of the data table. In some cases, it can also fill in another query statement (the query statement actually determines a set of data, and this set of data can also be considered to constitute a data table).

[0200] Specifically, the data model description information is generally a Schema XML (Extensible Markup Language) file, which describes the database table description information, measurement description information, dimension description information, and multidimensional operation field (MDX operation field) description information (including calculated member description information and / or named set description information) of a data model cube; the query data source of the MDX statement is determined to be the model name defined in the data model description information (generally speaking, the Schema XML defines the model name as cube, and the corresponding name in the MDX statement is from cube). In the MDX query engine, for a multidimensional operation field (MDX operation field), corresponding operation definition information (such as a calculation formula) is required to perform the corresponding operation. Specifically, the definition of the multidimensional operation field can be defined in the with part of the MDX statement, or in the data model description information (Schema XML file).

[0201] In this embodiment, the MDX query engine is a multidimensional query engine obtained by secondary development based on a general multidimensional analysis engine such as the Mondrian engine, that is, it not only executes queries according to MDX statements and data model description information like a general multidimensional analysis engine, but also generates MDX statements and data model description information before that. For details, please refer to the prior art description, and also refer to the description involved in the subsequent embodiments.

[0202] In this embodiment, the function of the query task execution text is further expanded, and the fourth query task execution text is introduced. The text has a specific function - to guide the data query big model to organize and output the query information according to the preset report query request format. Specifically, the fourth query task execution text can set prompt texts one by one according to the preset report query request format, so as to require the data query big model to adjust the format of the query information according to the prompt requirements one by one, so as to finally output the query information that conforms to the report query request format. This design is intended to ensure that the format of the query information matches the expected input format of the back-end data system (especially the database system that supports MDX queries), thereby improving the accuracy and efficiency of data queries.

[0203] Specifically, when performing data query, the query information output by the data query big model is directly submitted to the MDX query engine as a report query request. The MDX query engine receives the report query request, parses the target field of the user's query and the query restriction conditions involved, and then uses the constellation data model as a navigation map to perform query operations in the database and finally returns the business data that meets the conditions.

[0204] In summary, this application significantly enhances the interaction between the data query big model and the backend data system by introducing the fourth query task execution text and defining the implementation process of step S405 in detail. Specifically, it ensures that the format of the query information is highly consistent with the input requirements of the MDX query engine, so that based on the existing technical architecture of the MDX query engine, it is only necessary to directly input the query information as a general report query request into the MDX query engine to perform the query.

[0205] In one embodiment, the MDX query engine in step S4051 obtains the corresponding business data by querying in the following manner:

[0206] Step S40511, parsing the report query request according to a preset parsing method to obtain a target field for query; determining a basic field according to the target field;

[0207] Step S40512, determining a plurality of star data models including the basic fields from the constellation data model; wherein each of the star data models respectively records a fact table, a dimension table, and records the basic fields respectively included in each data table;

[0208] Step S40513, if the target field includes a multidimensional operation field, obtain corresponding operation definition information according to the multidimensional operation field; generate data model description information according to each of the star data models and the operation definition information; generate an MDX statement according to the report query request and the operation definition information; query the query result of the target field from the target data warehouse according to the MDX statement and the data model description information; determine that the query result of the target field is the business data obtained by the query;

[0209] Step S40514, if the target field does not include a multidimensional operation field, generate data model description information according to each of the star data models; generate an MDX statement according to the report query request; query the target field from the target data warehouse according to the MDX statement and the data model description information; determine that the query result of the target field is the business data obtained by the query.

[0210] Among them, in step S40511, the report query request is parsed to obtain the target field that the user wants to query. Specifically, in different query requirements, the target fields obtained by parsing are generally different. In this embodiment, the target fields obtained by parsing the report query request include basic fields and / or multi-dimensional operation fields. Generally speaking, the target fields queried by the report query request may include only basic fields (meaning that only basic data in the database is queried), or only multi-dimensional operation fields (meaning that only a certain data analysis result or a certain specified multi-dimensional data operation is queried, such as the year-on-year growth rate, without querying specific basic data), and may also include both basic fields and multi-dimensional operation fields (meaning that both basic data and data obtained by specified multi-dimensional data operations are queried). Among them, if the target field of the query includes a multidimensional operation field, the multidimensional operation field will also be bound to at least one basic field - that is, the multidimensional operation field actually specifies the operation of the bound basic field. However, it should be noted that in this case, the multidimensional operation field is the target field of the query, but the basic field bound to it is not the target field of the query. For example, the query request queries the "sales volume" "same period growth rate", where "sales volume" is the target field of the query, and "sales volume" is just the basic field referenced by the target field. The final query result should be the query result of "sales volume" and does not include the data of the "sales volume" field (unless the basic field "sales volume" is also the target field of the user's query). This step determines the basic field related to the query based on the target field (same as above, not necessarily the target field of the query), so that in step S40512, the star data model related to this query can be determined based on the basic field. Specifically, if the target field of the query is a basic field, then it is the basic field related to the query; if the target field of the query is a multidimensional operation field, then the basic field bound to it is the basic field related to the query. In some special cases, there is another type of aggregate field. This type of field is associated with a basic field. The associated basic field is the basic field related to the query. For example, for the aggregate field "average sales volume", the "sales volume" field is the basic field related to the query.

[0211] In step S40512, the star data model is determined from the constellation data model. Generally speaking, in a complex constellation data model, if different data tables are centered, different star data models can be determined. A data table can be used as a fact table in a star data model and as a dimension table in other star data models.

[0212] In this step, the main goal of determining the star data model from the constellation data model is to obtain the minimum available star data model related to the query. The minimum available star data model should only contain data tables related to the query, and the data tables should only contain basic fields determined in step S40511, wherein the data tables related to the query are divided into several cases: for dimension tables, they are considered to be data tables related to the query only when they contain the basic fields determined in step S40511 or are bound to query-related restriction information; there is no limitation for fact tables. Generally speaking, a fact table is a necessary data table in a star data model, even if it does not contain basic fields. There are many ways to determine the minimum available star data model, and specific reference may be made to the prior art in the field, which will not be elaborated here.

[0213] In step S40513, the operation definition information is used to define the multidimensional operation mode corresponding to the multidimensional operation field, which can also be said to define what operation is performed on the data of the basic field bound to the multidimensional operation field. Take an example of querying in an existing report query interface (including various query components) for explanation, assuming that the predefined multidimensional operation field is "sales volume", the user selects the "sales volume" metric field when querying, and selects the parameter of its "time" dimension field as 2024, and also selects the "sales volume" multidimensional operation field, submits the query, then, when the backend processes the query, it obtains the corresponding definition formula according to the "sales volume", such as "sales volume = (current data - previous data) / previous data", and substitutes the specific basic field (metric field and / or dimension field) and related parameters into the definition formula to obtain the operation definition information of the "sales volume" of the query, such as "sales volume in 2024 - sales volume in 2023) / sales volume in 2023". Of course, this example is only for illustrative purposes to facilitate understanding of the multidimensional operation fields and their operation definition information. The specific implementation details can be set according to the situation. How each of the star data models generates data model description information and how the report query request generates an MDX statement are prior arts in the art. Please refer to the prior arts for details and will not be elaborated here.

[0214] In step S40514, the target data warehouse includes data of each data table corresponding to the constellation data model. That is, the target data warehouse is actually the basis for modeling the constellation data model. In one embodiment, the target data warehouse is preferably an OLAP (On-Line Analytical Processing) database.

[0215] In this embodiment, the basic field is determined according to the target field queried by the user, and several star data models containing the basic field are determined. If the target field includes a multidimensional operation field, the corresponding operation definition information is obtained according to the multidimensional operation field through the MDX query engine; data model description information is generated according to each of the star data models and the operation definition information; MDX statements are generated according to the report query request and the operation definition information; and query results of the target field are obtained from the database according to the MDX statements and the data model description information. If the target field does not include a multidimensional operation field, data model description information is directly generated according to each of the star data models, and MDX statements are directly generated according to the report query request; the subsequent steps are similar according to the MDX statements and the data model description information. Through this embodiment, the MDX query engine can accurately parse the report query request, generate data model description information and MDX statements, and thus, in combination with the data model description information and the MDX statements, it can efficiently query the query results of the target field from the target data warehouse, and determine that the query results of the target field are the business data obtained by the query.

[0216] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present application, and the present application is also intended to include such modifications and modifications.

Claims

1. A data query large model training method, characterized in that: The following steps are involved: Obtaining a number of first natural query statements; If any of the first natural query statements contains a target basic field, generate a plurality of second natural query statements having the same query intent according to the first natural query statement; wherein the second natural query statement does not contain the target basic field in the first natural query statement; the target basic field is determined from a constellation data model corresponding to a target data warehouse; the constellation data model records a plurality of data tables and basic fields contained in each data table; Input each of the second natural query statements, the preset first task execution text, and the first query label information corresponding to each of the second natural query statements into a text processing model for training; wherein the first query label information includes the target basic field contained in the first natural query statement corresponding to the second natural query statement; the first task execution text is used to prompt the text processing model to output query information that meets the query intent of the second natural query statement, and the query information includes the basic field; Determine the trained text processing model as the data query macro model corresponding to the target data warehouse; The method further comprises the steps of: Obtain historical query statement data from the target data warehouse; Determine, according to the historical query statement data, a basic field whose number of historical queries is greater than a preset number; The other basic fields in the constellation data model corresponding to the target data warehouse, except for the basic field whose historical query times are greater than the preset times, are determined as target basic fields.

2. The data query large model training method according to claim 1 is characterized in that: The method further comprises the steps of: The user's selection operation information for the data warehouse list is obtained, and the target data warehouse is determined according to the selection operation information.

3. The data query large model training method according to any one of claims 1-2, characterized in that: The step of generating a plurality of second natural query statements having the same query intent according to the first natural query statement if any of the first natural query statements contains the target basic field comprises: If any of the first natural query statements contains a target basic field, construct a second task processing instruction according to the first natural query statement and a preset second task execution text; wherein the second task execution text is used to prompt the intelligent language model to replace the target basic field in the first natural query statement with other semantically matching words, so as to obtain a plurality of second natural query statements having the same query intent as the first natural query statement; The second task processing instruction is input into an intelligent language model to obtain a plurality of second natural query statements corresponding to the first natural query statement.

4. The data query large model training method according to claim 3 is characterized in that: The second task execution text is also used to prompt the intelligent language model, and the target basic field of the first natural query sentence is determined from the candidate basic fields; The step of constructing a second task processing instruction according to the first natural query statement and the preset second task execution text includes: Obtain each target basic field in the constellation data model corresponding to the target data warehouse to obtain candidate basic fields; A second task processing instruction is constructed according to the first natural query statement, the candidate basic fields and the preset second task execution text.

5. The data query large model training method according to any one of claims 1-2, characterized in that: The step of generating a plurality of second natural query statements having the same query intent according to the first natural query statement if any of the first natural query statements contains the target basic field comprises: If any of the first natural query statements contains a target basic field, obtain a number of words that semantically match the target basic field, replace the target basic field in the first natural query statement with the semantically matching words, and obtain a number of second natural query statements that have the same query intent as the first natural query statement.

6. The data query large model training method according to claim 5 is characterized in that: The step of acquiring a plurality of words that semantically match the target basic field includes: Calculate a vector of the target basic field according to the first natural query statement and the target basic field; The words whose vector similarity with the target basic field is greater than a preset similarity threshold are obtained as words that semantically match the target basic field.

7. The data query large model training method according to claim 1, characterized in that: The step of inputting each of the second natural query sentences, the preset first task execution text, and the first query tag information corresponding to each of the second natural query sentences into the text processing model for training includes: Constructing first training samples corresponding to each of the second natural query statements according to each of the second natural query statements, the preset first task execution text, and the first query label information corresponding to the second natural query statement; A training data set is obtained according to each of the first training samples; and the training data set is input into a text processing model for training.

8. The data query large model training method according to claim 7 is characterized in that: The method further comprises the steps of: In response to a user's update operation on the target data warehouse, acquiring update information of the constellation data model corresponding to the target data warehouse; If the update information indicates that a new basic field is added to the constellation data model, determining the new basic field as a new target basic field; Acquire a plurality of third natural query statements; if any of the third natural query statements contains the newly added target basic field, generate a plurality of fourth natural query statements having the same query intent according to the third natural query statement; wherein the fourth natural query statement does not contain the newly added target basic field in the third natural query statement; Constructing a second training sample corresponding to each of the fourth natural query statements according to each of the fourth natural query statements, the first task execution text, and the second query label information corresponding to the fourth natural query statement; wherein the second query label information includes the newly added target basic field contained in the third natural query statement corresponding to the fourth natural query statement; Each of the second training samples is updated into the training data set, and the text processing model is retrained with the updated training data set; the trained text processing model is determined as the data query macro model corresponding to the target data warehouse.

9. A data query method based on the data query large model obtained by any one of the methods of claims 1 to 8, characterized in that: The following steps are involved: Obtaining natural query statements input by users in the query interface of the target data warehouse; Acquire a constellation data model corresponding to the target data warehouse; Determining, according to the natural query statement, several relevant basic fields from the constellation data model as candidate basic fields; Constructing a task processing instruction according to the candidate basic fields, the natural query statement and a preset query task execution text; wherein the query task execution text is used to prompt the data query big model to output query information that meets the query intent of the natural query statement, and the query information is determined according to the candidate basic fields; Inputting the task processing instruction into the data query big model corresponding to the target data warehouse to obtain output query information; According to the query information and the constellation data model, corresponding business data is queried from the target data warehouse.

Images