Data processing method and device, large language model fine-tuning method and device

Abstract

The present disclosure provides a data processing method and device, relates to the technical field of computers, in particular to the technical field of large models, deep learning and the like. The specific implementation scheme is as follows: obtaining a to-be-searched text and data structure information of a heterogeneous storage system; obtaining a structured query statement and a relationship set based on the to-be-searched text, the data structure information and a large language model, the structured query statement and the relationship set comprising at least one structured query statement and an association relationship between any two structured query statements in the at least one structured query statement; obtaining a data access request of the heterogeneous storage system based on the structured query statement and the relationship set; and querying the heterogeneous storage system by using the data access request to obtain a query result of the to-be-searched text.

Description

Technical Field

[0001] This disclosure relates to the field of computers, specifically to the technical fields of large models and deep learning, and in particular to a data processing method and apparatus, a large language model fine-tuning method and apparatus, electronic devices, computer-readable storage media, and computer program products. Background Technology

[0002] With the rapid development of big data, data querying has become a daily necessity. However, traditional data querying methods generally involve full-text matching of the information to be queried with data in the database to be queried, resulting in low query accuracy. Summary of the Invention

[0003] This disclosure provides a data processing method and apparatus, a large language model fine-tuning method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

[0004] According to the first aspect, a data processing method is provided, which includes: acquiring the data structure information of the text to be queried and the heterogeneous storage system; obtaining a structured query statement and a set of relationships based on the text to be queried, the data structure information, and a large language model, wherein the structured query statement and the set of relationships include at least one structured query statement and the association relationship between any two structured query statements; obtaining a data access request for the heterogeneous storage system based on the structured query statement and the set of relationships; and querying the heterogeneous storage system using the data access request to obtain the query result of the text to be queried.

[0005] According to the second aspect, a method for fine-tuning a large language model is provided. This method includes: acquiring an initial large language model and a training dataset, wherein the training dataset includes at least one training data point, comprising entity segmentation sequences and structured query statements; selecting training data from the training dataset; inputting the training data into the large language model to obtain the structured query statements and relation sets output by the large language model; fine-tuning the large language model based on the training data and the structured query statements and relation sets; and obtaining a fully trained large language model in response to the large language model meeting the training completion conditions.

[0006] According to a third aspect, a data processing apparatus is provided, comprising: an information acquisition unit configured to acquire a text to be searched and data structure information of a heterogeneous storage system; a set obtaining unit configured to obtain a structured query statement and a set of relationships based on the text to be searched, the data structure information, and a large language model, wherein the structured query statement and the set of relationships include at least one structured query statement and an association relationship between any two structured query statements; a request obtaining unit configured to obtain a data access request for the heterogeneous storage system based on the structured query statement and the set of relationships; and a query unit configured to query the heterogeneous storage system using the data access request to obtain query results for the text to be searched.

[0007] According to the fourth aspect, a large language model fine-tuning apparatus is provided, comprising: a data acquisition unit configured to acquire an initial large language model and a training dataset, the training dataset including at least one training data, the training data including: entity segmentation sequences and structured query statements; a selection unit configured to select training data from the training dataset; an input unit configured to input the training data into the large language model to obtain the structured query statements and relation sets output by the large language model; a fine-tuning unit configured to fine-tune the large language model based on the training data and the structured query statements and relation sets; and a model obtaining unit configured to obtain the trained large language model in response to the large language model meeting the training completion conditions.

[0008] According to a fifth aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method as described in any implementation of the first or second aspect.

[0009] According to a sixth aspect, a non-transitory computer-readable storage medium is provided that stores computer instructions for causing a computer to perform the method described in any implementation of the first or second aspect.

[0010] According to a seventh aspect, a computer program product is provided, including a computer program that, when executed by a processor, implements the method as described in either the first or second aspect.

[0011] The data processing method and apparatus provided in the embodiments of this disclosure first obtain the data structure information of the text to be queried and the heterogeneous storage system; second, based on the text to be queried, the data structure information, and a large language model, obtain structured query statements and a set of relationships, wherein the structured query statements and the set of relationships include at least one structured query statement and the association relationship between any two structured query statements; third, based on the structured query statements and the set of relationships, obtain the data access request of the heterogeneous storage system; and finally, use the data access request to query the heterogeneous storage system to obtain the query result of the text to be queried. Therefore, the structured query statements and the set of relationships obtained by analyzing the text to be queried and the data structure information through the large language model can realize the overall access of storage media units in the heterogeneous storage system, improving the uniformity of access to storage media units of various different storage media; since the structured query statements and the set of relationships include the association relationship between any two structured query statements, the query of the text to be queried can be divided into a query plan that queries multiple storage media units, improving the accuracy of the text query.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0014] Figure 1 This is a flowchart of one embodiment of the data processing method according to the present disclosure;

[0015] Figure 2 This is a schematic diagram of a structure for accessing a heterogeneous storage system in this disclosure;

[0016] Figure 3 This is a flowchart of an embodiment of the large language model fine-tuning method disclosed herein;

[0017] Figure 4 This is a schematic diagram of the structure of an embodiment of the data processing apparatus according to the present disclosure;

[0018] Figure 5 This is a schematic diagram of a structure of one embodiment of the large language model fine-tuning device according to the present disclosure;

[0019] Figure 6 This is a block diagram of an electronic device used to implement the data processing method or large language model fine-tuning method of the embodiments of this disclosure. Detailed Implementation

[0020] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0021] In traditional technologies, data querying can be done in two ways: one is by querying data from a database, which requires the business to use specific methods of the backend database for data querying. This has a high learning cost for the business, and the business must accurately understand the storage format of the backend medium and the organization of business data in order to obtain detailed data; otherwise, it is impossible to describe the data query relationship. The other way is to retrieve information from a search engine. However, the way of querying information from a search engine is essentially similar to full-text information retrieval, and the accuracy of the query results is not high. Furthermore, there is no unified data querying method for heterogeneous storage systems of existing databases and search engines.

[0022] To address the shortcomings of traditional technologies, this disclosure proposes a data processing method that enables querying of abnormal storage systems and improves the accuracy of data retrieval. Figure 1 A flow 100 of an embodiment of the data processing method according to this disclosure is shown, the data processing method including the following steps:

[0023] Step 101: Obtain the data structure information of the text to be queried and the heterogeneous storage system.

[0024] In this embodiment, the text to be searched is the text for which data is to be queried. This text can be obtained by performing text analysis on the user request, such as... Figure 2 As shown.

[0025] In this embodiment, the heterogeneous storage system is a heterogeneous underlying storage engine. The heterogeneous storage system can store the data updated by the user and provide the user with data query services.

[0026] In this embodiment, the heterogeneous storage system mainly contains two types of data:

[0027] The first type is basic data that describes the basic information of business data. Basic data includes relevant descriptive information of the data table, as well as descriptions of the data fields of the data table, storage media, and other related information. The amount of basic data is generally not particularly large. Vector indexes need to be created. There are no special restrictions on the storage engine. Any vector index engine can be used, or a vector database can be used directly.

[0028] Another type is the actual data in the business database. Generally, each business database stores a separate data table, and different businesses use different storage media. Based on the storage media, they can be mainly divided into the following types:

[0029] Databases such as MySQL and SQL Server also support general IDs and aggregated indexes, and the data interface can directly support structured query statements.

[0030] Table storage: such as table and HBase, generally supports key-value queries, prefix scans, and data filtering.

[0031] Search engines, such as Elasticsearch, support various inverted index queries. Different search engines generally have their own query interfaces and support full-text search.

[0032] File storage: such as HDFS, AFS, etc., generally serializes all data directly to the remote file system according to a certain serialization data format.

[0033] In this embodiment, the data structure information is the basic data of the heterogeneous storage system. Through the data structure information, the relevant description information of the data table of each storage medium unit in the current heterogeneous storage system, the data field of the data table, and the description of the storage medium related information can be represented to the large language model. This makes it convenient for the structured query statement generated by the large language model to carry the parameters of the corresponding storage medium unit, thereby generating the corresponding access sub-request through the parameters.

[0034] It should be noted that when querying heterogeneous storage systems using the text to be queried, the main purpose is to access the actual data on the different storage media. Therefore, the actual data on each storage media is referred to as a storage media unit. By accessing the storage media unit, the query results of the text to be queried can be obtained.

[0035] In this technical solution, the collection, storage, use, processing, transmission, provision, and disclosure of the text to be searched are performed with authorization and comply with relevant laws and regulations. Information related to the user in the text to be searched is obtained with the user's permission, and data processing related to the user is conducted under confidentiality conditions.

[0036] Step 102: Based on the text to be queried, data structure information, and large language model, obtain the structured query statement and relation set.

[0037] In this embodiment, the structured query statement and the set of relationships include at least one structured query statement and the association relationship between any two structured query statements.

[0038] In this embodiment, since there are multiple storage media units in the heterogeneous storage system, the structured query statement and relation set are a set of different storage media units queried using a unified query method. Each structured query statement corresponds to a storage media unit in the heterogeneous storage system, and each structured query statement contains parameters of the corresponding storage media unit. By identifying the parameters in the structured query statement, the storage media unit corresponding to the structured query statement can be determined.

[0039] In this embodiment, the association between any two structured query statements in the structured query statement and relation set reflects the dependency relationship between the two structured query statements on the heterogeneous storage system. Specifically, the association relationship includes: parallel execution and sequential execution. Parallel execution means that the two structured query statements are executed at the same time. Sequential execution means that the execution result of one structured query statement depends on the execution result of the other structured query statement.

[0040] In this embodiment, the Large Language Model (LLM) is a deep learning model trained using a large amount of text data, capable of generating natural language text or understanding the meaning of language text. LLM can handle various natural language tasks, such as text classification, question answering, and dialogue, and is an important pathway to artificial intelligence.

[0041] In this embodiment, the large language model can parse the text to be queried, convert natural language into structured query statements, and assign each structured query statement to a storage medium unit in a heterogeneous storage system. Therefore, each structured query statement can access the corresponding storage medium unit to obtain the intermediate results of each structured query statement. Based on the relationship between the structured query statements and any two structured query statements in the relation set, the intermediate results are processed to obtain the query result of the text to be queried. This query result is the final result obtained by sequentially combining the intermediate results of the structured query statements and the structured query statements in the relation set. For example, the query text is: "Query information about the province with the highest GDP (Gross Domestic Product) in a certain country in the past two years." Using a large language model, the structured query statements and relation set are obtained as follows: First, a common table expression is used to find the years for the past two years. Then, the total GDP of each province in those two years is calculated. Next, the province with the highest GDP is selected. Finally, detailed information about that province is obtained. In other words, the structured query statements and relation set include four structured query statements: the first statement is used to find the years for the past two years using a common table expression; the second statement is used to calculate the total GDP of each province in those two years; the third statement is used to calculate the total GDP of each province in those two years; and the fourth statement is used to obtain detailed information about that province. The four structured query statements are related as follows: adjacent structured query statements from the first to the fourth are executed sequentially, and the subsequent structured query statement is executed based on the result of the preceding structured query statement. After accessing the corresponding storage medium unit through the first structured query statement, the intermediate result of the first structured query statement is obtained. Based on the intermediate result of the first structured query statement, the corresponding storage medium unit is accessed through the second structured query statement. Based on the above principle, multiple storage medium units are accessed sequentially to finally obtain the query result of the text to be queried.

[0042] Step 103: Based on the structured query statement and relation set, obtain the data access request of the heterogeneous storage system.

[0043] In this embodiment, the data access request is a real request to access the heterogeneous storage system. The data access request has corresponding access parameters, such as data type and data source. After obtaining the access parameters, the heterogeneous storage system reads the real data of the business database stored therein through the access parameters to obtain the query result of the text to be queried.

[0044] In this embodiment, the structured query statement and relation set include at least one structured query statement and the association between any two structured query statements. When the storage media units in the heterogeneous storage system queried through the structured query statement and relation set are all MySQL databases, the data access request of the heterogeneous storage system can be determined directly through the structured query statement and relation set.

[0045] Specifically, step 103 includes: determining the type of each storage medium unit corresponding to each structured query statement based on the parameters of each structured query statement and each structured query statement in the relation set; in response to the fact that the type of each storage medium unit is a MySQL database, converting each structured query statement into a corresponding access sub-request for accessing the MySQL database; determining the execution order of each structured query statement based on the structured query statement and the association relationship in the relation set, determining the sending order of the access sub-requests according to the execution order, and combining the access sub-requests together according to the sending order to obtain the data access request of the heterogeneous storage system.

[0046] like Figure 2 As shown, the structured query execution engine performs semantic and syntactic analysis on the received structured query statements and relation sets to determine the storage medium unit corresponding to each structured query statement. It then generates access sub-requests to access each storage medium unit and combines these sub-requests according to their relationships to obtain the data access requests for the heterogeneous storage system. It should be noted that the storage media of each storage medium unit is different, and therefore the form of the access sub-requests will differ.

[0047] Step 104: Use a data access request to query the heterogeneous storage system to obtain the query results for the text to be queried.

[0048] In this embodiment, the data access request includes at least one access sub-request combined in the order of sending. Step 104 includes sending each access sub-request in the order of sending, and calling the access interface of the corresponding storage medium unit in the heterogeneous storage system through each access sub-request to obtain the query result of the text to be queried. The query result is the result obtained after executing the last access sub-request in the data access request. The access interface is an interface pre-set for each storage medium unit for writing and reading data. Through the access sub-requests of each storage medium unit, data related to the access sub-request can be read from each storage medium unit. The execution entity running on it can send the access sub-requests sequentially in the order of the access sub-requests in the data access request to obtain the query result of the text to be queried.

[0049] The data processing method provided in this disclosure first obtains the data structure information of the text to be queried and the heterogeneous storage system; second, based on the text to be queried, the data structure information, and a large language model, it obtains structured query statements and a set of relationships, wherein the structured query statements and the set of relationships include at least one structured query statement and the association relationship between any two structured query statements, and each structured query statement corresponds to a storage medium unit in the heterogeneous storage system; third, based on the structured query statements and the set of relationships, it obtains a data access request for the heterogeneous storage system; finally, it uses the data access request to query the heterogeneous storage system to obtain the query result of the text to be queried. Therefore, the structured query statements and the set of relationships obtained by analyzing the text to be queried and the data structure information through the large language model can realize the overall access to storage medium units in the heterogeneous storage system, improving the uniformity of access to storage medium units of various different storage media; since the structured query statements and the set of relationships include the association relationship between any two structured query statements, the query of the text to be queried can be divided into a query plan that queries multiple storage medium units, improving the accuracy of the text query.

[0050] In some embodiments of this disclosure, the method further includes: receiving data change information from a heterogeneous storage system; generating new data structure information based on the data change information; and replacing the original data structure information with the new data structure information.

[0051] In this embodiment, data change information refers to changes in basic data in a heterogeneous storage system. When the stored data in the storage medium unit increases or decreases, the increase or decrease is reflected in the basic data and recorded.

[0052] In this embodiment, the heterogeneous storage system typically categorizes and stores incoming data according to user needs, providing basic data description information, data query characteristics, data schema, data volume, and other relevant information. It then selects the most suitable storage medium unit based on the actual characteristics of the business. Once this information is determined, two parts of information classification processing are performed:

[0053] Meta information processing: The basic information of the original meta data before the data enters the system is expanded and retrieved to supplement the full amount of relevant searchable information. This part will then be stored in a vector search engine in a structured manner; or the expanded data will be stored directly.

[0054] Data processing: The incoming data undergoes two rounds of standardization processing: basic processing and personalized advanced information processing. After processing, the data is stored in the corresponding storage media unit.

[0055] In this embodiment, using new data structure information instead of data structure information enables the large language model to determine the changes in basic data in the current heterogeneous storage system, thereby generating structured query statements and relation sets more accurately and improving the accuracy of the generated structured query statements and relation sets.

[0056] The optional implementation provides a data processing method that receives data change information from a heterogeneous storage system; generates new data structure information based on the data change information; and replaces the original data structure information with the new data structure information, thus providing a reliable means to effectively update the data structure information.

[0057] In some embodiments of this disclosure, the above method further includes storing the text to be queried, data structure information, and query results in a heterogeneous storage system.

[0058] In this embodiment, the text to be queried, data structure information, and query results can be stored in a heterogeneous storage system according to their respective types and regions.

[0059] The data processing method provided in this embodiment stores the text to be queried, data structure information, and query results in a heterogeneous storage system. The storage of multiple types of information in the heterogeneous storage system improves the diversity of data storage in the heterogeneous storage system.

[0060] In some optional implementations of this disclosure, the above-mentioned acquisition of the text to be searched and the data structure information of the heterogeneous storage system includes: acquiring the text to be searched; acquiring the original data stored by class in the heterogeneous storage system; and obtaining the data structure information of the heterogeneous storage system based on the text to be searched and the original data.

[0061] In this optional implementation, the original data consists of all the basic data in the heterogeneous storage system; the text to be searched is directly matched with the original data to obtain the data in the original data that matches the text to be searched, which is the data structure information of the heterogeneous storage system.

[0062] This optional implementation provides a method for obtaining the text to be searched and the data structure information of the heterogeneous storage system. It obtains the text to be searched; obtains the original data stored by class in the heterogeneous storage system; and obtains the data structure information of the heterogeneous storage system based on the text to be searched and the original data, thereby improving the accuracy of obtaining the data structure information.

[0063] In some embodiments of this disclosure, obtaining the data structure information of the heterogeneous storage system based on the text to be searched and the original data includes: vectorizing the text to be searched to obtain a text vector; vectorizing the original data to obtain an original vector; matching the text vector with the original vector to obtain a matching vector in the original vector; and using the original data corresponding to the matching vector as the data structure information.

[0064] In this optional implementation, vectorizing and matching the text to be searched and the original data can make the obtained data structure information more semantically similar to the text to be searched, thereby improving the accuracy of the obtained data structure information.

[0065] This optional implementation provides a method for obtaining data structure information of a heterogeneous storage system. It involves vectorizing the text to be searched to obtain a text vector; vectorizing the original data to obtain an original vector; matching the text vector with the original vector to obtain a matching vector within the original vector; and using the original data corresponding to the matching vector as data structure information. This vector retrieval method improves the accuracy of obtaining the data structure information.

[0066] In some optional implementations of this disclosure, obtaining the structured query statement and relation set based on the query text, data structure information, and large language model includes: sending the query text, data structure information, and query task splitting prompts to the large language model to obtain the structured query statement and relation set output by the large language model. The query task splitting prompts are used to prompt the large language model to split the query task of the query text based on the query text and data structure information, and generate the structured query statement and relation set for querying heterogeneous storage systems.

[0067] In this optional implementation, the structured query statement in the structured query statement and relation set corresponds to a storage medium unit. The large language model analyzes the query steps of the text to be queried, determines the storage medium unit of the heterogeneous storage system that needs to be queried at each step, obtains the structured query statement for each step based on the data structure information corresponding to the storage medium unit, and generates the association relationship between any two structured query statements to obtain the structured query statement and relation set.

[0068] The method for generating recommended videos provided by this optional implementation sends the text to be searched, data structure information, and query task decomposition prompts to a large language model, thereby obtaining a structured query statement and relation set output by the large language model, which improves the accuracy of obtaining the structured query statement and relation set.

[0069] Optionally, the above-mentioned process of obtaining a structured query statement and relation set based on the query text, data structure information, and large language model includes: dividing the query text into topic clauses to obtain multiple topic clauses; inputting the multiple topic clauses and data structure information into the large language model to obtain the structured query statement and relation set output by the large language model; and using query task splitting prompt words to prompt the large language model to split the query task of the query text based on the query text and data structure information to generate a structured query statement and relation set for querying heterogeneous storage systems.

[0070] In some optional implementations of this disclosure, obtaining the data access request of the heterogeneous storage system based on the structured query statement and the relation set includes: performing syntactic and semantic parsing on the structured query statement and the relation set to obtain the syntax tree of each structured query statement and the association relationship between any two syntax trees; determining the execution subgraph of the heterogeneous storage system based on the association relationship and each syntax tree; and obtaining the data access request of the heterogeneous storage system based on the execution subgraph.

[0071] In this optional implementation, each structured query statement in the structured query statement and relation set is used to query a storage medium unit and obtain intermediate results; a corresponding syntax tree can be generated for each structured query statement.

[0072] In this optional implementation, the execution subgraph of the heterogeneous storage system is a topology diagram of access to each storage medium unit in the heterogeneous storage system. The access sub-requests for accessing each storage medium unit can be determined through the execution subgraph, and all access sub-requests can be combined to obtain a data access request.

[0073] This optional implementation provides a method for obtaining data access requests by performing syntactic and semantic parsing on structured query statements and relation sets to obtain the syntax tree of each structured query statement and the association between any two syntax trees; based on the association and each syntax tree, the execution subgraph of the heterogeneous storage system is determined; based on the execution subgraph, the data access request of the heterogeneous storage system is obtained, thereby mapping the syntax tree to the execution subgraph of the heterogeneous storage system, and based on the execution subgraph, the data access request of the heterogeneous storage system is determined, providing a reliable implementation method for data access requests.

[0074] In some optional implementations of this disclosure, the above-mentioned determination of the execution subgraph of the heterogeneous storage system based on the association relationship and each syntax tree includes: mapping each node of each syntax tree to the first graph node of the corresponding storage medium unit in the heterogeneous storage system based on the structured query statement and the set of relationships, wherein the first graph node is a node pre-set for the storage medium unit; connecting the first graph nodes of each storage medium unit based on the node relationship between nodes in each syntax tree to obtain the subgraph unit of each storage medium unit; and connecting the subgraph units of each storage medium unit based on the association relationship to obtain the execution subgraph.

[0075] In this optional implementation, the first graph node is a node generated from the basic syntax elements in the storage medium unit. The basic syntax elements of the storage medium unit are the basic units for generating access to the storage medium unit. For example, in the transformation from the syntax tree to the first graph node of ElasticSearch, the SELECT: in the syntax tree is converted into the _search request of ElasticSearch, where _search is the basic syntax element of ElasticSearch.

[0076] In this optional implementation, the execution subgraph is a structural diagram for executing access to a heterogeneous storage system. The execution subgraph includes multiple subgraph units, each of which corresponds to a storage medium unit. The connecting edges between the subgraph units in the execution subgraph are used to reflect the relationship between them.

[0077] The optional implementation provides a method for obtaining the execution subgraph. Based on structured query statements and relation sets, it maps each node of each syntax tree to the first graph node of the corresponding storage medium unit in the heterogeneous storage system. The first graph node is a node pre-set for the storage medium unit. Based on the node relationships between nodes in each syntax tree, it connects the first graph nodes of each storage medium unit to obtain the subgraph unit of each storage medium unit. Based on the association relationship, it connects the subgraph units of each storage medium unit to obtain the execution subgraph. This provides a reliable implementation method for obtaining the execution subgraph and improves the accuracy of the execution subgraph.

[0078] Optionally, the above-mentioned determination of the execution subgraph of the heterogeneous storage system based on the association relationship and each syntax tree further includes: receiving the user's insertion information; determining the connection relationship between the insertion information and each syntax tree; mapping the insertion information to the user's second graph node, where the second graph node is a node pre-set for user input; and connecting the second graph node and the first graph node based on the connection relationship.

[0079] In some optional implementations of this disclosure, obtaining the data access request of the heterogeneous storage system based on the execution subgraph includes: determining the unit execution order of each subgraph unit based on the connection relationship of the subgraph units in the execution subgraph; determining the parameters in the data access request based on the graph nodes in each subgraph unit; determining the parameter execution order based on the node relationship of the graph nodes in each subgraph unit; and obtaining the data access request of the heterogeneous storage system based on the unit execution order, the parameters, and the parameter execution order.

[0080] In this optional implementation, the unit execution order is the access order of the heterogeneous storage system; the parameters in the data access request include: the parameters of each access sub-request, which are necessary parameters for accessing each storage medium unit. The parameters of the access sub-requests are different for different types of storage media; in order to implement the data access request, the sending order of each access sub-request can be determined by the unit execution order; based on the parameters and the parameter order, the access sub-requests for accessing each storage medium unit can be effectively filled.

[0081] The optional implementation provides a method for obtaining data access requests that determines the execution order of each subgraph unit based on the connection relationship of the subgraph units in the execution subgraph; determines the parameters in the data access request based on the graph nodes in each subgraph unit; and determines the parameter execution order based on the node relationship of the graph nodes in each subgraph unit, thereby improving the accuracy of obtaining data access requests.

[0082] In some optional implementations of this disclosure, obtaining the data access request of the heterogeneous storage system based on the structured query statement and relation set includes: parsing the structured query statement and relation set to obtain the relational algebra expression of each structured query statement; determining the execution subgraph of the heterogeneous storage system based on the relational algebra expression; and obtaining the data access request of the heterogeneous storage system based on the execution subgraph.

[0083] In this optional implementation, relational algebra expressions represent the parameters and parameter relationships in each structured query statement in the form of functional relations. One relational algebra expression represents one structured query statement, and another relational algebra expression can represent the association relationship between multiple structured query statements.

[0084] This optional implementation provides a method for obtaining data access requests by parsing structured query statements and relation sets to obtain relational algebra expressions for each structured query statement; determining the execution subgraph of the heterogeneous storage system based on the relational algebra expressions; and obtaining the data access requests of the heterogeneous storage system based on the execution subgraph. Thus, the relational algebra expressions are mapped to the execution subgraph of the heterogeneous storage system, and the data access requests of the heterogeneous storage system are determined based on the execution subgraph, providing another reliable implementation method for data access requests.

[0085] Based on this, this disclosure proposes a method for fine-tuning large language models. Figure 3 A flowchart 300 is shown as an embodiment of the large language model fine-tuning method according to the present disclosure, which includes the following steps:

[0086] Step 301: Obtain the initial large language model and training dataset.

[0087] In this embodiment, the execution entity running on the large language model fine-tuning method can obtain the training dataset and the initial large language model in various ways. For example, the execution entity can obtain the training dataset and the initial large language model stored in the database server through a wired or wireless connection. Alternatively, the user can obtain the training dataset and the initial large language model collected by the terminal by communicating with the terminal.

[0088] Here, the training dataset may include at least one training data set, which includes entity segmentation sequences and structured query statements. The sample graph structure is used to reflect the structured query statements and the relationships between multiple structured query statements.

[0089] It should be noted that the training dataset is a collection of data obtained based on a pre-defined database. This database contains multiple table structures, such as table name: employees, columns: id (integer), name (string), age (integer), department (string), etc. The process of obtaining the training dataset is as follows:

[0090] Obtain an initial dataset containing at least one set of initial data, which includes: natural language queries paired with SQL statements. For example: Natural language: "Get the names of all employees working in the marketing department." SQL: SELECT nameFROM employees WHERE department='marketing department'; For the natural language query, perform word segmentation and stop word removal (which may not always be necessary in Chinese processing) to obtain an entity word segmentation sequence. Perform format checks on the SQL statements to obtain structured query statements, where format checks include: monitoring for spaces and capitalization in the statements to ensure consistent statement format.

[0091] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, and public disclosure of the training dataset and the initial large language model are performed after authorization and comply with relevant laws and regulations.

[0092] In this embodiment, the initial large language model can be a pre-trained general large language model. For example, when a question is input into the initial large language model, the initial large language model only outputs a structured query statement to query the database. Only one database can be queried through this structured query statement.

[0093] In this embodiment, the initial large language model can use a Seq2Seq model architecture, in which the encoder part processes natural language input and the decoder part generates SQL statements.

[0094] Optionally, an attention mechanism can be added to the decoder so that the model can refer to specific parts of the natural language input when generating SQL.

[0095] Step 302: Select training data from the training dataset.

[0096] In this application, the method and quantity of selecting training data from the image sample set are not limited, nor are the number of training iterations for fine-tuning the large language model limited. For example, in a single fine-tuning training, multiple consecutive training datasets can be randomly selected, and the initial network loss value of the large language model can be calculated using the selected training datasets to adjust the parameters of the large language model.

[0097] Step 303: Input the training data into the large language model to obtain the structured query statements and relation sets output by the large language model.

[0098] In this embodiment, while inputting the training data into the large language model, it is necessary to input differential task splitting prompt words into the large language model, prompting the large language model to split the entity word segmentation sequence in the training data according to the preset database data structure information to obtain structured query statements and relation sets.

[0099] Step 304: Fine-tune the large language model based on the training data, structured query statements, and relation sets.

[0100] In this embodiment, the execution entity can select entity segmentation sequences from the training data obtained in step 302, and execute the training steps from step 303 to step 304 to complete a fine-tuning training of a large language model.

[0101] In this embodiment, during each fine-tuning training of the large language model, entity segmentation sequences are selected from the training data and input into the large language model. Based on the structured query statements and relation sets output by the large language model and the structured query statements in the training data, the network loss value of the large language model is calculated.

[0102] In this embodiment, the loss function used to calculate the network loss value of the large language model can be the cross-entropy loss function. The cross-entropy loss function measures the degree of difference between two different probability distributions in the same random variable, which in machine learning is represented as the difference between the true probability distribution and the predicted probability distribution. The smaller the value of the cross-entropy loss function, the better the prediction effect of the large language model.

[0103] Step 305: In response to the large language model meeting the training completion conditions, the trained large language model is obtained.

[0104] In this embodiment, the network loss value of the large language model can be used to detect whether the large language model meets the training completion condition. After the large language model meets the training completion condition, the trained large language model is obtained.

[0105] In this embodiment, the training completion condition includes: the network loss value of the large language model is less than a first loss threshold. The first loss threshold can be determined based on specific training requirements; for example, the first loss threshold may be 0.01.

[0106] Optionally, in this embodiment, in response to the large language model not meeting the training completion conditions, the relevant parameters in the large language model are adjusted to make the network loss value of the large language model converge, and based on the adjusted large language model, the above training steps 302-304 are continued.

[0107] In this embodiment, during the fine-tuning training of the large language model, the execution entity running on the large language model fine-tuning method records the loss and accuracy on the validation set in each iteration of the fine-tuning training stage, identifies signs of overfitting, and increases the learning rate if the large language model converges slowly; if the overfitting phenomenon of the large language model is serious, the learning rate is decreased.

[0108] In this embodiment, the performance metrics of the trained large language model can be evaluated using validation and test sets. Specifically, the performance metrics can mainly use two types: accuracy and the syntactic correctness of the structured query statements.

[0109] In this embodiment, after obtaining the trained large language model, it can be deployed to the production environment and made available for user queries via an API (Application Programming Interface). User input and structured query statements and relation sets generated by the model are continuously collected, and the accuracy of the model's output is monitored.

[0110] Optionally, the training dataset can be updated periodically to retrain or fine-tune the trained large language model in order to improve its performance.

[0111] The large language model fine-tuning method provided in this disclosure first obtains an initial large language model and a training dataset. The training dataset includes at least one training data point, comprising entity segmentation sequences and structured query statements. Second, training data is selected from the training dataset. Third, the training data is input into the large language model to obtain the structured query statements and relation set output by the large language model. Then, the large language model is fine-tuned based on the training data and the structured query statements and relation set. Finally, in response to the large language model meeting the training completion conditions, the trained large language model is obtained. Thus, by combining structured query statements and relations into a prediction graph structure, the graph relationships between multiple structured query statements can be effectively represented, providing a reliable foundation for large language model fine-tuning and improving the accuracy and reliability of large language model training.

[0112] Further reference Figure 4 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a data processing apparatus, which is similar to... Figure 1 Corresponding to the method embodiments shown, the device can be specifically used in various electronic devices.

[0113] like Figure 4 As shown, the data processing device 400 provided in this embodiment includes: an information acquisition unit 401, a set acquisition unit 402, a request acquisition unit 403, and a query unit 404. The information acquisition unit 401 can be configured to acquire the text to be searched and the data structure information of the heterogeneous storage system. The set acquisition unit 402 can be configured to obtain a structured query statement and a set of relationships based on the text to be searched, the data structure information, and a large language model. The structured query statement and the set of relationships include at least one structured query statement and the association relationship between any two structured query statements. Each structured query statement corresponds to a storage medium unit in the heterogeneous storage system. The request acquisition unit 403 can be configured to obtain a data access request for the heterogeneous storage system based on the structured query statement and the set of relationships. The query unit 404 can be configured to query the heterogeneous storage system using the data access request to obtain the query results for the text to be searched.

[0114] In this embodiment, the specific processing and technical effects of the information acquisition unit 401, the collection acquisition unit 402, the request acquisition unit 403, and the query unit 404 in the data processing device 400 can be found in reference respectively. Figure 1 The relevant descriptions of steps 101, 102, 103, and 104 in the corresponding embodiments will not be repeated here.

[0115] In some optional implementations of this embodiment, the apparatus 400 further includes: an update unit (not shown in the figure), which is configured to receive data change information from a heterogeneous storage system; generate new data structure information based on the data change information; and replace the data structure information with the new data structure information.

[0116] In some optional implementations of this embodiment, the above-mentioned device further includes a storage unit (not shown in the figure), which is configured to store the text to be searched, data structure information, and query results in a heterogeneous storage system.

[0117] In some optional implementations of this embodiment, the information acquisition unit 401 is further configured to: acquire the text to be searched; acquire the original data stored by category in the heterogeneous storage system; and obtain the data structure information of the heterogeneous storage system based on the text to be searched and the original data.

[0118] In some optional implementations of this embodiment, the information acquisition unit 401 is further configured to: perform vectorization processing on the text to be searched to obtain a text vector; perform vectorization processing on the original data to obtain an original vector; match the text vector with the original vector to obtain a matching vector in the original vector, and use the original data corresponding to the matching vector as data structure information.

[0119] In some optional implementations of this embodiment, the above-mentioned set-obtaining unit 402 is configured to: send the text to be searched, data structure information, and query task splitting prompt words to the large language model to obtain the structured query statement and relation set output by the large language model. The query task splitting prompt words are used to prompt the large language model to split the query task of the text to be searched based on the text to be searched and data structure information, and generate the structured query statement and relation set for querying the heterogeneous storage system.

[0120] In some optional implementations of this embodiment, the above-mentioned request obtaining unit 403 is configured to: perform syntactic and semantic parsing on the structured query statements and relation sets to obtain the syntax tree of each structured query statement and the association relationship between any two syntax trees; determine the execution subgraph of the heterogeneous storage system based on the association relationship and each syntax tree; and obtain the data access request of the heterogeneous storage system based on the execution subgraph.

[0121] In some optional implementations of this embodiment, the request obtaining unit 403 is further configured to: map each node of each syntax tree to a first graph node of the corresponding storage medium unit in the heterogeneous storage system based on the structured query statement and relation set, wherein the first graph node is a node pre-set for the storage medium unit; connect the first graph nodes of each storage medium unit based on the node relationship between nodes in each syntax tree to obtain the subgraph unit of each storage medium unit; and connect the subgraph units of each storage medium unit based on the association relationship to obtain the execution subgraph.

[0122] In some optional implementations of this embodiment, the request obtaining unit 403 is further configured to: determine the unit execution order of each subgraph unit based on the connection relationship of the subgraph units in the execution subgraph; determine the parameters in the data access request based on the graph nodes in each subgraph unit; determine the parameter execution order of the parameters based on the node relationship of the graph nodes in each subgraph unit; and obtain the data access request of the heterogeneous storage system based on the unit execution order, the parameters, and the parameter execution order.

[0123] In some optional implementations of this disclosure, the above-mentioned request obtaining unit 403 is configured to: parse the structured query statements and relation sets to obtain the relational algebra expressions of each structured query statement; determine the execution subgraph of the heterogeneous storage system based on the relational algebra expressions; and obtain the data access request of the heterogeneous storage system based on the execution subgraph.

[0124] The data processing apparatus provided in the embodiments of this disclosure firstly involves an information acquisition unit 401 acquiring the text to be searched and the data structure information of a heterogeneous storage system; secondly, a set acquisition unit 402, based on the text to be searched, the data structure information, and a large language model, obtaining a structured query statement and a set of relationships, wherein the structured query statement and the set of relationships include at least one structured query statement and the association relationship between any two structured query statements, and each structured query statement corresponds to a storage medium unit in the heterogeneous storage system; thirdly, a request acquisition unit 403, based on the structured query statement and the set of relationships, obtains a data access request for the heterogeneous storage system; and finally, a query unit 404 uses the data access request to query the heterogeneous storage system to obtain the query result of the text to be searched. Therefore, by analyzing the text and data structure information through a large language model, the structured query statements and relation sets obtained can achieve overall access to storage media units in heterogeneous storage systems, improving the uniformity of access to storage media units of various different storage media. Since the structured query statements and relation sets include the association between any two structured query statements, the query of the text to be queried can be divided into query plans that query multiple storage media units, improving the accuracy of the text query.

[0125] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0126] Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a large language model fine-tuning device, which is similar to... Figure 3 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0127] like Figure 5 As shown, the large language model fine-tuning device 500 provided in this embodiment includes: a data acquisition unit 501, a selection unit 502, an input unit 503, a fine-tuning unit 504, and a model acquisition unit 505. The data acquisition unit 501 can be configured to acquire an initial large language model and a training dataset. The training dataset includes at least one training data set, which includes entity segmentation sequences and structured query statements. The selection unit 502 can be configured to select training data from the training dataset. The input unit 503 can be configured to input the training data into the large language model to obtain the structured query statements and relation sets output by the large language model. The fine-tuning unit 504 can be configured to fine-tune the large language model based on the training data and the prediction graph structure. The model acquisition unit 505 can be configured to obtain the trained large language model in response to the large language model meeting the training completion conditions.

[0128] In this embodiment, the specific processing and technical effects of the data acquisition unit 501, selection unit 502, input unit 503, fine-tuning unit 504, and model acquisition unit 505 in the large language model fine-tuning device 500 can be found in the following references. Figure 3 The relevant descriptions of steps 301, 302, 303, 304, and 305 in the corresponding embodiments will not be repeated here.

[0129] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0130] Figure 6A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0131] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded from storage unit 608 into random access memory (RAM) 603. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0132] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0133] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as data processing methods or large language model fine-tuning methods. For example, in some embodiments, the data processing methods or large language model fine-tuning methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the data processing methods or large language model fine-tuning methods described above can be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform a data processing method or a large language model fine-tuning method by any other suitable means (e.g., by means of firmware).

[0134] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0135] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus or large language model fine-tuning apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0136] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0137] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0138] The systems and technologies described herein can be implemented in computing systems that include back-end components (e.g., as a data server), or computing systems that include middleware components (e.g., an information server), or computing systems that include front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0139] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0140] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0141] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data processing method, the method comprising: Obtain the data structure information of the text to be queried and the heterogeneous storage system; Based on the text to be queried, the data structure information, and the large language model, a set of structured query statements and relationships is obtained. This set includes at least one structured query statement and the association between any two structured query statements. The set of structured query statements and relationships uses a unified query method to query different storage media units. Each structured query statement corresponds to one type of storage media unit in the heterogeneous storage system, and each structured query statement contains parameters for that storage media unit. The association reflects the dependency relationship between the two structured query statements querying the heterogeneous storage system. Based on the structured query statement and relation set, a data access request for the heterogeneous storage system is obtained; the data access request includes at least one access sub-request combined in the order of sending; including: performing semantic and syntactic analysis on the received structured query statement and relation set, determining the storage medium unit corresponding to each structured query statement, generating the access sub-request to access each storage medium unit, and combining the access sub-requests according to the association relationship to obtain the data access request for the heterogeneous storage system; The data access request is used to query the heterogeneous storage system to obtain the query result of the text to be queried; the query result is the result obtained after executing the last access sub-request in the data access request.

2. The method according to claim 1, further comprising: Receive data change information from the heterogeneous storage system; Based on the data change information, new data structure information is generated; The new data structure information is used to replace the original data structure information.

3. The method according to claim 1, further comprising: The text to be queried, the data structure information, and the query results are stored in the heterogeneous storage system.

4. The method according to any one of claims 1-3, wherein, The acquisition of the data structure information of the text to be searched and the heterogeneous storage system includes: Get the text to be searched; Obtain the raw data stored by category in a heterogeneous storage system; Based on the text to be searched and the original data, the data structure information of the heterogeneous storage system is obtained.

5. The method according to claim 4, wherein, The process of obtaining the data structure information of the heterogeneous storage system based on the text to be searched and the original data includes: The text to be searched is vectorized to obtain a text vector; The original data is vectorized to obtain the original vector; The text vector is matched with the original vector to obtain the matching vector in the original vector, and the original data corresponding to the matching vector is used as data structure information.

6. The method according to any one of claims 1-3, wherein, The process of obtaining the structured query statement and relation set based on the text to be queried, the data structure information, and the large language model includes: The text to be searched, the data structure information, and query task splitting prompts are sent to the large language model to obtain the structured query statements and relation sets output by the large language model. The query task splitting prompts are used to prompt the large language model to split the query task of the text to be searched based on the text to be searched and the data structure information, and generate structured query statements and relation sets for querying heterogeneous storage systems.

7. The method according to any one of claims 1-3, wherein, The process of obtaining the data access request for the heterogeneous storage system based on the structured query statement and the set of relationships includes: The structured query statements and relation sets are parsed using syntax and semantics to obtain the syntax tree of each structured query statement and the association between any two syntax trees; Based on the aforementioned relationships and each syntax tree, the execution subgraph of the heterogeneous storage system is determined; Based on the execution subgraph, the data access request of the heterogeneous storage system is obtained.

8. The method according to claim 7, wherein, The process of determining the execution subgraph of the heterogeneous storage system based on the association relationships and each syntax tree includes: Based on the structured query statement and relation set, each node of each syntax tree is mapped to the first graph node of the corresponding storage medium unit in the heterogeneous storage system. The first graph node is a node pre-set for the storage medium unit. Based on the node relationships between nodes in each syntax tree, the first graph nodes of each storage medium unit are connected to obtain the subgraph units of each storage medium unit. Based on the aforementioned relationships, subgraph units of each storage medium unit are connected to obtain an execution subgraph.

9. The method according to claim 8, wherein, The process of obtaining the data access request for the heterogeneous storage system based on the execution subgraph includes: Based on the connection relationship of the subgraph units in the execution subgraph, the unit execution order of each subgraph unit is determined; Based on the graph nodes in each subgraph unit, determine the parameters in the data access request; Based on the node relationships of the graph nodes in each subgraph unit, the parameter execution order is determined. Based on the unit execution order, the parameters, and the parameter execution order, the data access request of the heterogeneous storage system is obtained.

10. The method according to any one of claims 1-3, wherein, The process of obtaining the data access request for the heterogeneous storage system based on the structured query statement and the set of relationships includes: Parse the structured query statements and relation set to obtain the relational algebra expressions for each structured query statement; Based on the relational algebra expression, the execution subgraph of the heterogeneous storage system is determined; Based on the execution subgraph, the data access request of the heterogeneous storage system is obtained.

11. A method for fine-tuning a large language model, the method comprising: Obtain an initial large language model and training dataset, wherein the training dataset includes at least one training data, which includes: entity segmentation sequences and structured query statements; Select training data from the training dataset; The training data is input into the large language model to obtain the structured query statements and relation sets output by the large language model. Based on the training data and the structured query statements and relation sets, the large language model is fine-tuned and trained. In response to the large language model satisfying the training completion condition, a trained large language model is obtained; The trained large language model is used in the data processing method of claim 1.

12. A data processing apparatus, the apparatus comprising: The information acquisition unit is configured to acquire the text to be searched and the data structure information of the heterogeneous storage system; The set of units is configured to obtain structured query statements and relation sets based on the text to be queried, the data structure information, and the large language model. The structured query statements and relation sets include at least one structured query statement and the association relationship between any two structured query statements. The structured query statements and relation sets are sets that query different storage media units using a unified query method. Each structured query statement corresponds to one type of storage media unit in the heterogeneous storage system, and each structured query statement contains parameters for the corresponding storage media unit. The association relationship reflects the dependency relationship between the two structured query statements querying the heterogeneous storage system. The request receiving unit is configured to obtain a data access request for the heterogeneous storage system based on the structured query statement and the set of relationships. The data access request includes at least one access sub-request combined in the order of sending. The process includes: performing semantic and syntactic analysis on the received structured query statement and the set of relationships to determine the storage medium unit corresponding to each structured query statement, generating the access sub-request to access each storage medium unit, and combining the access sub-requests according to the association relationship to obtain the data access request for the heterogeneous storage system. The query unit is configured to query the heterogeneous storage system using the data access request to obtain the query result of the text to be queried; the query result is the result obtained after executing the last access sub-request in the data access request.

13. The apparatus of claim 12, further comprising: The update unit is configured to receive data change information from the heterogeneous storage system; Based on the data change information, new data structure information is generated; The new data structure information is used to replace the original data structure information.

14. The apparatus of claim 12, further comprising: The storage unit is configured to store the text to be queried, the data structure information, and the query result in the heterogeneous storage system.

15. The apparatus according to any one of claims 12-14, wherein, The information acquisition unit is further configured to: acquire the text to be searched; acquire the original data stored by category in the heterogeneous storage system; and obtain the data structure information of the heterogeneous storage system based on the text to be searched and the original data.

16. The apparatus according to claim 15, wherein, The information acquisition unit is further configured to: perform vectorization processing on the text to be searched to obtain a text vector; perform vectorization processing on the original data to obtain an original vector; match the text vector with the original vector to obtain a matching vector in the original vector, and use the original data corresponding to the matching vector as data structure information.

17. The apparatus according to any one of claims 12-14, wherein, The set-obtaining unit is configured to: send the text to be searched, the data structure information, and query task splitting prompts to the large language model to obtain the structured query statements and relation sets output by the large language model. The query task splitting prompts are used to prompt the large language model to split the query task of the text to be searched based on the text to be searched and the data structure information, and generate structured query statements and relation sets for querying heterogeneous storage systems.

18. The apparatus according to any one of claims 12-14, wherein, The request receiving unit is configured to: perform syntactic and semantic parsing on the structured query statements and relation set to obtain the syntax tree of each structured query statement and the association relationship between any two syntax trees; and determine the execution subgraph of the heterogeneous storage system based on the association relationship and each syntax tree. Based on the execution subgraph, the data access request of the heterogeneous storage system is obtained.

19. The apparatus according to claim 18, wherein, The request receiving unit is further configured to: based on the structured query statement and relation set, map each node of each syntax tree to the first graph node of the corresponding storage medium unit in the heterogeneous storage system, wherein the first graph node is a node pre-set for the storage medium unit; Based on the node relationships between nodes in each syntax tree, the first graph nodes of each storage medium unit are connected to obtain the subgraph units of each storage medium unit. Based on the aforementioned relationships, subgraph units of each storage medium unit are connected to obtain an execution subgraph.

20. The apparatus according to claim 19, wherein, The request receiving unit is further configured to: determine the unit execution order of each subgraph unit based on the connection relationship of the subgraph units in the execution subgraph; determine the parameters in the data access request based on the graph nodes in each subgraph unit; and determine the parameter execution order of the parameters based on the node relationship of the graph nodes in each subgraph unit. Based on the unit execution order, the parameters, and the parameter execution order, the data access request of the heterogeneous storage system is obtained.

21. The apparatus according to any one of claims 12-14, wherein, The request receiving unit is configured to: parse the structured query statement and the set of relations to obtain the relational algebra expressions for each structured query statement; Based on the relational algebra expression, the execution subgraph of the heterogeneous storage system is determined; Based on the execution subgraph, the data access request of the heterogeneous storage system is obtained.

22. A large language model fine-tuning device, the device comprising: The data acquisition unit is configured to acquire an initial large language model and a training dataset, the training dataset including at least one training data, the training data including: entity segmentation sequences and structured query statements; The selection unit is configured to select training data from the training dataset; The input unit is configured to input the training data into the large language model to obtain the structured query statement and relation set output by the large language model; The fine-tuning unit is configured to fine-tune the large language model based on the training data and the structured query statements and relation sets. The model obtains units, which are configured to respond to the large language model satisfying the training completion condition, to obtain a trained large language model. The trained large language model is used in the data processing method of claim 1.

23. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-11.

24. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-11.

25. A computer program product comprising a computer program that, when executed by a processor, implements the method of any one of claims 1-11.

Images