Training Data Management System and Method Based on Heterogeneous Databases and Large Language Models

Abstract

Training Data Management System and Method Based on Heterogeneous Databases and Large Language Models. The system includes: a heterogeneous database system module, which includes multiple databases supporting different data types; a data management registration center module for managing the node information of the multiple databases and the registered data services; a preposed large language model module for parsing natural language instructions sent by a client and correspondingly generating standard execution statements for describing the data services; and a data service controller module for parsing the standard execution statements and converting them into target execution statements, and based on the target execution statements, calling corresponding data services from the corresponding node information in the data management registration center module to perform corresponding operations on the corresponding databases. This application can improve the efficiency of data services such as querying and collecting specific training data, and greatly reduce the complexity of data management.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and particularly to a training data management system and method based on heterogeneous databases and large language models. Background Art

[0002] Model training data management is a key link in the fields of machine learning and artificial intelligence, which involves multiple aspects such as data collection, storage, processing, verification, use, and maintenance. With the growth of the scale and diversity of training data, higher requirements are put forward for data management. Effective training data management can not only improve the quality, collection efficiency, and accuracy of model training data, but also further save computing power resources and improve the generalization ability of the model.

[0003] Traditional data management relies on database technology. With the emergence of the emerging field of artificial intelligence, a single database cannot meet the needs of training data management. Summary of the Invention

[0004] Based on this, it is necessary to provide a training data management system and method based on heterogeneous databases and large language models for the above technical problems.

[0005] In a first aspect, an embodiment of the present invention provides a training data management system based on heterogeneous databases and large language models, the system comprising:

[0006] A heterogeneous database system module, including multiple databases supporting different data types;

[0007] A data management registration center module for managing node information of the multiple databases and the registered data services;

[0008] A pre-large language model module for parsing natural language instructions sent by a client and correspondingly generating standard execution statements for describing the data services;

[0009] A data service controller module for parsing the standard execution statements and converting them into target execution statements, and calling corresponding data services from the corresponding node information in the data management registration center module based on the target execution statements to perform corresponding operations on the corresponding databases.

[0010] In some embodiments, the pre-large language model module includes a large language model, and the large language model is trained based on input-output statement pairs of multiple rounds of conversations;

[0011] Among them, the input statement of the first-round dialogue is the preliminary description of the data service, and the output statement of the first-round dialogue is the guidance for the user to supplement parameter information; the input statement of the intermediate-round dialogue is the interaction of parameter information supplementation, and the output statement of the intermediate-round dialogue is the feedback confirmation of parameter information supplementation; the output statement of the last-round dialogue is the description of the standard execution statement of the data service.

[0012] In some embodiments, the data service controller module parses the standard execution statement, generates corresponding structured data, and generates the target execution statement based on the structured data.

[0013] In some embodiments, the heterogeneous database system module includes a relational database, a document database, and a file database. A data entry record table is stored in the relational database; the data service includes a data entry service.

[0014] The data service controller module calls the data entry service based on the target execution statement, stores the corresponding training data in the corresponding database, and then stores the association information of the training data in the data entry record table.

[0015] In some embodiments, the association information of the training data includes at least one of data type, data fingerprint, data path, data id, data existence status, entry timestamp, data scale, uploading user, and data topic.

[0016] In some embodiments, a data processing record table is stored in the relational database, and the data service includes a data processing service.

[0017] The data service controller module calls the data processing service or the corresponding data processing program based on the target execution statement. After processing the training data, it stores the processing information of the training data in the data processing record table.

[0018] In some embodiments, the processing information of the training data includes at least one of the current timestamp, source data fingerprint, source data path, processing service path, processing program path, processing configuration file path, and processed data fingerprint.

[0019] In some embodiments, the data service controller module determines whether it is necessary to call the data processing service based on the target execution statement. If it is necessary to call, it checks whether the parameter configuration file corresponding to the data processing service exists in the file database. If not, it uploads the corresponding parameter configuration file to the file database. If so, it uses the data processing service and the parameter configuration file to process the training data, and stores the processing information of the training data in the data processing record table.

[0020] If no call is required, it is detected whether the corresponding data processing program exists in the file database. If not, the corresponding data processing program is uploaded to the file database. If so, it is detected whether the parameter configuration file corresponding to the data processing service exists in the file database. If not, the corresponding parameter configuration file is uploaded to the file database. If so, the training data is processed using the data processing program and the parameter configuration file, and the processing information of the training data is stored in the data processing record table.

[0021] In a second aspect, an embodiment of the present invention provides a training data management method based on heterogeneous databases and large language models. The method includes:

[0022] Construct multiple databases that support different data types;

[0023] Register multiple data services and generate node information for each data service;

[0024] Use a large language model to parse the natural language instructions sent by the client and correspondingly generate standard execution statements for describing the data service;

[0025] Parse the standard execution statements and convert them into target execution statements, and obtain the corresponding node information based on the target execution statements to call the corresponding data service to perform corresponding operations on the corresponding database.

[0026] In a third aspect, an embodiment of the present invention provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the steps described in the second aspect.

[0027] Compared with the prior art, the above method and system use a pre - placed large language model module to parse the natural language instructions sent by the client and correspondingly generate standard execution statements for describing the data service; use a data service controller module to parse the standard execution statements and convert them into target execution statements, and call the corresponding data service from the corresponding node information in the data management registration center module based on the target execution statements to perform corresponding operations on the corresponding database, which can improve the efficiency of data services such as querying and collecting specific training data and greatly reduce the complexity of data management. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic structural diagram of a training data management system based on heterogeneous databases and large language models in an embodiment of the present application;

[0029] Figure 2 It is a schematic flow diagram of the data warehousing service call in an embodiment of the present application;

[0030] Figure 3 It is a schematic flowchart of data processing service invocation in an embodiment of the present application;

[0031] Figure 4 It is a schematic flowchart of data query service invocation in an embodiment of the present application;

[0032] Figure 5 It is a schematic flowchart of data traceability service invocation in an embodiment of the present application;

[0033] Figure 6 It is a schematic flowchart of a training data management method based on heterogeneous databases and large language models in an embodiment of the present application;

[0034] Figure 7 It is a schematic structural diagram of a computer device in an embodiment of the present application. Detailed implementation manners

[0035] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the accompanying drawings required for description in the embodiments. Obviously, the accompanying drawings in the following description are only some examples or embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, the present invention can also be applied to other similar scenarios based on these drawings. Unless obvious from the language context or otherwise stated, the same reference numerals in the figures represent the same structure or operation.

[0036] As shown in the present invention and the claims, unless the context clearly indicates an exceptional situation, words such as "a", "an", "one", and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of the clearly identified steps and elements, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0037] Although the present invention makes various references to certain modules in the system according to the embodiments of the present invention, however, any number of different modules can be used and run on a computing device and / or a processor. The modules are only illustrative, and different aspects of the system and method can use different modules.

[0038] It should be understood that when a unit or module is described as "connected" or "coupled" to other units, modules, or blocks, it may refer to a direct connection or coupling, or communication with other units, modules, or blocks, or there may be intermediate units, modules, or blocks, unless the context clearly indicates otherwise. The term "and / or" used herein may include any and all combinations of one or more of the related listed items.

[0039] As Figure 1As shown in the figure, an embodiment of the present invention provides a training data management system based on a heterogeneous database and a large language model. The system includes: a heterogeneous database system module 102, including multiple databases supporting different data types; a data management registration center module 104, used to manage the node information of the multiple databases and the registered data services; a pre-large language model module 106, used to parse the natural language instructions sent by the client and correspondingly generate standard execution statements for describing the data services; a data service controller module 108, used to parse the standard execution statements and convert them into target execution statements, and call the corresponding data services from the corresponding node information in the data management registration center module based on the target execution statements to perform corresponding operations on the corresponding databases.

[0040] The heterogeneous database system module 102 includes a relational database, a document database, and a file database. The relational database stores a data warehousing record table and a data processing record table.

[0041] The data management registration center module 104 is the basis for the system to call databases and data services and can be built using database software. After each database in the system goes online, it is required to upload the IP address, database structure information, data access path, and access account information to the data management registration center module. The data management registration center module needs to send heartbeat messages regularly to monitor the availability and health status of all registered databases and data services.

[0042] The data warehousing record table stores the association information of training data, including at least one of data type, data fingerprint, data path, data id, data existence status, warehousing timestamp, data scale, uploading user, and data theme.

[0043] Among them, the data type refers to the data type currently stored. If it is a piece of data in a relational database, the type is Record. If it is a document in a document database, the type is Document. If it is a file in a file database, the type is File.

[0044] The data fingerprint is the unique identifier of the training data generated using algorithms such as SHA-256. For a record in a relational database, all fields can be concatenated into a string in the order in the table to generate the corresponding data fingerprint. For documents or binary files, the corresponding data fingerprint can be directly generated using algorithms such as SHA-256.

[0045] The data path indicates the storage location of the training data. For records in a relational database and documents in a document database, the path includes the URL of the corresponding database node, the belonging database, and the belonging data table. For object storage service files in a file database, the data path includes the URL of the service node where the object storage service file is located and the unique URL of the object storage service file.

[0046] The data ID is the unique identifier of the training data in the data warehousing record table. Training data stored in a relational database and a document database both have a unique ID. By combining the data path and the data ID, a unique piece of training data can be located. In a file database, the file can be directly located through the data path, so a data ID is not required.

[0047] The data existence status indicates whether a piece of training data exists, that is, it has not been deleted. If it has been deleted, the field value is No; if it still exists, the field value is Yes. After calling the data service to delete the training data, the data existence status of the training data will be automatically set to No, and the existence time limit of the deleted data warehousing record can be set. If the time limit is exceeded, the corresponding piece of training data in the data warehousing record table will be deleted. This mechanism is for querying all data warehousing records within a certain period after the training data is deleted.

[0048] The warehousing timestamp is the time when the training data is stored in the database.

[0049] The data size refers to the size of the stored training data, which can be the sum of the byte counts of the values of each field.

[0050] The uploading user is the user who uploads the training data in a heterogeneous database.

[0051] The data theme is an abstract generalization of the content of the training data or the field to which the training data belongs. The data theme is an optional field in the data warehousing record table and can be filled in manually when uploading the training data. For documents, the data theme can also be automatically filled by summarizing the document content through a large language model or the like.

[0052] The data processing record table stores the processing information of the training data, including at least one of the current timestamp, the source data fingerprint, the source data path, the processing service path, the processing program path, the processing configuration file path, and the fingerprint of the processed data.

[0053] Among them, the source data fingerprint and the fingerprint of the processed data are crucial when tracing the data lineage.

[0054] The data processing service path, the data processing program path, and the parameter configuration file path are optional values and are set according to their actual existence.

[0055] If it is to call the data processing service registered by the data management registration center module, the data processing service path field is required. If it is a data processing program written by the user himself, he can also call the api interface service provided by this system to call the data processing program.

[0056] The preposed large language model module 106 includes a large language model, which can understand natural language instructions input by humans in natural language and feedback the interpretation of semantics to the user in natural language. If some data service details are missing, such as the setting of specific parameters, the selection of data range, etc., it can be feedback to the user in natural language and guide the user to supplement it completely. After obtaining clear and complete instructions, it can generate standard execution statements according to the instructions. The standard execution statements are, for example, sql statements.

[0057] When preparing training data, the problem of unclear data description needs to be considered. Therefore, it is necessary to prepare multi-round dialogue data, including input-output pairs of multi-round dialogues. The large language model is trained based on the input-output statement pairs of multi-round dialogues. Among them, the input statement of the first round of dialogue is the preliminary description of the data service, and the output statement of the first round of dialogue is the guidance for the user to supplement parameter information; the input statement of the middle-round dialogue is the interaction of parameter information supplement, and the output statement of the middle-round dialogue is the feedback confirmation of parameter information supplement; the output statement of the last round of dialogue is the description of the standard execution statement of the data service.

[0058] Illustrate with examples:

[0059] -input: I need to collect some aviation-related data, and the requirement is newly stored in the database.

[0060] -output: Hello, may I ask if the data storage date is within one year?

[0061] -input: It's okay within two years.

[0062] -output: Okay, may I ask how much data is needed?

[0063] -input: More than 500MB.

[0064] -output: Okay, may I ask if text data is needed?

[0065] -input: Yes.

[0066] -output: May I ask if the downloaded data is stored in your user directory?

[0067] -input: Store it in the C:\Users\ABC\OUTPUT directory.

[0068] -output: {

[0069] Service category: Data query,

[0070] Data theme: Aviation,

[0071] Data type: Text data,

[0072] Data storage date: Less than two years,

[0073] Data scale: 500MB,

[0074] Is it downloaded: Yes,

[0075] Download path: C:\Users\ABC\OUTPUT

[0076] }

[0077] Please confirm whether the description of this data service is correct.

[0078] -input: Correct.

[0079]

[0080] The data service includes data storage service, data query and download service, data processing service, data traceability service, data deletion service, etc. These data services can all be implemented through the training data management system, which can improve the efficiency of data services such as querying and collecting specific training data, and greatly reduce the complexity of data management.

[0081] With the help of the large language model, instructions can be issued for various types of data services in a natural language manner. The standard execution statements processed by the large language model are statements that conform to the standard specifications. It can present the user's task requirements in the form of SQL statements. However, for heterogeneous databases, the SQL statements processed by the large language model cannot be directly run on the actual database nodes. Instead, the standard execution statements processed by the large language model are first parsed in a rule-based manner to generate SQL statements that can be actually executed on the data service nodes or DML statements and DQL statements that conform to other database specifications. The above process is the work that the data service controller module needs to complete.

[0082] For example: Taking the SQL statements output by the above large language model as an example, after being parsed by the data service controller module, the generated structured data is as follows:

[0083]

[0084] Among them, task_list is the description of the current data task. In this example, there is only one data query and download service. service_list is the actual data service executed, which is the specific service URL and the executed SQL statement of the search task. Here, the data path and data ID that meet the requirements are retrieved from the data storage record table first. After retrieving the corresponding data storage location, the actual data needs to be found from the document database. The data service controller module will retrieve the corresponding service URL on the node where the training data is located in the data management registration center module and generate a DML statement suitable for the query syntax of the corresponding database, and add it to service_list.

[0085] As Figure 2 shown, when the data storage service needs to be called, the preposed large language model module 106 parses the natural language instruction sent by the client, and correspondingly generates a standard execution statement for describing the data service. The data service controller module 108 parses the standard execution statement and converts it into a target execution statement, calls the SHA-256 algorithm to generate the data fingerprint corresponding to the data, and based on the target execution statement, calls the data storage service. After storing the corresponding training data in the corresponding database, the association information of the training data is stored in the data storage record table.

[0086] When the data processing service needs to be called, the preposed large language model module 106 parses the natural language instruction sent by the client, and correspondingly generates a standard execution statement for describing the data service. The data service controller module 108 parses the standard execution statement and converts it into a target execution statement, and based on the target execution statement, calls the data processing program corresponding to the data processing service. After processing the training data, the processing information of the training data is stored in the data processing record table.

[0087] As Figure 3 shown, the data service controller module 108 parses the standard execution statement and converts it into a target execution statement. The data service controller module determines whether the data processing service needs to be called based on the target execution statement. If it needs to be called, it checks whether the parameter configuration file corresponding to the data processing service exists in the file database. If not, it uploads the corresponding parameter configuration file to the file database. If so, it uses the data processing service and the parameter configuration file to process the training data, and stores the processing information of the training data in the data processing record table.

[0088] If no call is required, check whether the corresponding data processing program exists in the file database. If not, upload the corresponding data processing program to the file database. If so, check whether the parameter configuration file corresponding to the data processing service exists in the file database. If not, upload the corresponding parameter configuration file to the file database. If so, use the data processing program and the parameter configuration file to process the training data, and store the processing information of the training data in the data processing record table.

[0089] As Figure 4 shown, when the data query service needs to be called, the data service controller module 108 parses the standard execution statement, generates structured data including a task description and initial service call information, searches for the data path of the matching training data in the data storage record table, searches for the data service on the node where the data path is located, generates a DQL statement to query the training data from the corresponding database, and executes the DQL statement on the database.

[0090] As Figure 5 shown, when the data traceability service needs to be called, the data service controller module 108 parses the standard execution statement, generates structured data including a task description and initial service call information, searches for relevant records of the traceability data in the data processing record table, sequentially searches for the processing information of the traceability data, and structurally converts the processing information and returns it to the user.

[0091] Each module in the above training data management system based on heterogeneous databases and large language models can be implemented in whole or in part by software, hardware, and their combinations. The above modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0092] Based on the above training data management system, the present application also proposes a training data management method based on heterogeneous databases and large language models. As Figure 6 shown, the method includes:

[0093] S102, constructing multiple databases that support different data types;

[0094] S104, registering multiple data services and generating node information for each of the data services;

[0095] S106, using a large language model to parse natural language instructions sent by a client and correspondingly generating standard execution statements for describing the data services;

[0096] S108, Parse the standard execution statement and convert it into a target execution statement, obtain corresponding node information based on the target execution statement to call the corresponding data service, and perform corresponding operations on the corresponding database.

[0097] For the specific limitations on the training data management method based on heterogeneous databases and large language models, reference can be made to the limitations on the training data management system in the above text, which will not be elaborated here.

[0098] In one embodiment, the embodiment of the present invention provides a computer device, which may be a server, and its internal structure diagram may be as Figure 7 shown. The computer device includes a processor, a memory, and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it realizes the steps in any of the above embodiments of the training data management method based on heterogeneous databases and large language models.

[0099] Those skilled in the art can understand that Figure 7 the structure shown in

[0100] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above various methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0101] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope described in this specification.

[0102] The above embodiments only represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A training data management system based on heterogeneous databases and large language models, characterized in that: The system comprises: A heterogeneous database system module includes a plurality of databases supporting different data types; the heterogeneous database system module includes a relational database, a document database, and a file database; The data management registration center module is used to manage multiple databases and node information of registered data services; A front-end large language model module is used to parse the natural language instructions sent by the client, and to generate corresponding standard execution statements for describing the data service; the front-end large language model module includes a large language model, and the large language model is trained based on input and output statement pairs of multiple rounds of dialogue; wherein the input statement of the first round of dialogue is a preliminary description of the data service, and the output statement of the first round of dialogue is a guide for the user to supplement parameter information; the input statement of the middle round of dialogue is the interaction of parameter information supplementation, and the output statement of the middle round of dialogue is the feedback confirmation of parameter information supplementation; the output statement of the last round of dialogue is a description of the standard execution statement of the data service; The data service controller module is used to parse the standard execution statement and convert it into a target execution statement, and based on the target execution statement, call the corresponding data service from the corresponding node information in the data management registration center module to perform the corresponding operation on the corresponding database.

2. The system according to claim 1, characterized in that The data service controller module parses the standard execution statement, generates corresponding structured data, and generates the target execution statement based on the structured data.

3. The system according to claim 1, characterized in that The relational database stores a data entry record table; the data service includes a data entry service; The data service controller module calls the data warehousing service based on the target execution statement, stores the corresponding training data in the corresponding database, and then stores the associated information of the training data in the data warehousing record table.

4. The system according to claim 3, characterized in that The associated information of the training data includes at least one of a data type, a data fingerprint, a data path, a data ID, a data existence status, a storage timestamp, a data size, an uploading user, and a data subject.

5. The system according to claim 3, characterized in that The relational database stores a data processing record table, and the data service includes a data processing service; The data service controller module calls the data processing service or the corresponding data processing program based on the target execution statement, and after processing the training data, stores the processing information of the training data in the data processing record table.

6. The system according to claim 5, characterized in that The processing information of the training data includes at least one of a current timestamp, a source data fingerprint, a source data path, a processing service path, a processing program path, a processing configuration file path, and a processed data fingerprint.

7. The system according to claim 5, characterized in that The data service controller module determines whether a data processing service needs to be called based on the target execution statement. If so, it detects whether a parameter configuration file corresponding to the data processing service exists in the file database. If not, it uploads the corresponding parameter configuration file to the file database. If so, it processes the training data using the data processing service and the parameter configuration file, and stores the processing information of the training data in the data processing record table. If no call is required, check whether the corresponding data processing program exists in the file database. If not, upload the corresponding data processing program to the file database. If so, check whether the parameter configuration file corresponding to the data processing service exists in the file database. If not, upload the corresponding parameter configuration file to the file database. If so, use the data processing program and the parameter configuration file to process the training data, and store the processing information of the training data in the data processing record table.

8. A training data management method based on heterogeneous databases and large language models, characterized in that: The method comprises: Build multiple databases that support different data types, including relational databases, document databases, and file databases; Register multiple data services and generate node information of each of the data services; The natural language instructions sent by the client are parsed using a large language model, and a corresponding standard execution statement for describing the data service is generated; the large language model is trained based on input and output statement pairs of multiple rounds of dialogue; wherein the input statement of the first round of dialogue is a preliminary description of the data service, and the output statement of the first round of dialogue is a guide for the user to supplement parameter information; the input statement of the middle round of dialogue is the interaction of parameter information supplementation, and the output statement of the middle round of dialogue is the feedback confirmation of parameter information supplementation; the output statement of the last round of dialogue is a description of the standard execution statement of the data service; The standard execution statement is parsed and converted into a target execution statement, and corresponding node information is acquired based on the target execution statement to call a corresponding data service, so as to perform a corresponding operation on a corresponding database.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to claim 8 are implemented.

Images