Heterogeneous database and large language model-based training data management system and method

Abstract

The invention discloses a training data management system and method based on a heterogeneous database and a large language model, and the system comprises a heterogeneous database system module which comprises a plurality of databases supporting different data types; the data management registration center module is used for managing the plurality of databases and the node information of the registered data service; the front large language model module is used for analyzing a natural language instruction sent by a client and correspondingly generating a standard execution statement for describing the data service; and the data service controller module is used for analyzing the standard execution statement and converting the standard execution statement into a target execution statement, and calling a corresponding data service from the corresponding node information in the data management registration center module based on the target execution statement so as to execute a corresponding operation on a corresponding database. According to the method, the efficiency of querying and collecting data services such as specific training data can be improved, and the complexity of data management is greatly reduced.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a training data management system and method based on heterogeneous databases and large language models. Background Art

[0002] Model training data management is a key link in the field of machine learning and artificial intelligence. It involves data collection, storage, processing, verification, use and maintenance. With the growth of training data scale and diversity, higher requirements are placed on data management. Effective training data management can not only improve the quality, collection efficiency and accuracy of model training data, but also further save computing resources and improve the generalization ability of the model.

[0003] Traditional data management relies on database technology. With the emergence of the emerging field of artificial intelligence, a single database cannot meet the needs of training data management. Summary of the invention

[0004] Based on this, it is necessary to provide a training data management system and method based on heterogeneous databases and large language models to address the above technical problems.

[0005] In a first aspect, an embodiment of the present invention provides a training data management system based on a heterogeneous database and a large language model, the system comprising: Heterogeneous database system module, including multiple databases supporting different data types; A data management registration center module, used to manage the multiple databases and node information of the registered data services; A front-end large language model module is used to parse the natural language instructions sent by the client and generate corresponding standard execution statements for describing the data service; The data service controller module is used to parse the standard execution statement and convert it into a target execution statement, and based on the target execution statement, call the corresponding data service from the corresponding node information in the data management registration center module to perform the corresponding operation on the corresponding database.

[0006] In some embodiments, the front-end large language model module includes a large language model, and the large language model is trained based on input and output sentence pairs of multiple rounds of dialogue; Among them, the input statement of the first round of dialogue is a preliminary description of the data service, and the output statement of the first round of dialogue is a guide for users to supplement parameter information; the input statement of the middle round of dialogue is the interaction of parameter information supplementation, and the output statement of the middle round of dialogue is the feedback confirmation of parameter information supplementation; the output statement of the last round of dialogue is a description of the standard execution statement of the data service.

[0007] In some embodiments, the data service controller module parses the standard execution statement, generates corresponding structured data, and generates the target execution statement based on the structured data.

[0008] In some embodiments, the heterogeneous database system module includes a relational database, a document database, and a file database, and the relational database stores a data entry record table; the data service includes a data entry service; The data service controller module calls the data warehousing service based on the target execution statement, stores the corresponding training data in the corresponding database, and then stores the associated information of the training data in the data warehousing record table.

[0009] In some embodiments, the associated information of the training data includes at least one of a data type, a data fingerprint, a data path, a data id, a data existence status, a storage timestamp, a data size, an uploading user, and a data subject.

[0010] In some embodiments, the relational database stores a data processing record table, and the data service includes a data processing service; The data service controller module calls the data processing service or the corresponding data processing program based on the target execution statement, and after processing the training data, stores the processing information of the training data in the data processing record table.

[0011] In some embodiments, the processing information of the training data includes at least one of a current timestamp, a source data fingerprint, a source data path, a processing service path, a processing program path, a processing configuration file path, and a processed data fingerprint.

[0012] In some embodiments, the data service controller module determines whether a data processing service needs to be called based on the target execution statement. If a call is needed, it detects whether a parameter configuration file corresponding to the data processing service exists in the file database. If not, it uploads the corresponding parameter configuration file to the file database. If so, it processes the training data using the data processing service and the parameter configuration file, and stores the processing information of the training data in the data processing record table. If no call is required, check whether the corresponding data processing program exists in the file database. If not, upload the corresponding data processing program to the file database. If so, check whether the parameter configuration file corresponding to the data processing service exists in the file database. If not, upload the corresponding parameter configuration file to the file database. If so, use the data processing program and the parameter configuration file to process the training data, and store the processing information of the training data in the data processing record table.

[0013] In a second aspect, an embodiment of the present invention provides a training data management method based on a heterogeneous database and a large language model, the method comprising: Build multiple databases that support different data types; Register multiple data services and generate node information of each of the data services; Utilize the large language model to parse the natural language instructions sent by the client, and generate corresponding standard execution statements for describing the data service; The standard execution statement is parsed and converted into a target execution statement, and corresponding node information is acquired based on the target execution statement to call a corresponding data service, so as to perform a corresponding operation on a corresponding database.

[0014] In a third aspect, an embodiment of the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the steps described in the second aspect.

[0015] Compared with the prior art, the above method and system utilize a front-end large language model module to parse the natural language instructions sent by the client, and generate corresponding standard execution statements for describing the data service; utilize a data service controller module to parse the standard execution statements and convert them into target execution statements, and based on the target execution statements, call the corresponding data service from the corresponding node information in the data management registration center module to perform corresponding operations on the corresponding database, which can improve the efficiency of querying and collecting data services such as specific training data, and greatly reduce the complexity of data management. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a structural diagram of a training data management system based on a heterogeneous database and a large language model in one embodiment of the present application; Figure 2 This is a flow chart of calling the data storage service in one embodiment of the present application; Figure 3 This is a flow chart of calling a data processing service in an embodiment of the present application; Figure 4 A schematic diagram of a process of calling a data query service in an embodiment of the present application; Figure 5 A flow chart of calling a data tracing service in an embodiment of the present application; Figure 6 This is a flow chart of a training data management method based on a heterogeneous database and a large language model in one embodiment of the present application; Figure 7 It is a schematic diagram of the structure of a computer device in one embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of the present invention. For ordinary technicians in this field, the present invention can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0018] As shown in the present invention and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular, but also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list, and the method or device may also include other steps or elements.

[0019] Although the present invention makes various references to certain modules in the system according to an embodiment of the present invention, any number of different modules can be used and run on a computing device and / or processor. The modules are only illustrative, and different aspects of the system and method can use different modules.

[0020] It should be understood that when a unit or module is described as being "connected," "coupled" to other units, modules, or blocks, it may refer to being directly connected or coupled, or communicating with other units, modules, or blocks, or there may be intermediate units, modules, or blocks, unless the context clearly indicates otherwise. The term "and / or" as used herein may include any and all combinations of one or more of the associated listed items.

[0021] like Figure 1 As shown, an embodiment of the present invention provides a training data management system based on heterogeneous databases and large language models, and the system includes: a heterogeneous database system module 102, including multiple databases supporting different data types; a data management registration center module 104, used to manage the multiple databases and the node information of the registered data services; a front-end large language model module 106, used to parse the natural language instructions sent by the client, and generate corresponding standard execution statements for describing the data services; a data service controller module 108, used to parse the standard execution statements and convert them into target execution statements, and call the corresponding data service from the corresponding node information in the data management registration center module based on the target execution statement to perform corresponding operations on the corresponding database.

[0022] The heterogeneous database system module 102 includes a relational database, a document database, and a file database. The relational database stores a data entry record table and a data processing record table.

[0023] The data management registration center module 104 is the basis for the system to call databases and data services, and can be constructed using database software. After each database in the system goes online, it is necessary to upload the IP address, database structure information, data access path, and access account information to the data management registration center module. The data management registration center module needs to send heartbeat messages regularly to monitor the availability and health status of all registered databases and data services.

[0024] The data entry record table stores associated information of the training data, including at least one of a data type, a data fingerprint, a data path, a data id, a data existence status, an entry timestamp, a data size, an uploading user, and a data subject.

[0025] The data type refers to the type of data currently stored. If it is a piece of data in a relational database, the type is Record; if it is a document in a document database, the type is Document; if it is a file in a file database, the type is File.

[0026] Data fingerprint is a unique identifier of training data generated using algorithms such as SHA-256. For a record in a relational database, all fields can be concatenated into a string in the order of the table to generate the corresponding data fingerprint. For documents or binary files, algorithms such as SHA-256 can be used directly to generate the corresponding data fingerprint.

[0027] The data path indicates the storage location of the training data. For records in a relational database and documents in a document database, the path includes the corresponding database node URL, the database to which it belongs, and the data table to which it belongs. For object storage service files in a file database, the data path includes the service node URL where the object storage service file is located and the unique URL of the object storage service file.

[0028] The data id is the unique identifier of the training data in the data entry record table. The training data stored in the relational database and the training data in the document database have unique ids. The only piece of training data can be located through the data path + data id. In the file database, the file can be directly located through the data path, so the data id is not required.

[0029] The data existence status indicates whether this piece of training data exists, that is, it has not been deleted. If it has been deleted, this field is No, and if it still exists, this field is Yes. After calling the data service to delete the training data, the training data existence status will be automatically set to No, and the existence time limit of the deleted data storage record can be set. If the time limit is exceeded, the training data in the data storage record table will be deleted. This mechanism is for querying all data storage records within a period of time after the training data is deleted.

[0030] The entry timestamp is the time when the training data is stored in the database.

[0031] The data size refers to the size of the training data stored. The data size can be the sum of the number of bytes of the value of each field.

[0032] The uploading user is the user who uploads the training data in the heterogeneous database.

[0033] The data subject is an abstract summary of the training data content, or the field to which the training data belongs. The data subject is an optional field in the data entry record table, which can be filled in manually when uploading training data. For documents, the document content can also be summarized by the subject through a large language model, and the data subject can be automatically filled in.

[0034] The data processing record table stores processing information of the training data, including at least one of a current timestamp, a source data fingerprint, a source data path, a processing service path, a processing program path, a processing configuration file path, and a processed data fingerprint.

[0035] Among them, the source data fingerprint and the processed data fingerprint are the key to tracing data lineage relationships.

[0036] The data processing service path, data processing program path, and parameter configuration file path are optional values ​​and are set based on whether they actually exist.

[0037] If you are calling the data processing service registered by the data management registration center module, the data processing service path field is required. If it is a data processing program written by the user, you can also call the API interface service provided by this system to call the data processing program.

[0038] The front-end large language model module 106 includes a large language model, which can understand natural language instructions input by humans in natural language, and feed back the interpretation of semantics to the user in natural language. If some data service details are missing, such as the setting of specific parameters, the selection of data range, etc., they can be fed back to the user in natural language and the user can be guided to complete them. After obtaining clear and complete instructions, standard execution statements can be generated according to the instructions, and the standard execution statements are, for example, SQL statements.

[0039] When preparing training data, it is necessary to consider the problem of unclear data description, so it is necessary to prepare multi-round dialogue data, including input and output pairs of multi-round dialogues. The large language model is trained based on the input and output sentence pairs of multi-round dialogues. Among them, the input sentence of the first round of dialogue is the preliminary description of the data service, and the output sentence of the first round of dialogue is the guidance for the user to supplement the parameter information; the input sentence of the middle round of dialogue is the interaction of parameter information supplementation, and the output sentence of the middle round of dialogue is the feedback confirmation of parameter information supplementation; the output sentence of the last round of dialogue is the description of the standard execution sentence of the data service.

[0040] For example: -input: I need to collect some aviation-related data, which must be newly entered into the database.

[0041] -output: Hello, is it okay for the data to be entered into the database within one year?

[0042] -input: within two years.

[0043] -output: OK, how much data do you need?

[0044] -Input: 500MB or more.

[0045] -output: OK, do you need text data?

[0046] -input: yes.

[0047] -output: Should the downloaded data be stored in your user directory?

[0048] -input: Stored in the C:\Users\ABC\OUTPUT directory.

[0049] -output: { Service category: data query, Data subject: Aviation, Data type: text data, Data entry date: less than two years, Data size: 500MB, Download: Yes, Download path: C:\Users\ABC\OUTPUT } Please confirm whether the description of this data service is correct.

[0050] -input: Correct.

[0051]

[0052] Data services include data storage services, data query and download services, data processing services, data tracing services, data deletion services, etc. These data services can be implemented through the training data management system, which can improve the efficiency of data services such as querying and collecting specific training data, and greatly reduce the complexity of data management.

[0053] With the help of the large language model, instructions can be issued in natural language for various types of data services. The standard execution statements processed by the large language model are statements that comply with standard specifications. They can present the user's task requirements in the form of SQL statements. However, for heterogeneous databases, the SQL statements processed by the large language model cannot be directly run in the actual database nodes. Instead, the standard execution statements processed by the large language model are first parsed in a regular manner to generate SQL statements that can be actually executed on the data service node or DML statements and DQL statements that comply with other database specifications. The above process is the work that the data service controller module needs to complete.

[0054] For example, taking the SQL statement output by the above large language model as an example, after being parsed by the data service controller module, the generated structured data is as follows:

[0055] Among them, task_list is a description of the current data task. In this example, there is only one data query and download service. service_list is the data service actually executed, which is the specific service URL of the search task and the SQL statement executed. Here, the data path and data ID that meet the requirements are first retrieved from the data entry record table. After retrieving the corresponding data storage location, the actual data needs to be found from the document database. The data service controller module will retrieve the corresponding service URL on the node where the training data is located in the data management registration center module, and generate a DML statement suitable for the corresponding database query syntax, and add it to service_list.

[0056] like Figure 2 As shown, when it is necessary to call the data warehousing service, the front-end large language model module 106 parses the natural language instructions sent by the client, and generates a corresponding standard execution statement for describing the data service. The data service controller module 108 parses the standard execution statement and converts it into a target execution statement, calls the SHA-256 algorithm to generate a data fingerprint corresponding to the data, calls the data warehousing service based on the target execution statement, stores the corresponding training data in the corresponding database, and then stores the associated information of the training data in the data warehousing record table.

[0057] When it is necessary to call a data processing service, the front-end large language model module 106 parses the natural language instructions sent by the client and generates a corresponding standard execution statement for describing the data service. The data service controller module 108 parses the standard execution statement and converts it into a target execution statement, and calls the data processing program corresponding to the data processing service based on the target execution statement. After processing the training data, the processing information of the training data is stored in the data processing record table.

[0058] like Figure 3 As shown, the data service controller module 108 parses the standard execution statement and converts it into a target execution statement. The data service controller module determines whether it is necessary to call the data processing service based on the target execution statement. If it is necessary to call, it detects whether the parameter configuration file corresponding to the data processing service exists in the file database. If not, the corresponding parameter configuration file is uploaded to the file database. If so, the training data is processed using the data processing service and the parameter configuration file, and the processing information of the training data is stored in the data processing record table.

[0059] If no call is required, check whether the corresponding data processing program exists in the file database. If not, upload the corresponding data processing program to the file database. If so, check whether the parameter configuration file corresponding to the data processing service exists in the file database. If not, upload the corresponding parameter configuration file to the file database. If so, use the data processing program and the parameter configuration file to process the training data, and store the processing information of the training data in the data processing record table.

[0060] like Figure 4 As shown, when it is necessary to call the data query service, the data service controller module 108 parses the standard execution statement, generates structured data including the task description and the initial service call information, searches the data path of the matching training data in the data entry record table, searches the data service on the node where the data path is located, generates a DQL statement for querying the training data from the corresponding database, and executes the DQL statement on the database.

[0061] like Figure 5 As shown, when it is necessary to call the data traceability service, the data service controller module 108 parses the standard execution statement, generates structured data including the task description and the initial service call information, searches for relevant records of the traceability data in the data processing record table, searches for the processing information of the traceability data in sequence, converts the processing information into a structured form and returns it to the user.

[0062] Each module in the training data management system based on heterogeneous databases and large language models can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0063] Based on the above training data management system, this application also proposes a training data management method based on heterogeneous databases and large language models, such as Figure 6 As shown, the method includes: S102, constructing multiple databases supporting different data types; S104, registering multiple data services and generating node information of each of the data services; S106, using the large language model to parse the natural language instructions sent by the client, and correspondingly generate standard execution statements for describing the data service; S108, parsing the standard execution statement and converting it into a target execution statement, acquiring corresponding node information based on the target execution statement to call a corresponding data service, so as to execute a corresponding operation on a corresponding database.

[0064] For the specific limitations on the training data management method based on heterogeneous databases and large language models, please refer to the limitations on the training data management system above, which will not be repeated here.

[0065] In one embodiment, the present invention provides a computer device, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the steps in any of the above-mentioned training data management method embodiments based on heterogeneous databases and large language models are implemented.

[0066] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0067] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0068] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0069] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A training data management system based on heterogeneous databases and large language models, characterized in that: The system comprises: Heterogeneous database system module, including multiple databases supporting different data types; A data management registration center module, used to manage the multiple databases and node information of the registered data services; A front-end large language model module is used to parse the natural language instructions sent by the client and generate corresponding standard execution statements for describing the data service; The data service controller module is used to parse the standard execution statement and convert it into a target execution statement, and based on the target execution statement, call the corresponding data service from the corresponding node information in the data management registration center module to perform the corresponding operation on the corresponding database.

2. The system according to claim 1, characterized in that The front-end large language model module includes a large language model, and the large language model is trained based on input and output sentence pairs of multiple rounds of dialogue; Among them, the input statement of the first round of dialogue is a preliminary description of the data service, and the output statement of the first round of dialogue is a guide for users to supplement parameter information; the input statement of the middle round of dialogue is the interaction of parameter information supplementation, and the output statement of the middle round of dialogue is the feedback confirmation of parameter information supplementation; the output statement of the last round of dialogue is a description of the standard execution statement of the data service.

3. The system according to claim 1, characterized in that The data service controller module parses the standard execution statement, generates corresponding structured data, and generates the target execution statement based on the structured data.

4. The system according to claim 1, characterized in that The heterogeneous database system module includes a relational database, a document database, and a file database. The relational database stores a data entry record table. The data service includes a data entry service. The data service controller module calls the data warehousing service based on the target execution statement, stores the corresponding training data in the corresponding database, and then stores the associated information of the training data in the data warehousing record table.

5. The system according to claim 4, characterized in that The associated information of the training data includes at least one of a data type, a data fingerprint, a data path, a data ID, a data existence status, a storage timestamp, a data size, an uploading user, and a data subject.

6. The system according to claim 4, characterized in that The relational database stores a data processing record table, and the data service includes a data processing service; The data service controller module calls the data processing service or the corresponding data processing program based on the target execution statement, and after processing the training data, stores the processing information of the training data in the data processing record table.

7. The system according to claim 6, characterized in that The processing information of the training data includes at least one of a current timestamp, a source data fingerprint, a source data path, a processing service path, a processing program path, a processing configuration file path, and a processed data fingerprint.

8. The system according to claim 6, characterized in that The data service controller module determines whether a data processing service needs to be called based on the target execution statement. If so, it detects whether a parameter configuration file corresponding to the data processing service exists in the file database. If not, it uploads the corresponding parameter configuration file to the file database. If so, it processes the training data using the data processing service and the parameter configuration file, and stores the processing information of the training data in the data processing record table. If no call is required, check whether the corresponding data processing program exists in the file database. If not, upload the corresponding data processing program to the file database. If so, check whether the parameter configuration file corresponding to the data processing service exists in the file database. If not, upload the corresponding parameter configuration file to the file database. If so, use the data processing program and the parameter configuration file to process the training data, and store the processing information of the training data in the data processing record table.

9. A training data management method based on heterogeneous databases and large language models, characterized in that: The method comprises: Build multiple databases that support different data types; Register multiple data services and generate node information of each of the data services; Utilize the large language model to parse the natural language instructions sent by the client, and generate corresponding standard execution statements for describing the data service; The standard execution statement is parsed and converted into a target execution statement, and corresponding node information is acquired based on the target execution statement to call a corresponding data service, so as to perform a corresponding operation on a corresponding database.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to claim 9 are implemented.

Images