Method and device for matching database by using large model in data governance industry

Abstract

The invention provides a method and a device for matching a database by using a large model in a data governance industry. The method and the device can improve database query efficiency. The method for using the large model matching database in the data governance industry comprises the steps of extracting features of a target database and table information under the target database; performing vectorization processing on the features of the target database and table information in the target database; establishing a library vector library and a table vector library based on the vectorized features of the target database and table information in the target database; vectorizing the query information to obtain a vectorized query vector result; based on the similarity between the query vector result and the library vector library, determining a target database corresponding to the to-be-queried library vector library; calculating the similarity between the query vector result and a table vector library under a target database corresponding to the to-be-queried library vector library so as to determine the to-be-queried table vector library; and outputting a query result based on the table information corresponding to the to-be-queried table vector library.

Description

Technical Field

[0001] The present invention relates to the field of database query technology, and in particular to a method and device for matching databases using a large model in the data management industry. Background Art

[0002] With the rapid development of big data and artificial intelligence technologies, data has become a core resource for enterprise operations and decision-making. As a key link to ensure data quality, security and compliance, data governance is becoming increasingly important. As the foundation of the development of the big data industry, databases have evolved from simple data storage to complex data management systems. In a big data environment, a database may contain multiple databases and tables. The amount of data in the database is huge and complex. Traditional matching methods often take a lot of time to traverse and compare, and it is difficult to quickly and accurately match the correct database and table from multiple databases and tables. Summary of the invention

[0003] In order to solve the above technical problems, the present invention is proposed. The embodiments of the present invention provide a method and device for matching a database using a large model in the data governance industry, which can improve the efficiency of database query.

[0004] According to one aspect of the present invention, a method for matching databases using a large model in the data governance industry is provided, comprising: extracting features of a target database and table information under the target database; vectorizing the features of the target database and the table information under the target database; establishing a library vector library and a table vector library based on the features of the target database after vectorization and the table information under the target database; vectorizing query information to obtain a vectorized query vector result; determining a target database corresponding to the library vector library to be queried based on the similarity between the query vector result and the library vector library; calculating the similarity between the query vector result and the table vector library under the target database corresponding to the library vector library to be queried to determine the table vector library to be queried; and outputting the query result based on the table information corresponding to the table vector library to be queried.

[0005] In one embodiment, based on the similarity between the query vector result and the library vector library, determining the target database corresponding to the library vector library to be queried includes: for each target database, calculating the cosine similarity between the query vector result and the library vector library; wherein, the closer the cosine similarity value is to 1, the more similar the query vector result is to the library vector library, and the closer the cosine similarity value is to -1, the less similar the query vector result is to the library vector library; selecting the record with the smallest absolute value of the cosine distance to obtain a library selection result; and determining the corresponding target database based on the library selection result.

[0006] In one embodiment, after selecting the record with the smallest absolute value of the cosine distance and obtaining the library selection result, the method for matching databases using a large model in the data governance industry includes: based on the library selection result, obtaining a table vector library to be queried under the corresponding target database; wherein, calculating the similarity between the query vector result and the table vector library under the target database corresponding to the library vector library to be queried to determine the table vector library to be queried, including: retrieving the table vector library to be queried with the query vector result as a condition; based on the cosine similarity algorithm, obtaining a similarity list between the query vector result and the table vector library; wherein the elements in the similarity list include the similarity distances between the table information in the table vector library and the similarity list and the query vector result; sorting all the similarity distances in the similarity list; when there is a similarity distance greater than or equal to a preset threshold in the table vector library to be queried, and the number of similarity distances greater than or equal to the preset threshold is greater than a preset number, recording the table vector library to be queried in the first table selection result.

[0007] In one embodiment, the method of using a large model to match a database in the data governance industry also includes: establishing an association relationship between tables based on a foreign key relationship; and organizing the main table and the corresponding foreign key association table into key-value pairs for storage.

[0008] In one embodiment, a method for using a large model matching database in the data governance industry also includes: when the query information has a multi-table joint query, obtaining a foreign key associated table associated with the table vector library to be queried; recording the foreign key associated table in a first table selection result to form a second table selection result; wherein, based on the table information corresponding to the table vector library to be queried, outputting the query result includes: outputting the query result based on the first table selection result and the second table selection result.

[0009] In one embodiment, the characteristics of the target database include a name and a remark; the characteristics of the target database and the table information under the target database are vectorized, including: reading the name of the target database and its remark to form a first description text; reading the table information under the target database; extracting the content characteristics of the table information and forming a second description text; splicing the first description text and the second description text to form a spliced ​​text content, and vectorizing the spliced ​​text content; wherein, based on the characteristics of the target database after vectorization and the table information under the target database, a library vector library and a table vector library are established, including: using the spliced ​​text content after vectorization as an index to construct a library vector library.

[0010] In one embodiment, the features of the target database and the table information under the target database are vectorized, including: extracting detailed content of the table information under the target database; splicing and correcting the detailed content to obtain a corrected third description text; vectorizing the third description text to generate a vector result and establish an index; wherein, based on the features of the target database after vectorization and the table information under the target database, a library vector library and a table vector library are established, including: establishing a table vector library based on the vector result; wherein each feature of the target database corresponds to a table vector library, and each of the table vector libraries corresponds to at least one data table.

[0011] In one embodiment, the detailed content includes: table name, table remarks, column name and column remarks; wherein, the detailed content is spliced ​​and corrected to obtain a corrected third description text, including: splicing the table name, the table remarks, the column name and the column remarks into a preliminary text; cleaning the preliminary text and trimming the data format to obtain a corrected third description text.

[0012] In one embodiment, query information is vectorized to obtain a vectorized query vector result, including: extracting key elements in the natural language of the query information based on a large language model; wherein the key elements include time, place, and query object; concatenating the extracted key elements into a first query text; removing punctuation marks in the first query text and unifying the case to obtain a second query text; and vectorizing the second query text to obtain a vectorized query vector result.

[0013] According to another aspect of the present invention, there is provided an apparatus for matching databases using a large model in the data governance industry, comprising: an extraction module for extracting features of a target database and table information under the target database; a processing module for vectorizing the features of the target database and the table information under the target database; an establishment module for establishing a library vector library and a table vector library based on the features of the target database after vectorization and the table information under the target database; a vectorization module for vectorizing query information to obtain a vectorized query vector result; a determination module for determining a target database corresponding to a library vector library to be queried based on a similarity between the query vector result and the library vector library; a calculation module for calculating a similarity between the query vector result and the table vector library under the target database corresponding to the library vector library to be queried, so as to determine the table vector library to be queried; and an output module for outputting a query result based on the table information corresponding to the table vector library to be queried.

[0014] The method and device for using a large model to match a database in the data governance industry provided by the present invention can quickly identify the correlation and similarity between data by vectorizing the database and table information, thereby significantly shortening the matching time, improving the overall speed of data processing, and improving the accuracy and timeliness of database matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and other purposes, features and advantages of the present invention will become more apparent by describing the embodiments of the present invention in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0016] Figure 1 It is a flowchart of a method for matching databases using a large model in the data governance industry provided by an exemplary embodiment of the present invention.

[0017] Figure 2 It is a structural diagram of a data structure provided by an exemplary embodiment of the present invention.

[0018] Figure 3 It is a schematic diagram of the structure of an apparatus for matching a database using a large model in the data governance industry provided by an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0019] Below, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described here.

[0020] Figure 1 FIG. 1 is a flow chart of a method for matching a database using a large model in the data governance industry provided by an exemplary embodiment of the present invention. Figure 1 As shown in the figure, the methods of using large model matching database in the data governance industry include:

[0021] Step 100: Extract the characteristics of the target database and the table information under the target database.

[0022] Multiple databases in the environment (eg, databases to be queried or databases to be used for query) are set as target databases, and features of each target database are extracted for recalling the corresponding database in the context of multi-database retrieval.

[0023] In some embodiments, the characteristics of the target database may include a name and its remarks, and the table information under the target database may include a table name, a table remark, a column name and a column remark. By vectorizing the characteristics, the difficulty of subsequent calculation and retrieval can be reduced and the efficiency can be improved.

[0024] Step 200: vectorize the characteristics of the target database and the table information under the target database.

[0025] The features of the target database are vectorized respectively, and all table information under each database is vectorized, so that the database can be matched first, and then the corresponding data table can be matched, thus improving the efficiency and accuracy of matching databases.

[0026] Step 300: Based on the characteristics of the target database after vectorization processing and the table information under the target database, a library vector library and a table vector library are established.

[0027] For example, extract the features of each database, process them through a large language model to form a description text, then perform vectorization processing, build a library vector library for database recall, and then extract detailed information for each table in each database, perform preprocessing, vectorization processing, and create an index to build a table vector library.

[0028] Step 400: vectorize the query information to obtain a vectorized query vector result.

[0029] After obtaining the query information input by the user, a large language model can be used to extract key elements in the natural language of the user's query, such as time, entity, etc., to understand the user's query intent and reduce the impact of differences in expression or tone on the semantics of the query statement. The processed natural language is reassembled into a new query text.

[0030] Step 500: Based on the similarity between the query vector result and the library vector library, determine the target database corresponding to the library vector library to be queried.

[0031] Evaluate the similarity between the query vector result and the library vector library, select the library vector library with the highest similarity as the library vector library to be queried, and recall the corresponding target database using the library vector library as an index.

[0032] Step 600: Calculate the similarity between the query vector result and the table vector library under the target database corresponding to the library vector library to be queried, so as to determine the table vector library to be queried.

[0033] According to the selected library vector library, the table vector library with the highest similarity is retrieved therefrom to further determine the query table result.

[0034] Step 700: Output the query result based on the table information corresponding to the table vector library to be queried.

[0035] According to the table information corresponding to the table vector library, the relevant table or data is found and returned, which is the query result. By converting the database table information into vector form and storing it in the table vector library, the system can efficiently process complex query requests and quickly return results related to user needs.

[0036] In one embodiment, step 500 may include: for each target database, calculating the cosine similarity between the query vector result and the library vector library; wherein, the closer the cosine similarity value is to 1, the more similar the query vector result is to the library vector library, and the closer the cosine similarity value is to -1, the less similar the query vector result is to the library vector library; selecting the record with the smallest absolute value of the cosine distance to obtain a library selection result; and determining the corresponding target database based on the library selection result.

[0037] The query vector result obtained in step 400 is used to compare the similarity with the database vector library established in step 300. For each database vector, the cosine similarity with the query vector result is calculated. Cosine similarity is a commonly used measurement method that evaluates the similarity of two vectors by calculating the cosine value of the angle between them. The cosine value of a 0-degree angle is 1, while the cosine value of any other angle is not greater than 1; and its minimum value is -1. Thus, the cosine value of the angle between two vectors determines whether the two vectors point in roughly the same direction. When the two vectors have the same direction, the cosine similarity value is 1; when the angle between the two vectors is 90°, the cosine similarity value is 0; when the two vectors point in completely opposite directions, the cosine similarity value is -1. This result is independent of the length of the vector and is only related to the direction in which the vector points. Cosine similarity is usually used in positive space, so the value given is between -1 and 1. The closer the value is to 1, the more similar the two vectors are; the closer the value is to -1, the less similar the two vectors are. The record with the smallest absolute value of cosine distance is selected to obtain the database selection result. In other words, the most similar target database is selected by calculating the cosine similarity with the query vector result. After the target database is determined, the data table can be selected.

[0038] In one embodiment, after selecting the record with the smallest absolute value of the cosine distance and obtaining the library selection result, the method of using a large model to match the database in the data governance industry may include: based on the library selection result, obtaining the table vector library to be queried under the corresponding target database; wherein the above step 600 can be adjusted accordingly: using the query vector result as a condition to retrieve the table vector library to be queried; based on the cosine similarity algorithm, obtaining a similarity list between the query vector result and the table vector library; wherein the elements in the similarity list include the similarity distances between the table information in the table vector library and the similarity list and the query vector result; sorting all the similarity distances in the similarity list; when there is a similarity distance greater than or equal to a preset threshold in the table vector library to be queried, and the number of similarity distances greater than or equal to the preset threshold is greater than the preset number, the table vector library to be queried is recorded in the first table selection result.

[0039] For example, Figure 2 is a schematic diagram of a data structure provided by an exemplary embodiment of the present invention, with reference to Figure 2 , KEY is the database name, VAL is the data table vector library, the database names include DB1, DB2, DB3, DB4 and DB5, the data table vector library includes Vecdb1, Vecdb2, Vecdb3, Vecdb4 and Vecdb5, if the selected library in step 500 is DB2, then the table vector library obtained at this time is Vecdb2, and the query vector result obtained in step 400 is used as a condition to retrieve the table vector library Vecdb2. The same cosine algorithm as that for calculating the similarity of the library vector library can be used to calculate the similarity between the query vector result and the table vector library, and obtain the similarity list sim, which contains all the table information and the similarity distance between the table and ebd. Sort all the elements in the similarity list sim and set a reasonable threshold for screening. Only tables whose similarity exceeds the threshold will be recorded in the final table selection result TBs (first table selection result).

[0040] In one embodiment, the method of using a large model to match a database in the data governance industry may also include: establishing an association relationship between tables based on a foreign key relationship; and organizing the main table and the corresponding foreign key association table into key-value pairs for storage.

[0041] When performing database queries, especially when it involves association queries between multiple tables, in order to ensure the completeness and accuracy of the query results, you can use foreign key (FK) relationships to supplement the information of other tables associated with the main query table. Foreign key information can be obtained through metadata tables (such as INFORMATION_SCHEMA) or intercepted from Create table statements, and the recalled table TBs and its foreign key associated table are organized into key-value pairs. For example, if the foreign keys of table t1 are t2 and t3, the record can be as follows: t1: [t2, t3].

[0042] In one embodiment, the method for using a large model to match a database in the data governance industry may also include: when the query information has a multi-table joint query, obtaining a foreign key association table associated with the table vector library to be queried; recording the foreign key association table in the first table selection result to form a second table selection result; wherein the above step 700 can be adjusted accordingly: based on the first table selection result and the second table selection result, outputting the query result.

[0043] For example, by building a prompt question big model, if the query involves multi-table joint query, let the big model select the corresponding table FKs. If the table already exists in TBs, there is no need to add the table to FKs. In the table selection result TBs and table selection result FKs, the query results are finally output according to the actual situation or actual needs. In other words, in addition to single-table query, multi-table joint query can also be established to increase the scope of the query, thereby improving the query accuracy.

[0044] In some embodiments, the characteristics of the target database include a name and a remark; the above step 200 may include: reading the name of the target database and its remark to form a first description text; reading the table information under the target database; extracting the content characteristics of the table information and forming a second description text; concatenating the first description text and the second description text to form a concatenated text content, and vectorizing the concatenated text content; wherein the above step 300 may be correspondingly adjusted to: using the vectorized concatenated text content as an index to construct a library vector library.

[0045] For example, first connect to the target database through the DBMS (database management system) component, read the name of the target database and its notes, form a preliminary description text d1 (first description text), then read the table information under the database, let the big model summarize all the table information through the prompt technology, obtain the content involved in the database in the table information, and form text d2 (second description text). Splice the contents of d1 and d2 to form text d, and vectorize the content of text d. Use the vectorized vector as an index to build a vector database vec_db.

[0046] As an example, embedding technology can be used to convert key data into vector form and build a vector database. The vector database can store and index a large amount of vectorized data for fast retrieval. The management and operation of the vector library can utilize a variety of mature solutions, such as vector database systems such as Faiss and Chromadb, which are specially designed to handle the storage, indexing and similarity search of large-scale vectors, and can significantly improve the processing speed and accuracy, thereby meeting the complex similarity retrieval requirements of modern applications. Embedding technology can use large models for vectorization processing, such as common embedding models such as text2vec, bge, bert, etc. The vector database should at least include: the stored vector results as indexes, the actual stored data (table structure) and the traceability information of the table structure.

[0047] In one embodiment, the above step 200 may include: extracting detailed contents of the table information under the target database; splicing and correcting the detailed contents to obtain a corrected third description text; vectorizing the third description text to generate a vector result and establish an index; wherein the above step 300 may be correspondingly adjusted to: establishing a table vector library based on the vector result; wherein each feature of the target database corresponds to a table vector library, and each table vector library corresponds to at least one data table.

[0048] Extract detailed information of each table in each database, including table name, table comment, column name and column comment, and concatenate this information into text t. Perform preprocessing and correction on the concatenated text t, then perform vectorization on the corrected text, generate vector results and create indexes. The saved content is all the information of the data table, such as the create table statement.

[0049] After each database is established and saved separately, a vector database is finally constructed. Figure 2 The hashtable data structure shown is used to store the created vector library.

[0050] In one embodiment, the detailed content includes: table name, table remarks, column name and column remarks; wherein, the detailed content is spliced ​​and corrected to obtain a corrected third description text, including: splicing the table name, table remarks, column name and column remarks into a preliminary text; cleaning the preliminary text and trimming the data format to obtain a corrected third description text.

[0051] The detailed content is spliced ​​and corrected. The correction includes cleaning irrelevant content, punctuation and other information in the remarks and trimming the data format, removing all punctuation and unifying the upper and lower case, such as trimming varchar(255) to varchar, etc. After the correction is completed, the results are re-spliced ​​into a new query text, that is, the third description text.

[0052] In one embodiment, the above step 400 may include: extracting key elements in the natural language of the query information based on a large language model; wherein the key elements include time, place and query object; concatenating the extracted key elements into a first query text; removing punctuation marks in the first query text and unifying the case to obtain a second query text; vectorizing the second query text to obtain a vectorized query vector result.

[0053] Large language models (LLMs) are used to extract key elements in the natural language of user queries, such as time and entities, to understand the user's query intent and reduce the impact of differences in expression or tone on the semantics of query statements. Large language models are complex neural networks trained based on a large amount of data, which can capture and simulate the complexity and diversity of language. These models usually use the Transformer architecture, which is a model based on the self-attention mechanism that can process sequence data such as text. The self-attention mechanism allows the model to consider other words in the entire text sequence when processing a word, so as to better understand the context. After the extracted key elements are spliced ​​into the first query text, all punctuation is removed and the capitalization is unified. Then, a large model (such as Bert, GPT, etc.) is used to extract and identify key information and elements, and the large model is used to identify entities such as time, place, and query object, and then the results are reassembled into the second query text. The second query text is vectorized using the same embedding method / model as before to obtain the query vector result.

[0054] Figure 3 is a schematic diagram of a structure of an apparatus for matching a database using a large model in a data management industry provided by an exemplary embodiment of the present invention, such as Figure 3 As shown, the device 3 for using a large model to match a database in the data governance industry includes: an extraction module 31, which is used to extract the characteristics of the target database and the table information under the target database; a processing module 32, which is used to vectorize the characteristics of the target database and the table information under the target database; an establishment module 33, which is used to establish a library vector library and a table vector library based on the characteristics of the target database after vectorization and the table information under the target database; a vectorization module 34, which is used to vectorize the query information to obtain the vectorized query vector result; a determination module 35, which is used to determine the target database corresponding to the library vector library to be queried based on the similarity between the query vector result and the library vector library; a calculation module 36, which is used to calculate the similarity between the query vector result and the table vector library under the target database corresponding to the library vector library to be queried, so as to determine the table vector library to be queried; an output module 37, which is used to output the query result based on the table information corresponding to the table vector library to be queried.

[0055] The device provided by the present invention uses a large model to match the database in the data governance industry. By vectorizing the database and table information, it can quickly identify the correlation and similarity between data, thereby significantly shortening the matching time, improving the overall speed of data processing, and improving the accuracy and timeliness of database matching.

[0056] In one embodiment, the determination module 35 can be configured as follows: for each target database, calculate the cosine similarity between the query vector result and the library vector library; wherein, the closer the cosine similarity value is to 1, the more similar the query vector result is to the library vector library, and the closer the cosine similarity value is to -1, the less similar the query vector result is to the library vector library; select the record with the smallest absolute value of the cosine distance to obtain the library selection result; determine the corresponding target database based on the library selection result.

[0057] In one embodiment, the device 3 for using a large model matching database in the data governance industry can be configured as follows: based on the library selection result, obtain the table vector library to be queried under the corresponding target database; wherein the above-mentioned calculation module 36 can be configured accordingly as follows: retrieve the table vector library to be queried with the query vector result as a condition; based on the cosine similarity algorithm, obtain a similarity list between the query vector result and the table vector library; wherein the elements in the similarity list include the similarity distances between the table information in the table vector library and the similarity list and the query vector result; sort all the similarity distances in the similarity list; when there is a similarity distance greater than or equal to a preset threshold in the table vector library to be queried, and the number of similarity distances greater than or equal to the preset threshold is greater than a preset number, record the table vector library to be queried in the first table selection result.

[0058] In one embodiment, the device 3 for using a large model matching database in the data governance industry can be configured as follows: based on the foreign key relationship, an association relationship between tables is established; and the main table and the corresponding foreign key association table are organized into key-value pairs for storage.

[0059] In one embodiment, the device 3 for using a large model matching database in the data governance industry can be configured as follows: when the query information exists in a multi-table joint query, obtain the foreign key association table associated with the table vector library to be queried; record the foreign key association table in the first table selection result to form a second table selection result; wherein the above-mentioned output module 37 can be correspondingly configured as follows: based on the first table selection result and the second table selection result, output the query result.

[0060] In one embodiment, the characteristics of the target database include a name and a remark; the processing module 32 can be configured to: read the name of the target database and its remark to form a first description text; read the table information under the target database; extract the content characteristics of the table information and form a second description text; concatenate the first description text and the second description text to form a concatenated text content, and vectorize the concatenated text content; wherein the establishment module 33 can be configured to: use the concatenated text content after vectorization as an index to construct a library vector library.

[0061] In one embodiment, the processing module 32 can be configured to: extract detailed contents of the table information under the target database; splice and correct the detailed contents to obtain a corrected third description text; vectorize the third description text to generate a vector result and establish an index; wherein the establishment module 33 can be configured to: establish a table vector library based on the vector result; wherein each feature of the target database corresponds to a table vector library, and each table vector library corresponds to at least one data table.

[0062] In one embodiment, the detailed content includes: table name, table remarks, column name and column remarks; the above-mentioned processing module 32 can also be configured to: splice the table name, table remarks, column name and column remarks into preliminary text; clean the preliminary text and trim the data format to obtain a revised third description text.

[0063] In one embodiment, the above-mentioned vectorization module 34 can be configured as follows: based on a large language model, extract key elements in the natural language of the query information; wherein the key elements include time, place and query object; concatenate the extracted key elements into a first query text; remove punctuation marks in the first query text and unify the case to obtain a second query text; vectorize the second query text to obtain a vectorized query vector result.

[0064] An embodiment of the present invention provides a device for using a large model matching database in the data governance industry. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. From a hardware perspective, in addition to the CPU, memory, network interface, and non-volatile memory, the device in the embodiment where the device is located can generally include other hardware, such as a forwarding chip responsible for processing messages, etc. Taking software implementation as an example, as a device in a logical sense, it is formed by the CPU of the device where it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution.

[0065] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores a computer program for executing the method of using a large model to match a database in the data governance industry according to any of the above embodiments.

[0066] In addition to the above-mentioned methods and devices, an embodiment of the present invention may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method of using a large model matching database in the data governance industry according to various embodiments of the present invention described in the above "Exemplary Method" section of this specification.

[0067] According to another aspect of the present invention, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; and a processor for executing the method of using a large model matching database in the data governance industry according to any of the above embodiments.

[0068] In addition, an embodiment of the present invention may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, causes the processor to execute the steps of the method of using a large model matching database in the data governance industry according to various embodiments of the present invention described in the above “Exemplary Method” section of this specification.

[0069] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for using a large model matching database in the data governance industry, characterized in that: include: Extracting characteristics of a target database and table information under the target database; Performing vectorization processing on the characteristics of the target database and table information under the target database; Based on the characteristics of the target database after vectorization processing and the table information under the target database, a library vector library and a table vector library are established; Perform vectorization processing on the query information to obtain a vectorized query vector result; Determining a target database corresponding to the library vector library to be queried based on the similarity between the query vector result and the library vector library; Calculating the similarity between the query vector result and the table vector library under the target database corresponding to the library vector library to be queried, so as to determine the table vector library to be queried; Based on the table information corresponding to the table vector library to be queried, the query result is output.

2. The method for using a large model matching database in the data governance industry according to claim 1, characterized in that: Determining a target database corresponding to the library vector library to be queried based on the similarity between the query vector result and the library vector library includes: For each target database, the cosine similarity between the query vector result and the library vector library is calculated; wherein, the closer the cosine similarity value is to 1, the more similar the query vector result is to the library vector library, and the closer the cosine similarity value is to -1, the less similar the query vector result is to the library vector library; Select the record with the smallest absolute value of cosine distance to obtain the library selection result; A corresponding target database is determined based on the database selection result.

3. The method for using a large model matching database in the data governance industry according to claim 2, characterized in that: After selecting the record with the smallest absolute value of the cosine distance and obtaining the database selection result, the methods of using large models to match databases in the data governance industry include: Based on the library selection result, obtaining a table vector library to be queried under the corresponding target database; The step of calculating the similarity between the query vector result and the table vector library under the target database corresponding to the library vector library to be queried, so as to determine the table vector library to be queried, includes: Using the query vector result as a condition to search the table vector library to be queried; Based on the cosine similarity algorithm, a similarity list between the query vector result and the table vector library is obtained; wherein the elements in the similarity list include similarity distances between the table information in the table vector library and the similarity list and the query vector result; Sorting all similarity distances in the similarity list; When there is a similarity distance greater than or equal to a preset threshold in the table vector library to be queried, and the number of similarity distances greater than or equal to the preset threshold is greater than a preset number, the table vector library to be queried is recorded in the first table selection result.

4. The method for using a large model matching database in the data governance industry according to claim 3, characterized in that: Methods for using large models to match databases in the data governance industry also include: Establish associations between tables based on foreign key relationships; The main table and the corresponding foreign key association table are organized into key-value pairs for storage.

5. The method for using a large model matching database in the data governance industry according to claim 4, characterized in that: Methods for using large models to match databases in the data governance industry also include: When the query information contains multiple table joint queries, obtain the foreign key association table associated with the table vector library to be queried; Recording the foreign key association table into the first table selection result to form a second table selection result; Among them, based on the table information corresponding to the table vector library to be queried, the query results are output, including: Based on the first table selection result and the second table selection result, a query result is output.

6. The method for using a large model matching database in the data governance industry according to claim 1, characterized in that: The characteristics of the target database include a name and a remark; the characteristics of the target database and the table information under the target database are vectorized, including: Read the name of the target database and its remarks to form a first description text; Read the table information under the target database; Extracting content features of the table information and forming a second description text; splicing the first description text and the second description text to form spliced ​​text content, and performing vectorization processing on the spliced ​​text content; Wherein, based on the characteristics of the target database after vectorization processing and the table information under the target database, a library vector library and a table vector library are established, including: The vectorized concatenated text content is used as an index to construct a library vector library.

7. The method for using a large model matching database in the data governance industry according to claim 1, characterized in that: Vectorizing the characteristics of the target database and table information under the target database includes: Extract the detailed contents of the table information under the target database; splicing and correcting the detailed contents to obtain a corrected third description text; Performing vectorization processing on the third description text, generating a vector result and establishing an index; Wherein, based on the characteristics of the target database after vectorization processing and the table information under the target database, a library vector library and a table vector library are established, including: Based on the vector results, a table vector library is established; wherein each feature of the target database corresponds to a table vector library, and each table vector library corresponds to at least one data table.

8. The method for using a large model matching database in the data governance industry according to claim 7, characterized in that: The detailed content includes: table name, table remarks, column name and column remarks; wherein the detailed content is spliced ​​and corrected to obtain a corrected third description text, including: splicing the table name, the table remark, the column name and the column remark into a preliminary text; The preliminary text is cleaned and the data format is cut to obtain a revised third description text.

9. The method for using a large model matching database in the data governance industry according to claim 1, characterized in that: Perform vectorization on the query information to obtain vectorized query vector results, including: Extract key elements in the natural language of the query information based on a large language model; wherein the key elements include time, place and query object; splicing the extracted key elements into a first query text; Remove punctuation marks from the first query text and unify the capitalization to obtain a second query text; The second query text is vectorized to obtain a vectorized query vector result.

10. A device for using a large model matching database in the data management industry, characterized in that: include: An extraction module, used to extract the characteristics of a target database and table information under the target database; A processing module, used for performing vectorization processing on the characteristics of the target database and the table information under the target database; An establishment module, used to establish a library vector library and a table vector library based on the characteristics of the target database after vectorization processing and the table information under the target database; A vectorization module is used to perform vectorization processing on the query information to obtain a vectorized query vector result; A determination module, configured to determine a target database corresponding to the library vector library to be queried based on the similarity between the query vector result and the library vector library; A calculation module, used for calculating the similarity between the query vector result and the table vector library under the target database corresponding to the library vector library to be queried, so as to determine the table vector library to be queried; The output module is used to output the query result based on the table information corresponding to the table vector library to be queried.

Images