A data map construction method and device, electronic equipment and storage medium
By automatically identifying the relationships in the database table structure through large language models and community discovery algorithms, the problem of low efficiency, error-proneness, and high cost in data map construction in existing technologies is solved, and efficient and accurate data map construction and dynamic updates are achieved.
Patent Information
- Application Number
- CN202610939129.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-26
- Publication Date
- 2026-08-25
AI Technical Summary
Existing technologies for constructing data maps are inefficient, error-prone, costly, and difficult to update dynamically. In particular, manual analysis of large-scale databases is time-consuming, labor-intensive, and prone to errors.
By leveraging large language models to automatically identify the relationships between database table structures, and by initializing the data map and filling in the related table structures, combined with community discovery algorithms and SQL log data, automated and intelligent data map construction is achieved.
It significantly improves the efficiency and accuracy of data map construction, reduces labor costs, supports dynamic updates, and reduces the time and resource consumption of repetitive work.
Smart Images
Figure CN122633657A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data management, and more particularly to a data map construction method and apparatus, electronic device and computer-readable storage medium. Background Technology
[0002] In the wave of digital transformation, enterprises and organizations are experiencing explosive growth in their data assets, with the number and complexity of table structures in databases continuously increasing. Data maps, as a tool that intuitively presents the distribution, relationships, and metadata of data assets, have become a core support for data governance, data development, and business decision-making. Through data maps, users can quickly locate target data and understand the logical relationships between data, thereby improving data utilization efficiency and lowering the barrier to data use. Therefore, the quality of data map construction directly affects the release of the value of data assets.
[0003] In existing technologies, the construction of data maps often relies on manual intervention or semi-automatic methods based on fixed rules. After obtaining the table structure data of the target database, technicians usually need to manually analyze the meaning of the fields and business logic of each table structure to determine the relationships between tables, such as through primary key-foreign key correspondences and field naming similarity.
[0004] However, as databases grow in size, the number of table structures often reaches hundreds or even thousands, and field types and business semantics become increasingly complex. Manual analysis is not only time-consuming and labor-intensive, but also prone to inaccurate judgments of relationships due to differences in individual experience, overlooking potential semantic relationships or misjudging relationships between unrelated table structures. Therefore, existing methods of manually constructing data maps suffer from inefficiency, error-proneness, high costs, and difficulty in dynamic updates. Summary of the Invention
[0005] In view of this, it is necessary to provide a data map construction method and apparatus, electronic device and computer-readable storage device to solve the technical problems of low efficiency, error susceptibility, high cost and difficulty in dynamic updating of existing data map construction methods.
[0006] To address the aforementioned technical problems, firstly, this application provides a data map construction method, comprising: Retrieve table structure data for multiple table structures in the target database; Based on the large language model, the first related table structure with an association relationship is selected from the multiple table structures according to the table structure data; An initial data map is provided, and the structure of the first association table is filled into the initial data map based on the association relationship to obtain a target data map corresponding to the target database.
[0007] In one possible embodiment, it further includes: Obtain SQL log data related to the multiple table structures in the target database, wherein the SQL log data includes at least Join operation data; Based on the Join operation data, select a second related table structure from the plurality of table structures that has a relationship; Based on the aforementioned relationship, the structure of the second association table is filled into the initial data map.
[0008] In one possible embodiment, it further includes: Based on the community detection algorithm, the table structure in the target data map is divided into communities, resulting in multiple table structure communities. For any of the table structure communities, community business description information for each table structure community is generated based on the table structure data in the table structure community using a large language model. Based on a large language model, a global data domain description of the target data map is generated according to all the community business description information.
[0009] In one possible embodiment, it further includes: For any of the table structure communities, the community business description information is semantically vectorized to obtain a community business description vector, and the community business description vector is stored in a vector database corresponding to the table structure community. For any of the target data maps, the global data domain description is semantically vectorized to obtain a global description vector, and the global description vector is stored in the vector database in correspondence with the target data map.
[0010] In one possible embodiment, before selecting the first related table structure with an association relationship from the plurality of table structures based on the table structure data according to the large language model, the method further includes: For any of the table structure data, complete the table structure data based on the large language model.
[0011] In one possible embodiment, before filling the first association table structure into the initial data map based on the association relationship, the method further includes: For any two first association table structures that have the aforementioned relationship, check whether the overlap rate of table data in the two first association table structures is greater than or equal to a preset threshold. If the overlap rate of the table data is greater than or equal to the preset threshold, it is determined that the association relationship exists; If the overlap rate of the table data is less than the preset threshold, it is determined that the association relationship does not exist.
[0012] In one possible embodiment, before filling the first association table structure into the initial data map based on the association relationship, the method further includes: For any two first association table structures that have the aforementioned relationship, check whether there is an explicit foreign key constraint between the two first association table structures; If an explicit foreign key constraint exists, the association relationship is confirmed to exist. If no explicit foreign key constraint exists, the association relationship is determined to be non-existent.
[0013] Secondly, this application provides a data map construction apparatus, comprising: The data acquisition module is used to acquire table structure data of multiple table structures in the target database; The association relationship determination module is used to select a first associated table structure with an association relationship from the multiple table structures based on the table structure data according to the large language model; A data map construction module is used to provide an initial data map, which fills the first association table structure into the initial data map based on the association relationship, so as to obtain a target data map corresponding to the target database.
[0014] Thirdly, this application also provides an electronic device, including a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, it implements a data map construction method as described in any of the above technical solutions.
[0015] This application also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements a data map construction method as described in any of the above technical solutions.
[0016] The technical effects of this application are as follows: Compared with existing technologies, the data map construction method, apparatus, electronic device, and computer-readable storage medium provided in this application, when constructing a data map, select a first related table structure with a relationship from multiple table structures of the target database based on a large language model according to the table structure data. Then, the first related table structure is filled into a blank initial data map to obtain a target data map corresponding to the target database. The automatic selection of the first related table structure based on the large language model replaces traditional manual analysis or fixed rule matching, eliminating the need for technicians to analyze the semantics and business logic of fields table by table, significantly shortening the relationship identification cycle. Simultaneously, the automatic filling of the first related table structure into the initial data map based on the relationship eliminates the manual input step, thereby significantly improving the efficiency of data map construction. Because the large language model has powerful semantic understanding capabilities, it can… This approach breaks free from the reliance on fixed field naming rules, accurately identifies relationships between table structures, and reduces omissions and misjudgments caused by differences in human experience or rule limitations. It also automatically fills the initial data map with the structure of the first related table, further reducing formatting errors and misalignments during manual data entry, thus improving the accuracy of the target data map. Furthermore, this application reduces reliance on highly experienced technical personnel through intelligent and automated processes, lowering labor costs. The improved efficiency and accuracy of data map construction also reduce time and resource consumption from rework, significantly lowering the cost of data map construction. Additionally, when the table structure in the target database changes, the large language model can quickly re-identify relationships and update the target data map without manually re-organizing the entire table structure, enabling dynamic and efficient updates. This solves the technical problems of inefficiency, error-proneness, high cost, and difficulty in dynamic updates inherent in existing data map construction methods. Attached Figure Description
[0017] Figure 1 A schematic flowchart illustrating a data map construction method provided in an embodiment of this application; Figure 2 A flowchart illustrating a data map construction method provided in another embodiment of this application; Figure 3 A flowchart illustrating a data map construction method provided in yet another embodiment of this application; Figure 4 This is a schematic diagram of the structure of a data map building apparatus provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0018] The preferred embodiments of this application are described in detail below with reference to the accompanying drawings, which constitute a part of this application and are used together with the embodiments of this application to illustrate the principles of this application, but are not intended to limit the scope of this application.
[0019] This application provides a data map construction method and a data map construction device, which are described in detail below.
[0020] This application provides a data map construction method, the flowchart of which is shown below. Figure 1 As shown, it includes: Step S101: Obtain table structure data for multiple table structures in the target database.
[0021] In this step, the target database is the database where the metadata for building the data map is stored. Specifically, it could be a company's business database, a big data platform's user database, etc. Within the target database, data is stored in a table structure; that is, the data is categorized and stored in corresponding table structures. The data stored in each table structure, as well as the structural parameters of the table structures, are the same as the table structure data described in this application.
[0022] Furthermore, in this application, the table structure data specifically includes three parts: table name, field name, and comments. The table name is the name used to uniquely identify each table structure in the database. Its core function is to identify and distinguish each table structure; the table name is unique within the same target database. The field name is the unique identifier of the smallest data storage unit in the table structure, used to define the specific business dimension of the data stored in that unit. It is usually directly related to the business meaning of the corresponding data (such as describing user attributes, order status, product attributes, etc.), and the field name is unique within the same table structure. Comments are supplementary explanatory text for the table name or field name, and their core function is to explain and supplement the table structure or fields.
[0023] Step S102: Based on the large language model, select the first related table structure from multiple table structures according to the table structure data.
[0024] In this step, the Large Language Model (LLM) is an artificial intelligence model capable of understanding and generating human language. Its core relies on architectures such as Transformer, trained on massive amounts of text data (such as books, web pages, and documents) to master the semantic, logical, and contextual rules of language. Large Language Models can deeply understand context, such as coherent dialogue and parsing complex questions, and can generate text that conforms to human expression habits, encompassing capabilities such as question answering, translation, creation, and code assistance. Common examples include GPT and Deepseek, which fall under the category of Large Language Models.
[0025] Specifically, in this step, a query input statement for table structure relationships can be constructed in the large language model, along with all the table structure data obtained in step S101. The large language model then outputs the first related table structure selected from multiple table structures. For example, a Prompt can be constructed: "You are a database relationship expert. Here is a summary of all tables in the database: [List all table structure data obtained in step S101]. Please analyze and recommend possible inter-table relationships (e.g., field X of table A may be related to field Y of table B), and briefly explain the reasons (e.g., similar business logic or similar names). Please output in the format 'Table A - Table B (related field AX - related field BY): reason'. Table A and Table B are the first related table structures."
[0026] Furthermore, in this embodiment, before selecting the first related table structure from multiple table structures based on the table structure data using the large language model, the table structure data can be completed using the large language model. Specifically, for incomplete table structure data, the large language model generates the missing table structure data to complete the original table structure data. For example, for a table structure with empty comments, a prompt can be constructed in the large language model: "You are a database expert. Known table name: [Table_i.Name], field names: [Field1, Field2,..., FieldN], existing comments: empty. Please generate: a detailed Chinese description of the business entity or process represented by this table." Completing partially missing table structure data using the large language model can improve the accuracy and reliability of the data map constructed based on the table structure data in subsequent steps.
[0027] Furthermore, in this step, after selecting the first associated table structure with a relationship from multiple table structures based on the large language model and the table structure data, the relationship between the first associated table structures can be verified. In this embodiment, the relationship is verified based on the overlap rate of table data in the first associated table structures, specifically including: for any two first associated table structures with a relationship, checking whether the overlap rate of table data in the two first associated table structures is greater than or equal to a preset threshold; if the overlap rate is greater than or equal to the preset threshold, the relationship is determined to exist; if the overlap rate is less than the preset threshold, the relationship is determined not to exist. The table data overlap rate is the ratio of duplicate data to all data in the two first associated table structures with a relationship. For example, for two first-related table structures (Table_A, Field_A) -> (Table_B, Field_B), the amount of duplicate data existing in both Table_A.Field_A and Table_B.Field_B is counted, and the total amount of data in both Table_A.Field_A and Table_B.Field_B is also counted. The ratio of the amount of duplicate data to the total amount of data is calculated as the table data overlap rate. This overlap rate is compared with a preset threshold. If the table data overlap rate is greater than or equal to the preset threshold, the association is determined to exist, meaning Table_A and Table_B are two first-related table structures. Conversely, if the table data overlap rate is less than the preset threshold, the association is determined to not exist, meaning Table_A and Table_B are not two first-related table structures.
[0028] It is understood that the aforementioned verification of the association relationship through the overlap rate of table data is merely an example of a specific method for verifying the association relationship in this embodiment. In some other embodiments of this application, it can also be to check whether there are explicit foreign key constraints on the two first association table structures; if explicit foreign key constraints exist, the association relationship is determined to exist; if no explicit foreign key constraints exist, the association relationship is determined to not exist. Specifically, explicit foreign key constraints require that the foreign key column of the child table must be related to the primary key or unique key of the parent table, and the foreign key value of the child table must exist in the corresponding column of the parent table. In this application, the two first association table structures can be defined as a child table and a parent table respectively, and the association relationship can be verified based on the foreign key column of the child table and the primary key or unique key of the parent table.
[0029] Step S103: Provide an initial data map. Based on the relationship, fill the first association table structure into the initial data map to obtain the target data map corresponding to the target database.
[0030] In this step, the data map is initialized as a blank data map with a data map structure but without any data. It includes multiple data nodes and edges connecting the data nodes. The data nodes are used to store corresponding data, while the edges are used to represent the relationships between the data nodes.
[0031] Based on the aforementioned initialization data map, in this step, filling the first associated table structure into the initialization data map based on the association relationships specifically involves: based on the association relationships and the distribution of edges in the initialization data map, filling the first associated table structure into each node of the initialization data map. After all table structure data is filled into the initialization data map, the initialization data map is converted into a target data map corresponding to the target database. For example, for a first associated table structure (table A and table B) with an association relationship, two nodes connected by an edge are selected in the initialization data map, and tables A and B are stored in these two nodes respectively.
[0032] Compared with existing technologies, the data map construction method, apparatus, electronic device, and computer-readable storage medium provided in this application, when constructing a data map, select a first related table structure with a relationship from multiple table structures of the target database based on a large language model according to the table structure data. Then, the first related table structure is filled into a blank initial data map to obtain a target data map corresponding to the target database. The automatic selection of the first related table structure based on the large language model replaces traditional manual analysis or fixed rule matching, eliminating the need for technicians to analyze the semantics and business logic of fields table by table, significantly shortening the relationship identification cycle. Simultaneously, the automatic filling of the first related table structure into the initial data map based on the relationship eliminates the manual input step, thereby significantly improving the efficiency of data map construction. Because the large language model has powerful semantic understanding capabilities, it can… This approach breaks free from the reliance on fixed field naming rules, accurately identifies relationships between table structures, and reduces omissions and misjudgments caused by differences in human experience or rule limitations. It also automatically fills the initial data map with the structure of the first related table, further reducing formatting errors and misalignments during manual data entry, thus improving the accuracy of the target data map. Furthermore, this application reduces reliance on highly experienced technical personnel through intelligent and automated processes, lowering labor costs. The improved efficiency and accuracy of data map construction also reduce time and resource consumption from rework, significantly lowering the cost of data map construction. Additionally, when the table structure in the target database changes, the large language model can quickly re-identify relationships and update the target data map without manually re-organizing the entire table structure, enabling dynamic and efficient updates. This solves the technical problems of inefficiency, error-proneness, high cost, and difficulty in dynamic updates inherent in existing data map construction methods.
[0033] Please refer to Figure 2 This application also provides a method for constructing a data map, the specific steps of which include: Step S201: Obtain table structure data for multiple table structures in the target database.
[0034] Step S202: Based on the large language model, select the first related table structure from multiple table structures according to the table structure data.
[0035] Step S203: Provide an initial data map. Based on the relationship, fill the first association table structure into the initial data map to obtain the target data map corresponding to the target database.
[0036] It is understood that steps S201 to S203 in this embodiment are largely the same as steps S101 to S103 in the previous embodiment. For details, please refer to the specific description in the previous embodiment, which will not be repeated here.
[0037] Step S204: Based on the community detection algorithm, divide the table structure in the target data map into communities to obtain multiple table structure communities.
[0038] Community detection algorithms are used to identify substructures (i.e., "communities") with tightly connected internal structures and sparse external connections from complex networks composed of nodes and edges. The core logic is to classify nodes in the network, i.e., to divide them into communities, utilizing the network's "clustering" characteristic. In this step, this corresponds to dividing the table structures in the target data map into communities, which can be represented as Community_j in this embodiment, where j is the community name. Specifically, the community detection algorithm used in this step is the Leiden algorithm. The Leiden algorithm achieves efficient clustering through three steps: "local optimization - community aggregation - global optimization." First, it fine-tunes the local node affiliation to improve modularity; then, it aggregates communities into supernodes to construct a new network; finally, it optimizes the new network, balancing efficiency and accuracy, and can more accurately identify tightly connected substructures in the network. It is understood that the aforementioned community detection algorithm, specifically the Leiden algorithm, is only an example of a community detection algorithm in this embodiment. In other embodiments of this application, the community detection algorithm may also be the Louvain algorithm, a hierarchical clustering method, a label propagation algorithm, or other algorithms, which can be flexibly selected according to actual needs.
[0039] Step S205: For any table structure community, generate community business description information for each table structure community based on the table structure data in the table structure community using the large language model.
[0040] In this step, the corresponding Prompt can also be constructed in the large language model to obtain the community business description information of each table structure community output by the large language model. For example, in this embodiment, a Prompt can be constructed as follows: "You are a business analyst. The following are the table information of multiple tables belonging to the same business domain: [List the table names and business descriptions of all tables in the table structure community Community_j]. Please analyze these tables comprehensively to generate a Chinese paragraph that comprehensively describes the core objectives, main data, and processes of this business domain (community)."
[0041] Furthermore, in this embodiment, after generating community business description information for each table-structured community, the community business description information can be semantically vectorized to obtain community business description vectors. These vectors are then stored in a vector database, corresponding to the table-structured communities. Semantic vectorization is a technique that converts textual information into computer-processable numerical vectors. Its core is to allow vectors to carry semantic information, mapping semantic similarity to the distance between different vectors in the vector space. Converting textual community business description information into a computable vector form allows for more efficient retrieval of this information, with high computational efficiency, making it more suitable for large-scale data processing.
[0042] Step S206: Generate a global data domain description of the target data map based on the large language model and all community business description information.
[0043] In this step, a global data domain description of the target data map output by the large language model can also be obtained by constructing a corresponding Prompt in the large language model. For example, in this embodiment, a Prompt can be constructed as follows: "You are an enterprise architect. The following are descriptions of several major business areas in the enterprise data warehouse: [List the community business description information of all table structure communities]. Please comprehensively analyze these business areas and generate a Chinese summary describing the overall goals, scope, and core value of the entire enterprise data domain (or theme)."
[0044] Furthermore, in this embodiment, after generating the global data domain description of the target data map, the global data domain description can be semantically vectorized to obtain a global description vector, and the global description vector and the target data map can be stored in a vector database.
[0045] Compared with related technologies, since this embodiment includes the technical content of the foregoing embodiments, it also has the technical effects of the foregoing embodiments. For details, please refer to the specific descriptions in the foregoing embodiments, which will not be repeated here. In addition, in this embodiment, based on the large language model, the table structure in the target data map is divided into communities according to the community detection algorithm to obtain multiple table structure communities. Based on the large language model, community business description information of each table structure community is generated according to the table structure data in the table structure communities. Based on the large language model, a global data domain description of the target data map is generated according to all community business description information. This can form a hierarchical business semantic description of table structure -> community -> global, which can better describe the data map and facilitate the querying of data in the data map.
[0046] Please refer to Figure 3 This application also provides a data map construction method, which specifically includes the following steps: Step S301: Obtain the table structure data of multiple table structures in the target database.
[0047] Step S302: Based on the large language model, select the first related table structure from multiple table structures according to the table structure data.
[0048] Step S303: Provide an initial data map. Based on the relationship, fill the first association table structure into the initial data map to obtain the target data map corresponding to the target database.
[0049] It is understood that steps S301 to S303 in this embodiment are largely the same as steps S101 to S103 in the previous embodiment. For details, please refer to the specific description in the previous embodiment, which will not be repeated here.
[0050] Step S304: Obtain SQL log data related to multiple table structures in the target database. The SQL log data shall include at least Join operation data.
[0051] In this step, the SQL log data is a structured file that records the target database's own operations, states, and events. Specifically, this step involves retrieving SQL log data related to table structure operations. In this embodiment, at least the Join operation data related to the table structure needs to be retrieved from the SQL log data. The storage format of the Join operation data is as follows: { "timestamp": "2023-10-15 09:30:25", "sql": "SELECT o.order_id, c.name FROM orders o JOIN customers c ON o.customer_id = c.id", "execution_count": 128 } Step S305: Select the second related table structure from multiple table structures based on the Join operation data.
[0052] As shown above, the Join operation data must include at least the operation processes of orders o and orders c. In this step, it can be determined that orders o and orders c are second related table structures with a relationship.
[0053] Similarly, in this embodiment, the association relationship of the second association table structure can also be verified. For details, please refer to the specific description of verifying the association relationship of the first association table structure in the foregoing embodiment.
[0054] Step S306: Fill the second association table structure into the initial data map based on the association relationship.
[0055] This step S306 is largely the same as step S103 in the previous embodiment, and you can refer to the specific description in the previous embodiment for details.
[0056] Compared with related technologies, since this embodiment includes the technical content of the foregoing embodiments, it also has the technical effects of the foregoing embodiments. For details, please refer to the specific descriptions in the foregoing embodiments, which will not be repeated here. In addition, in this embodiment, a second related table structure with a relationship is selected from multiple table structures based on the Join operation data in the SQL log data of the target database. This can provide more diverse ways to determine the relationship between table structures, thereby discovering more related table structures in the target database and improving the completeness and accuracy of the data map.
[0057] Please refer to Figure 4 This application provides a data map construction apparatus, including: Data acquisition module 401 is used to acquire table structure data of multiple table structures in the target database; The association relationship determination module 402 is used to select the first associated table structure with an association relationship from multiple table structures based on the table structure data according to the large language model; The data map construction module 403 is used to provide an initial data map. Based on the relationship, the structure of the first association table is filled into the initial data map to obtain the target data map corresponding to the target database.
[0058] The data map construction apparatus provided in the above embodiments can implement the technical solutions described in the above data map construction method embodiments. The specific implementation principles of each module or unit can be found in the corresponding content in the above data map construction method embodiments, and will not be repeated here.
[0059] Furthermore, in this application, the data acquisition module 401 is also used to acquire SQL log data related to multiple table structures in the target database, and the SQL log data includes at least Join operation data; the relationship determination module 402 is also used to select a second related table structure with a relationship from multiple table structures based on the Join operation data; the data map construction module 403 is also used to fill the second related table structure into the initial data map based on the relationship.
[0060] It also includes a community segmentation module 404, which is used to segment the table structure in the target data map into communities based on the community discovery algorithm, resulting in multiple table structure communities; and a description information generation module 405, which is used to generate community business description information for each table structure community based on the table structure data in the table structure communities using a large language model; and to generate a global data domain description of the target data map based on all community business description information using a large language model.
[0061] It also includes a data completion module 406, which is used to complete table structure data based on a large language model.
[0062] The data validation module 407 is used to check whether the overlap rate of the table data in the two first associated table structures is greater than or equal to a preset threshold. If the overlap rate is greater than or equal to the preset threshold, the association relationship is determined to exist; if the overlap rate is less than the preset threshold, the association relationship is determined to not exist. The data validation module 407 is also used to check whether there are explicit foreign key constraints in the two first associated table structures. If explicit foreign key constraints exist, the association relationship is determined to exist; if no explicit foreign key constraints exist, the association relationship is determined to not exist.
[0063] Please refer to Figure 5 This application also provides an electronic device 500. The electronic device 500 includes a processor 501, a first memory 502, a second memory 503, and a display 504. Figure 5 Only some of the components of the electronic device 500 are shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead.
[0064] In some embodiments, processor 501 may be a central processing unit (CPU), microprocessor, or other data processing chip, used to run program code stored in memory 502 or process data, such as the data map construction method in this application.
[0065] In some embodiments, the processor 501 may be local or remote. In some embodiments, the processor 501 may be implemented on a cloud platform. In one embodiment, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an intranet, a multi-cloud, or any combination thereof.
[0066] In some embodiments, the first memory 502 and the second memory 503 may be internal storage units of the electronic device 500, such as a hard disk or memory of the electronic device 500. In other embodiments, the first memory 502 and the second memory 503 may also be external storage devices of the electronic device 500, such as plug-in hard disks, smart media cards (SMC), secure digital (SD) cards, flash cards, etc., equipped on the electronic device 500.
[0067] Furthermore, the first memory 502 and the second memory 503 may include both internal storage units of the electronic device 500 and external storage devices. The first memory 502 is used to store application software and various types of data installed on the electronic device 500; the second memory 503 is used to store user information data based on the aforementioned user information storage method.
[0068] In some embodiments, display 504 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 504 is used to display information from electronic device 500 and to display a visual user interface. Components 501-504 of electronic device 500 communicate with each other via a system bus.
[0069] In one embodiment, when the processor 501 executes the data map construction program in the first memory 502, the following steps can be implemented based on the user information data stored in the second memory 503: Retrieve table structure data for multiple table structures in the target database; Based on the large language model, the first related table structure with a relationship is selected from multiple table structures according to the table structure data; An initial data map is provided. Based on the relationship, the structure of the first association table is filled into the initial data map to obtain the target data map corresponding to the target database.
[0070] It should be understood that when the processor 501 executes the data map construction program in the first memory 502, in addition to the functions mentioned above, it can also perform other functions, as detailed in the description of the corresponding method embodiments above.
[0071] Furthermore, this application does not specifically limit the type of electronic device 500 mentioned in the embodiments. Electronic device 500 can be a mobile phone, tablet computer, personal digital assistant (PDA), wearable device, laptop computer, or other portable electronic devices. Exemplary embodiments of portable electronic devices include, but are not limited to, portable electronic devices running iOS, Android, Microsoft, or other operating systems. The aforementioned portable electronic device can also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of this application, electronic device 500 may not be a portable electronic device, but rather a desktop computer with a touch-sensitive surface (e.g., a touch panel).
[0072] Accordingly, this application also provides a computer-readable storage medium for storing computer-readable programs or instructions. When the programs or instructions are executed by a processor, they can implement the steps or functions in the user information storage method and data map construction method provided in the above-described method embodiments.
[0073] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.), and the computer program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.
[0074] The user information storage method, data map construction method, data map construction device, electronic device, and computer-readable storage medium provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A method for constructing a data map, characterized in that, include: Retrieve table structure data for multiple table structures in the target database; Based on the large language model, the first related table structure with an association relationship is selected from the multiple table structures according to the table structure data; An initial data map is provided, and the structure of the first association table is filled into the initial data map based on the association relationship to obtain a target data map corresponding to the target database.
2. The data map construction method according to claim 1, characterized in that, Also includes: Obtain SQL log data related to the multiple table structures in the target database, wherein the SQL log data includes at least Join operation data; Based on the Join operation data, select a second related table structure from the plurality of table structures that has a relationship; Based on the aforementioned relationship, the structure of the second association table is filled into the initial data map.
3. The data map construction method according to claim 1, characterized in that, Also includes: Based on the community detection algorithm, the table structure in the target data map is divided into communities, resulting in multiple table structure communities. For any of the table structure communities, community business description information for each table structure community is generated based on the table structure data in the table structure community using a large language model. Based on a large language model, a global data domain description of the target data map is generated according to all the community business description information.
4. The data map construction method according to claim 3, characterized in that, Also includes: For any of the table structure communities, the community business description information is semantically vectorized to obtain a community business description vector, and the community business description vector is stored in a vector database corresponding to the table structure community. For any of the target data maps, the global data domain description is semantically vectorized to obtain a global description vector, and the global description vector is stored in the vector database in correspondence with the target data map.
5. The data map construction method according to claim 1, characterized in that, Before selecting the first related table structure with an association relationship from the multiple table structures based on the table structure data using the large language model, the method further includes: For any of the table structure data, complete the table structure data based on the large language model.
6. The data map construction method according to claim 1, characterized in that, Before filling the first association table structure into the initial data map based on the association relationship, the method further includes: For any two first association table structures that have the aforementioned relationship, check whether the overlap rate of table data in the two first association table structures is greater than or equal to a preset threshold. If the overlap rate of the table data is greater than or equal to the preset threshold, it is determined that the association relationship exists; If the overlap rate of the table data is less than the preset threshold, it is determined that the association relationship does not exist.
7. The data map construction method according to claim 1, characterized in that, Before filling the first association table structure into the initial data map based on the association relationship, the method further includes: For any two first association table structures that have the aforementioned relationship, check whether there is an explicit foreign key constraint between the two first association table structures; If an explicit foreign key constraint exists, the association relationship is confirmed to exist. If no explicit foreign key constraint exists, the association relationship is determined to be non-existent.
8. A data map construction device, characterized in that, include: The data acquisition module is used to acquire table structure data of multiple table structures in the target database; The association relationship determination module is used to select a first associated table structure with an association relationship from the multiple table structures based on the table structure data according to the large language model; A data map construction module is used to provide an initial data map, which fills the first association table structure into the initial data map based on the association relationship, so as to obtain a target data map corresponding to the target database.
9. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the data map construction method as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the data map construction method as described in any one of claims 1-7.