Dimensional data model quality assessment method, device, storage medium and program product

Abstract

The present application provides a dimensional data model quality assessment method, device, storage medium and program product, which relates to the field of database technology. The method includes: first, determining the dimension table and fact table of the dimensional data model based on the metadata of multiple data tables; then, determining the graph structure model corresponding to the dimensional data model according to the dimension table and the fact table; and performing quality assessment on the dimensional data model according to the graph structure model. In an embodiment of the present application, the graph structure model corresponding to the dimensional data model is obtained, and the quality of the dimensional data model is assessed based on the graph structure model, which can realize automatic assessment of the quality of the dimensional data model and improve the efficiency, flexibility and accuracy of the quality assessment of the dimensional data model.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a dimensional data model quality assessment method, device, storage medium and program product. Background Art

[0002] In the context of cloud networks, the explosive growth of data volume and the increasing complexity of data application scenarios have directly affected the analysis efficiency of the data warehouse and the accuracy of decision support. The dimensional data model is a core component of data warehouse design, which organizes data into fact tables and dimension tables for efficient query and analysis. In data warehouse design, the construction of dimensional data models is a core and complex task that supports upper-level business operations such as instance diagnosis and damaged impact surface extraction. Therefore, it needs to be highly complete and accurate to avoid data inconsistency and logical errors.

[0003] Currently, the quality assessment method of dimensional data models relies on manual rule setting and manual review, which is not only time-consuming and labor-intensive, but also difficult to cope with the complexity and dynamic changes of cloud network services. Summary of the invention

[0004] The embodiments of the present application provide a dimensional data model quality assessment method, device, storage medium and program product to realize automatic assessment of dimensional data model quality, thereby improving the efficiency, flexibility and accuracy of quality assessment.

[0005] In a first aspect, an embodiment of the present application provides a method for evaluating the quality of a dimensional data model, comprising: determining a dimension table and a fact table of the dimensional data model based on metadata of multiple data tables; determining a graph structure model corresponding to the dimensional data model based on the dimension table and the fact table; and performing quality evaluation on the dimensional data model based on the graph structure model.

[0006] In a second aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the above methods when executing the computer program.

[0007] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, any of the above methods is implemented.

[0008] In a fourth aspect, an embodiment of the present application provides a computer program product, the computer program product including a computer program, and the computer program implements any of the above methods when executed by a processor.

[0009] Compared with the prior art, this application has the following advantages:

[0010] The present application provides a dimensional data model quality assessment method, device, storage medium and program product. First, based on the metadata of multiple data tables, the dimension table and fact table of the dimensional data model are determined; then, according to the dimension table and fact table, the graph structure model corresponding to the dimensional data model is determined; according to the graph structure model, the quality of the dimensional data model is assessed. In an embodiment of the present application, the graph structure model corresponding to the dimensional data model is obtained, and the quality of the dimensional data model is assessed based on the graph structure model, which can realize automatic assessment of the quality of the dimensional data model and improve the efficiency, flexibility and accuracy of the quality assessment of the dimensional data model.

[0011] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments according to the present application and should not be regarded as limiting the scope of the present application.

[0013] Figure 1 A schematic diagram of an application scenario of a dimensional data model quality assessment method according to an embodiment of the present application.

[0014] Figure 2 This is a flowchart of a dimensional data model quality assessment method according to an embodiment of the present application.

[0015] Figure 3 This is a flowchart of a dimensional data model quality assessment method according to an embodiment of the present application.

[0016] Figure 4 This is a structural block diagram of a dimensional data model quality assessment device according to an embodiment of the present application.

[0017] Figure 5 A block diagram of an electronic device used to implement an embodiment of the present application. DETAILED DESCRIPTION

[0018] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the concept or scope of the present application. Therefore, the drawings and descriptions are considered to be exemplary in nature and not restrictive.

[0019] To facilitate understanding of the technical solutions of the embodiments of the present application, the following describes the related technologies of the embodiments of the present application. The following related technologies can be combined with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application.

[0020] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0021] In the related art, the quality assessment of dimensional data models relies on predefined rule sets for data inspection, which requires manual creation and maintenance of a large number of rules, making it difficult to adapt to changes in complex dimensional data models and lacking flexibility and intelligence. In the embodiment of the present application, a graph structure model corresponding to the dimensional data model is obtained, and the quality of the dimensional data model is assessed based on the graph structure model, which can realize automatic assessment of the quality of the dimensional data model, reduce the need for manual intervention, and improve the flexibility, real-time and accuracy of verification.

[0022] Figure 1 A schematic diagram of an application scenario of the dimensional data model quality assessment method provided in this application. Figure 1 As shown, it includes a stream processing layer, a data storage layer, a large model verification layer, and a post-operation layer.

[0023] In the stream processing layer, the change data capture (CDC) capability of the stream processing engine (e.g., Flink engine) is used to capture data change operations (e.g., insert, update, delete, etc.) in the preset data, and obtain the modeling logic of the table structure corresponding to the data change operation from the preset database. The preset database may include but is not limited to a pre-built relational database. The modeling logic specifically includes metadata, which includes but is not limited to: the table name, field name, field type, annotation information, etc. corresponding to the source table and the target table respectively. The annotation information includes the association relationship between different data tables. The data change operation and metadata are converted into SQL (Structured Query Language) statements through JAVA code. The stream processing engine obtains the table content data corresponding to the metadata from the source table database, executes the SQL statement, outputs the actual data of the dimensional model, and stores it in the dimensional model data warehouse.

[0024] In the data storage layer, the knowledge graph technology is used to model a graph structure database. Specifically, entity recognition is performed in the metadata to obtain the dimension table of the dimensional data model, and the relationship between entities is identified in the metadata to obtain the fact table of the dimensional data model. The dimension table is mapped to the node in the graph structure model, and the annotation information in the metadata is obtained. Based on the annotation information, the foreign key information in the fact table is mapped to the edge in the graph structure model to obtain the graph structure model, such as Figure 1 The "extract the overall graph structure" shown in is saved in the graph structure database. The common subgraph structure in the graph structure model is identified by using a community detection algorithm (e.g., Louvain algorithm) and saved. Specifically, the Louvain algorithm is run on the graph structure model to identify the community. For each identified community, its internal connection pattern, node attributes and other characteristics are analyzed to determine which are repeated or representative subgraph structures. The graph pattern matching technology is used to identify the similarities between different communities and find the common subgraph structure that can be reused in multiple scenarios. The common subgraph structure can be used for subsequent graph structure model construction to improve the efficiency of graph structure model construction. Specifically, when constructing a new graph structure model, the common subgraph structure can be inserted according to specific needs, which can not only speed up the modeling process, but also ensure the consistency and standardization of the graph structure model. The common subgraph structure can also be used to check the structural integrity of the graph structure model to ensure the connectivity and rationality of the graph structure model. Specifically, when it is necessary to verify the connectivity and rationality of the graph structure model, the newly constructed graph structure model can be compared with the existing general subgraph structure to check whether there are isolated nodes, ensure the connectivity of the graph structure model, check whether the loop meets expectations, and prevent potential data flow deadlocks. In addition, the graph structure database is used to save metadata for generating training samples for large language models (LLMs), such as Figure 1 The data storage layer also includes a database and a dimensional model data warehouse for storing source tables, where the source table is the table content data corresponding to the metadata, such as Figure 1 Source table A…source table N shown in .

[0025] In the large model verification layer, the overall graph structure is extracted from the graph structure database, that is, the graph structure model is obtained, and the graph traversal algorithm and template are used to generate natural language, that is, description information in natural language form. Specifically, the graph traversal algorithm includes but is not limited to depth-first search (DFS) and breadth-first search (BFS). The graph traversal algorithm is used to access all nodes in the graph structure model, and the node description information is generated according to the access order of the nodes. For example, after traversing and accessing a certain node, the nodes directly connected to the node and their connection relationships can be described. Obtain a pre-defined template and convert the node description information into description information in natural language form according to the template.

[0026] The description information and quality assessment operation information in natural language form are converted into a logical reasoning path through LLM understanding. In this embodiment, the logical reasoning path is an SQL statement, wherein the quality assessment operation information refers to the information for operating the data table for quality assessment, which can be set according to specific needs. For example, operations such as query, insert, and delete. The LLM in this embodiment is obtained by continuing to train the open source large language model using training samples. The training sample is obtained in the following manner: using the metadata stored in the graph structure database to determine the sample graph structure, converting the sample graph structure into sample description information in natural language form, obtaining the sample operation information and the SQL statement corresponding to the sample operation information, and using the sample description information, the sample operation information, and the SQL statement corresponding to the sample operation information as the training sample. Extract metadata from the database storing the source table, and sample the table content data of the source table to obtain the sampled source data. Based on the metadata and the sampled source data, use the SQL parsing library to execute the SQL statement output by the LLM to obtain the execution result. Obtain the corresponding metadata and sampling result data from the dimensional model data warehouse, that is, the result table corresponding to the quality assessment operation information. Compare the execution results with the result table, determine the quality of the dimensional data model based on the comparison results, implement data integrity and logical consistency checks, and generate an inspection result report.

[0027] In the post-operation layer, quality monitoring is performed on dimensional data models that have passed the inspection. Specifically, a data change message queue is established to continuously obtain data change events and SQL execution logs. Based on data change events and SQL execution logs, the data change results in the SQL parsing library are monitored. For example, the data quality of the change results is detected using a preset rule engine, including but not limited to: data integrity, accuracy, missing values, outlier detection, etc. In addition, logical consistency checks are also included. If the data change results do not meet the preset standards, an alarm notification and inspection result report are sent through a callback mechanism or email. The error type, location, and recommended repair plan are recorded in the inspection result report. The data execution results are corrected according to the inspection result report to improve response efficiency.

[0028] In the embodiment of the present application, the quality assessment of the dimensional data model is performed based on the graph structure model, which overcomes the limitations of the related technology that relies on manual rule setting and manual review, and improves the efficiency of dimensional data model verification. At the same time, the large language model is used to deeply understand the logical relationship of the dimensional data model and perform intelligent reasoning, which significantly improves the accuracy and automation level of logical consistency verification. Therefore, this embodiment can effectively solve the efficiency, flexibility and accuracy problems in the quality assessment of complex dimensional data models, and ensure the high quality of dimensional data models in data warehouses.

[0029] An embodiment of the present application provides a method for evaluating the quality of a dimensional data model. The method in this embodiment can be applied to servers, terminal devices, platforms, apparatuses, etc. with computing and processing capabilities, wherein the server can be a server cluster or a single server, a server deployed in the cloud, or a local server.

[0030] like Figure 2 The flowchart of the dimensional data model quality assessment method according to an embodiment of the present application is shown, including:

[0031] Step S201: Determine dimension tables and fact tables of a dimensional data model based on metadata of multiple data tables.

[0032] The metadata of a data table refers to information describing the data table and its contents, including but not limited to: table name, field name, field type, annotation information, etc. The annotation information includes the association relationship between different data tables.

[0033] Step S202: Determine the graph structure model corresponding to the dimensional data model according to the dimension table and the fact table.

[0034] The fact table stores specific service measurement data, such as sales volume, revenue, cost, etc., and usually contains a large amount of numerical data to quantify service activities. The dimension table stores descriptive information about the data in the fact table, such as time, location, product, user, etc.

[0035] According to the information stored in the dimension table and fact table, mapping processing is performed to obtain a graph structure model.

[0036] Step S203: perform quality assessment on the dimensional data model according to the graph structure model.

[0037] Based on the graph structure model, the logical reasoning path of the dimensional data model is determined, and then the quality of the dimensional data model is evaluated, such as data integrity and logical consistency verification.

[0038] The dimensional data model quality assessment method provided in the embodiment of the present application first determines the dimension table and fact table of the dimensional data model based on the metadata of multiple data tables; then, determines the graph structure model corresponding to the dimensional data model according to the dimension table and the fact table; and performs quality assessment on the dimensional data model according to the graph structure model. In the embodiment of the present application, the graph structure model corresponding to the dimensional data model is obtained, and the quality assessment of the dimensional data model is performed based on the graph structure model, which can realize automatic assessment of the quality of the dimensional data model and improve the efficiency, flexibility and accuracy of the quality assessment of the dimensional data model.

[0039] The following introduces the specific implementation process of the above steps through multiple implementation methods:

[0040] In one implementation, step S201, based on metadata of multiple data tables, determines dimension tables and fact tables of a dimensional data model, including: step S2011, performing entity recognition in metadata to obtain dimension tables. step S2012, performing relationship recognition between entities in metadata to obtain fact tables.

[0041] In practical applications, metadata is parsed, and entity recognition models are used to identify entities in metadata, and tables containing multiple entities are identified as dimension tables. Entities include but are not limited to: time, location, product, user, etc. The entity recognition model can be trained using a neural network model. The relationship between entities in the metadata is identified, and tables containing quantitative data are identified as fact tables.

[0042] In this embodiment, through entity recognition and relationship recognition, dimension tables and fact tables can be obtained as the basis for constructing a graph structure model.

[0043] In one implementation, step S202, determining the graph structure model corresponding to the dimensional data model according to the dimension table and the fact table, includes: step S2021, mapping the dimension table to a node in the graph structure model. Step S2022, mapping the foreign key information in the fact table to an edge in the graph structure model to obtain the graph structure model.

[0044] Among them, the foreign key information is the value of the foreign key field in the fact table, and the value of the foreign key field corresponds to the value of the primary key field in the dimension table. The primary key field is a unique identification field in the dimension table, which is used to uniquely identify each row of records in the table. Through foreign keys, the fact table can establish an association relationship with one or more dimension tables. The foreign key field constraint can ensure that the value of the foreign key field in the fact table exists in the dimension table, thereby maintaining the integrity of the data. Through foreign key fields, the quantitative data in the fact table can be combined with the descriptive information in the dimension table for multidimensional analysis. By mapping the dimension table and the fact table to nodes and edges respectively, a graph structure model corresponding to the dimensional data model is obtained, so that the quality of the dimensional data model can be evaluated using the graph structure model.

[0045] In one implementation, in step S2022, mapping the foreign key information in the fact table to the edge in the graph structure model includes: obtaining annotation information in a table configuration file and / or metadata; the table configuration file is used to configure the association relationship between multiple data tables. According to the table configuration file and / or the annotation information, mapping the foreign key information in the fact table to the edge in the graph structure model.

[0046] In practical applications, the mapping rules for mapping foreign key information in the fact table to edges can be determined based on the annotation information in the table configuration file and / or metadata. The table configuration file can be a file that pre-configures the association relationship between multiple tables according to service needs to cope with complex service logic. The annotation information in the metadata records the simple association relationship between different data tables, and the mapping rules can also be determined by the annotation information. Therefore, according to needs, you can choose to determine the mapping rules based on the table configuration file and / or annotation information to map the foreign key information in the fact table to the edges in the graph structure model.

[0047] In this embodiment, the foreign key information in the fact table can be mapped to the edge in the graph structure model according to the table configuration file and / or annotation information, which can meet the needs of service logic of different complexity and has flexibility.

[0048] In one implementation, step S203, performing quality assessment on the dimensional data model according to the graph structure model, includes: step S2031, converting the graph structure model into description information in natural language form; step S2032, inputting the description information and quality assessment operation information into a large language model to obtain a logical reasoning path corresponding to the quality assessment operation information output by the large language model, wherein the large language model is trained using sample description information in natural language form of a sample graph structure, sample operation information, and sample logical reasoning paths corresponding to the sample operation information; step S2033, performing quality assessment on the dimensional data model according to the logical reasoning path.

[0049] Specifically, a graph traversal algorithm and a template are used to generate description information in natural language. The graph traversal algorithm includes but is not limited to DFS and BFS. A graph traversal algorithm is used to access all nodes in the graph structure model, and node description information is generated according to the access order of the nodes. For example, after traversing and accessing a certain node, the nodes directly connected to the node and their connection relationships can be described. A predefined template is obtained, and the node description information is converted into description information in natural language according to the template.

[0050] In one example, the graph structure model is as follows: A – B, A – C, B – D; the generated description information in natural language form is: "Node A is connected to nodes B and C, and node B is connected to node D."

[0051] The description information and quality assessment operation information are input into the large language model, understood by the large language model, and the logical reasoning path output by the large language model is obtained. Among them, the quality assessment operation information refers to the information for operating the data table for quality assessment, which can be set according to specific needs. For example, query, insert, delete and other operations. Among them, the logical reasoning path refers to the various steps of the reasoning process to obtain the final result based on the quality assessment operation information and the graph structure model. Each step is a reasonable inference based on the result of the previous step and the connection relationship between the nodes and edges in the graph structure model. The logical reasoning path can be code written in multiple programming languages. According to the logical reasoning path, forward reasoning from the source table to the result table and reverse reasoning from the result table to the source table can be performed to realize the quality assessment of the dimensional data model. Specifically, forward logical reasoning: the data of the source table is converted into the result table through SQL query or other data processing methods. For example, aggregation operation, connection (JOIN), filtering (WHERE), sorting, etc. Reverse logical reasoning: Determine the field composition and meaning of the result table, obtain the SQL query used to generate the result table or the script used for the data extraction, transformation, and loading (ETL) process, and trace back to the source table based on the field composition and meaning of the result table, as well as the SQL query or ETL script used to generate the result table, to check whether the expected results can be obtained from the source table.

[0052] Among them, the large language model in this embodiment can be obtained by training a pre-trained large language model using sample description information in the natural language form of the sample graph structure, sample operation information and sample logical reasoning paths corresponding to the sample operation information as training samples.

[0053] In this embodiment, the graph structure model is used as a bridge between the large language model and the dimensional data model to promote effective interaction between the two. The large language model is used to understand the logical relationship of the dimensional data model and output the logical reasoning path, which can improve the accuracy and automation level of the dimensional data model quality assessment results.

[0054] In one implementation, step S2033, the logical reasoning path includes SQL statements, and the quality of the dimensional data model is evaluated according to the logical reasoning path, including: based on the metadata and the table content data corresponding to the metadata, using the SQL parsing library to execute the SQL statement to obtain the execution result; according to the execution result, the quality of the dimensional data model is evaluated.

[0055] The logical reasoning path may be a code written in SQL, i.e., an SQL statement. The training sample of the large language model that outputs the SQL statement is obtained in the following manner: using metadata to determine the sample graph structure, converting the sample graph structure into sample description information in natural language form, obtaining sample operation information and the SQL statement corresponding to the sample operation information, and using the sample description information, the sample operation information, and the SQL statement corresponding to the sample operation information as training samples.

[0056] Use the SQL parsing library to read metadata and the table content data corresponding to the metadata, execute SQL statements, obtain execution results, and judge the quality of the dimensional data model based on the execution results.

[0057] In this embodiment, the quality assessment of the dimensional data model is implemented using an SQL parsing library, which has SQL optimization capabilities and an error handling mechanism, and can ensure the accuracy of the quality assessment results.

[0058] In one implementation, quality assessment is performed on the dimensional data model based on the execution result, including: obtaining a result table corresponding to the quality assessment operation information in the data warehouse of the dimensional data model; comparing the execution result with the result table, and determining the quality assessment result of the dimensional data model based on the comparison result.

[0059] In practical applications, the stream processing engine Flink is used to capture data change operations (such as insert, update, delete, etc.) in relational data. The data change operations and metadata are converted into Flink SQL statements through JAVA code. The Flink engine obtains the table content data corresponding to the metadata from the source table database, executes the Flink SQL statement, obtains the result table, and stores it in the data warehouse of the dimensional data model. The result table corresponding to the quality assessment operation information is obtained in the data warehouse of the dimensional data model; the execution result of the SQL parsing library is compared with the result table, and the quality assessment result of the dimensional data model is determined according to the matching degree between the execution result of the SQL parsing library and the result table, so as to realize the data integrity and logical consistency verification of the dimensional data model. If the matching degree is low or does not match, it means that the quality of the dimensional data model does not meet expectations.

[0060] In one implementation, after performing a quality assessment on the dimensional data model according to the graph structure model, the dimensional data model quality assessment method further includes: obtaining data change events and SQL execution logs of the data warehouse of the dimensional data model based on a message queue corresponding to the data change; monitoring data change results in the data warehouse of the dimensional data model based on the data change events and the SQL execution log; and correcting the data change results if the data change results do not meet preset standards.

[0061] Specifically, a data change message queue is established to continuously obtain data change events and SQL execution logs. Based on data change events and SQL execution logs, the data change results in the SQL parsing library are monitored. For example, the data quality of the change results is detected using a preset rule engine, including but not limited to: data integrity, accuracy, missing values, outlier detection, etc. In addition, a logical consistency check is also performed. If the data change results do not meet the preset standards, an alarm notification and an inspection result report are sent through a callback mechanism or email. The error type, location, and recommended repair plan are recorded in the inspection result report. The data execution results are corrected according to the inspection result report to improve response efficiency.

[0062] An embodiment of the present application provides a method for evaluating the quality of a dimensional data model. The method in this embodiment can be applied to servers, terminal devices, platforms, apparatuses, etc. with computing and processing capabilities, wherein the server can be a server cluster or a single server, a server deployed in the cloud, or a local server.

[0063] like Figure 3 The flowchart of the dimensional data model quality assessment method according to an embodiment of the present application is shown, including:

[0064] Step S301, obtaining metadata of multiple data tables, performing entity recognition in the metadata, and obtaining a dimension table.

[0065] Step S302: Identify the relationships between entities in the metadata to obtain a fact table.

[0066] Step S303: Map the dimension table to a node in the graph structure model.

[0067] Step S304: Obtain annotation information in the table configuration file and / or metadata, and map the foreign key information in the fact table to edges in the graph structure model according to the table configuration file and / or the annotation information.

[0068] Step S305: convert the graph structure model into description information in natural language form.

[0069] Step S306: input the description information and the quality assessment operation information into the large language model to obtain a logical reasoning path corresponding to the quality assessment operation information output by the large language model, wherein the logical reasoning path includes an SQL statement.

[0070] Step S307: Based on the metadata and the table content data corresponding to the metadata, the SQL statement is executed using the SQL parsing library to obtain an execution result.

[0071] Step S308, obtaining a result table corresponding to the quality assessment operation information in the data warehouse of the dimensional data model; comparing the execution result with the result table, and determining the quality assessment result of the dimensional data model according to the comparison result.

[0072] In an embodiment of the present application, a graph structure model is used as a bridge between the dimensional data model and the large language model. The large language model is used to understand the graph structure model, obtain a logical reasoning path, and then perform a quality assessment on the dimensional data model. This can achieve automatic quality assessment of the dimensional data model, thereby improving the efficiency, flexibility, and accuracy of quality assessment of the dimensional data model.

[0073] The specific implementation process of each step in the embodiment of the present application can refer to the corresponding description in the above embodiment, and has corresponding beneficial effects, which will not be repeated here.

[0074] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides a dimensional data model quality assessment device. Figure 4 The figure is a structural block diagram of a dimensional data model quality assessment device according to an embodiment of the present application, the device comprising:

[0075] The table determination module 401 is used to determine the dimension table and fact table of the dimensional data model based on the metadata of multiple data tables.

[0076] The graph determination module 402 is used to determine the graph structure model corresponding to the dimensional data model according to the dimension table and the fact table.

[0077] The quality assessment module 403 is used to perform quality assessment on the dimensional data model according to the graph structure model.

[0078] The dimensional data model quality assessment device provided in the embodiment of the present application first determines the dimension table and fact table of the dimensional data model based on the metadata of multiple data tables; then, determines the graph structure model corresponding to the dimensional data model according to the dimension table and the fact table; determines the logical reasoning path of the dimensional data model according to the graph structure model; and performs quality assessment on the dimensional data model according to the logical reasoning path. In the embodiment of the present application, the graph structure model corresponding to the dimensional data model is obtained, and the quality assessment of the dimensional data model is performed based on the graph structure model, which can realize automatic assessment of the quality of the dimensional data model and improve the quality assessment efficiency of the dimensional data model.

[0079] In one implementation, the table determination module 401 is used to: perform entity recognition in metadata to obtain a dimension table; and perform relationship recognition between entities in metadata to obtain a fact table.

[0080] In one implementation, the graph determination module 402 is used to: map the dimension table to a node in a graph structure model; map the foreign key information in the fact table to an edge in the graph structure model, to obtain a graph structure model.

[0081] In one implementation, when mapping foreign key information in a fact table to edges in a graph structure model, the graph determination module 402 is used to: obtain annotation information in a table configuration file and / or metadata; the table configuration file is used to configure the association relationship between multiple data tables; and map the foreign key information in the fact table to edges in the graph structure model based on the table configuration file and / or the annotation information.

[0082] In one implementation, the quality assessment module 403 is used to: convert the graph structure model into description information in natural language form; input the description information and quality assessment operation information into a large language model to obtain a logical reasoning path corresponding to the quality assessment operation information output by the large language model, wherein the large language model is trained using sample description information in natural language form of a sample graph structure, sample operation information, and sample logical reasoning paths corresponding to the sample operation information; and perform quality assessment on the dimensional data model based on the logical reasoning path.

[0083] In one implementation, the logical reasoning path includes SQL statements, and the quality assessment module 403 is used to: execute SQL statements based on metadata and table content data corresponding to the metadata using an SQL parsing library to obtain execution results; and perform quality assessment on the dimensional data model based on the execution results.

[0084] In one implementation, when the quality assessment module 403 performs quality assessment on the dimensional data model based on the execution result, it is used to: obtain a result table corresponding to the quality assessment operation information in the data warehouse of the dimensional data model; compare the execution result with the result table, and determine the quality assessment result of the dimensional data model based on the comparison result.

[0085] In one implementation, the device is also used to: obtain data change events and SQL execution logs of the data warehouse of the dimensional data model based on the message queue corresponding to the data change; monitor the data change results in the data warehouse of the dimensional data model based on the data change events and SQL execution logs; if the data change results do not meet the preset standards, correct the data change results.

[0086] The functions of each module in the embodiment of the present application can be referred to the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.

[0087] Figure 5 FIG. 1 is a block diagram of an electronic device used to implement an embodiment of the present application. Figure 5As shown, the electronic device includes: a memory 510 and a processor 520. The memory 510 stores a computer program that can be run on the processor 520. When the processor 520 executes the computer program, the method in the above embodiment is implemented. The number of the memory 510 and the processor 520 can be one or more.

[0088] The electronic device also includes:

[0089] The communication interface 530 is used to communicate with external devices and perform data exchange transmission.

[0090] If the memory 510, the processor 520 and the communication interface 530 are implemented independently, the memory 510, the processor 520 and the communication interface 530 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0091] Optionally, in a specific implementation, if the memory 510, the processor 520 and the communication interface 530 are integrated on a chip, the memory 510, the processor 520 and the communication interface 530 can communicate with each other through an internal interface.

[0092] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present application when the program is executed by a processor.

[0093] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the method provided in the embodiment of the present application is implemented.

[0094] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided by the embodiment of the present application.

[0095] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.

[0096] It should be understood that the above processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor that supports the Advanced RISC Machines (ARM) architecture.

[0097] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of exemplary but not limiting description, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

[0098] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0099] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0100] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0101] Any process or method described in the flow chart or otherwise described herein can be understood as a module, fragment or portion of a code representing one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0102] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.

[0103] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0104] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.

[0105] The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope recorded in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for evaluating the quality of a dimensional data model, characterized in that: include: Based on the metadata of multiple data tables, determine the dimension tables and fact tables of the dimensional data model; Determine a graph structure model corresponding to the dimensional data model according to the dimension table and the fact table; Performing a quality assessment on the dimensional data model according to the graph structure model; The step of performing quality assessment on the dimensional data model according to the graph structure model includes: Converting the graph structure model into description information in natural language form; The description information and the quality assessment operation information are input into a large language model to obtain the large language model. The outputted logical reasoning path corresponding to the quality assessment operation information, wherein the large language model is trained using sample description information in the natural language form of a sample graph structure, sample operation information, and the sample logical reasoning path corresponding to the sample operation information; According to the logical reasoning path, performing quality assessment on the dimensional data model; The logical reasoning path includes an SQL statement, and the quality assessment of the dimensional data model according to the logical reasoning path includes: Based on the metadata and the table content data corresponding to the metadata, the SQL statement is executed using the SQL parsing library to obtain an execution result; According to the execution result, the quality of the dimensional data model is evaluated.

2. The method according to claim 1, characterized in that The step of determining the dimension table and the fact table of the dimensional data model based on the metadata of the plurality of data tables includes: Perform entity recognition in the metadata to obtain the dimension table; Relationships between the entities are identified in the metadata to obtain the fact table.

3. The method according to claim 1, characterized in that The step of determining the graph structure model corresponding to the dimensional data model according to the dimension table and the fact table includes: The dimension table is mapped to a node in the graph structure model, and the foreign key information in the fact table is mapped to an edge in the graph structure model to obtain the graph structure model.

4. The method according to claim 3, characterized in that Mapping the foreign key information in the fact table to the edges in the graph structure model includes: Acquire a table configuration file and / or annotation information in the metadata; the table configuration file is used to configure the association relationship between the multiple data tables; According to the table configuration file and / or the annotation information, the foreign key information in the fact table is mapped to the edge in the graph structure model.

5. The method according to claim 1, characterized in that The step of performing quality assessment on the dimensional data model according to the execution result includes: Obtaining a result table corresponding to the quality assessment operation information in a data warehouse of the dimensional data model; The execution result is compared with the result table, and a quality assessment result of the dimensional data model is determined according to the comparison result.

6. The method according to claim 1 or 5, characterized in that: After performing quality assessment on the dimensional data model according to the graph structure model, the method further includes: Based on the message queue corresponding to the data change, obtain the data change event and SQL execution log of the data warehouse of the dimensional data model; Based on the data change event and the SQL execution log, monitoring the data change result in the data warehouse of the dimensional data model; If the data change result does not meet the preset standard, the data change result is corrected.

7. An electronic device, characterized in that: The electronic device comprises a memory, a processor and a computer program stored in the memory, and the processor implements the method according to any one of claims 1 to 6 when executing the computer program.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images