A method, device, computer device and storage medium for evaluating a data warehouse

Abstract

The application provides a method and device for evaluating a data warehouse, a computer device and a storage medium, which can be applied to the fields of cloud computing and intelligent transportation, and is used to solve the problem of low quality of the designed data warehouse. The method comprises the following steps: obtaining at least one target relationship chain composed of each data unit satisfying a preset unit association condition based on obtained data logical relationships; determining the multiplexing degree of each data unit as a parent data unit, and determining the coincidence degree between the corresponding parent data units of each two data units as child data units; and performing quality evaluation on the data warehouse based on the obtained multiplexing degrees, coincidence degrees and chain number of the at least one target relationship chain, and obtaining a corresponding quality evaluation result. Through multi-angle quality evaluation of the data warehouse, the health status of the data warehouse can be comprehensively measured, and an accurate quality evaluation result can be obtained.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, computer equipment, and storage medium for evaluating a data warehouse. Background Technology

[0002] With the continuous development of technology, more and more devices can integrate raw data from multiple heterogeneous data sources into a data warehouse through data warehouse design. After integration, the devices can reorganize and analyze the raw data according to themes to provide different data processing services.

[0003] When designing a data warehouse, the first step is typically to name data units according to a naming convention, thus creating different data units that can be used to store different types of data. After establishing these data units, a data retrieval standard is followed to define the rules for retrieval between each pair of data units. These rules allow data to be retrieved from one data unit for processing, and the processed results are then stored in another data unit. Therefore, through the retrieval of different data units, the device can perform various data processing functions based on the raw data, providing diverse data processing services.

[0004] However, data retrieval specifications are usually simple and fragmented. Although they make it easy for developers to flexibly use data retrieval specifications to design data warehouses with rich and diverse functions, the lack of a systematic evaluation standard makes it easy for devices to put low-quality data warehouses with messy call relationships and redundant data units into use, resulting in a large amount of unnecessary resource consumption in storage and computing resources.

[0005] It is evident that, in addition to traditional naming conventions and data retrieval conventions, related technologies require an evaluation method for the designed data warehouse to avoid designing a data warehouse with high resource consumption, thereby improving the quality of the designed data warehouse. Summary of the Invention

[0006] This application provides a method, apparatus, computer equipment, and storage medium for evaluating data warehouses, which addresses the problem of low quality in the designed data warehouses.

[0007] Firstly, a method for evaluating a data warehouse is provided, including:

[0008] Obtain the logical relationships between the data units contained in the data warehouse to be evaluated, wherein each logical relationship represents that, between two corresponding data units, the parent data unit is the data source of the child data unit;

[0009] Based on the obtained logical relationships of each data unit, at least one target relationship chain that satisfies the preset unit association conditions is obtained;

[0010] Based on the aforementioned data logical relationships, the reusability of each data unit as a parent data unit is determined, as well as the overlap between the parent data units corresponding to each pair of data units when they are child data units.

[0011] Based on the obtained reusability, overlap and number of chains of the at least one target relation chain, the data warehouse is quality assessed to obtain the corresponding quality assessment results.

[0012] Secondly, an apparatus for evaluating a data warehouse is provided, comprising:

[0013] Acquisition module: used to acquire the logical relationships between the data units contained in the data warehouse to be evaluated. Each logical relationship represents that, between two corresponding data units, the parent data unit is the data source of the child data unit.

[0014] Processing module: used to obtain at least one target relationship chain that satisfies the preset unit association conditions, based on the obtained logical relationships of each data unit;

[0015] The processing module is also used to: determine the reusability of each data unit as a parent data unit based on the logical relationships of the data, and determine the overlap between the parent data units corresponding to each pair of data units when they are child data units;

[0016] The processing module is further configured to: perform a quality assessment on the data warehouse based on the obtained reusability, overlap and the number of chains of the at least one target relation chain, and obtain the corresponding quality assessment results.

[0017] Optionally, the acquisition module is specifically used for:

[0018] Iterate through all the data units contained in the data warehouse to be evaluated and create a list of data units;

[0019] Based on the data retrieval rules of the data warehouse, the data source of each data unit in the data unit list is determined sequentially to obtain each parent data unit and the child data units corresponding to each parent data unit.

[0020] Based on the correspondence between each parent data unit and its corresponding child data unit, a relationship list is established to obtain the data logical relationship between each data unit.

[0021] Optionally, the data warehouse is set up with multiple ordered data warehouse levels, and each data unit is associated with a corresponding data warehouse level. Each data warehouse level represents the data purpose of the corresponding data unit.

[0022] The processing module is specifically used for:

[0023] Based on the obtained logical relationships of each data unit, each candidate relationship chain composed of the data units is obtained, wherein there is a corresponding logical relationship between two adjacent data units in each candidate relationship chain;

[0024] From the candidate relation chains, select the candidate relation chain that does not form a circular chain, contains no more than a preset number threshold number of data units, and has no cross-level data warehouse hierarchy between any two adjacent data units as the target relation chain, and obtain the at least one target relation chain.

[0025] Optionally, the processing module is specifically used for:

[0026] For each of the data units, perform the following operations respectively:

[0027] Based on the aforementioned data logical relationships, count the number of second units of the corresponding child data units when a data unit is used as a parent data unit;

[0028] The number of the second unit is used as the reuse rate of the data unit.

[0029] Optionally, the processing module is specifically used for:

[0030] For every two data units, perform the following operations:

[0031] Based on the aforementioned data logical relationships, at least one first relation chain is obtained when a data unit is only a sub-data unit, and at least one second relation chain is obtained when another data unit is only a sub-data unit.

[0032] Each first relation chain and each second relation chain is matched to see if there are any identical data units, and the number of third units with identical data units is counted.

[0033] The number of the third unit is used as the overlap between the corresponding parent data units when the first data unit and the second data unit are respectively used as sub-data units.

[0034] Optionally, the processing module is specifically used for:

[0035] The reuse evaluation value of the data warehouse is determined based on the ratio between the sum of the reuse values ​​and the total number of data units.

[0036] Based on the number of overlaps that are greater than a preset overlap threshold among the statistically obtained overlap degrees, the overlap evaluation value of the data warehouse is determined.

[0037] A reasonable evaluation value for the data warehouse is determined based on the ratio between the number of chains and the total number of units.

[0038] The quality assessment result is determined based on the weighted sum of the reuse assessment value, the overlap assessment value, and the reasonable assessment value.

[0039] Optionally, the processing module is specifically used for:

[0040] The number of overlaps with overlaps greater than a preset overlap threshold is counted among the various overlap degrees.

[0041] Count the number of combinations of any two data units in each of the aforementioned data units;

[0042] The difference between the specified value and the ratio of the number of overlaps to the number of combinations is used as the overlap evaluation value.

[0043] Optionally, the processing module is specifically used for:

[0044] The weighted sum of the reuse evaluation value, the overlap evaluation value, and the reasonable evaluation value is used as the quality score of the data warehouse.

[0045] Based on the preset mapping relationship between each score range and each quality level, the target quality level corresponding to the target score range containing the quality score is determined. Each quality level represents the proportion of the amount of data units and data logical relationships to be optimized in the data warehouse to the total amount of each data unit and each data logical relationship.

[0046] The target quality level is used as the quality assessment result.

[0047] Thirdly, a computer program product is provided, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0048] Fourthly, a computer device is provided, comprising:

[0049] Memory, used to store program instructions;

[0050] A processor is configured to invoke program instructions stored in the memory and execute the method described in the first aspect according to the obtained program instructions.

[0051] Fifthly, a computer-readable storage medium is provided, the computer-readable storage medium storing computer-executable instructions for causing a computer to perform the method as described in the first aspect.

[0052] In this embodiment of the application, by obtaining the data logical relationship between each data unit contained in the data warehouse to be evaluated, the lineage relationship between each data unit can be obtained, and the parent data unit is the data source of the child data unit, so that a directed edge can be formed between each pair of parent data units and child data units.

[0053] Based on the logical relationships between the data, each directed edge can form multiple directed paths, which are multiple relational chains composed of each data unit. By measuring the number of target relational chains that satisfy the preset unit association conditions among the multiple relational chains, we can measure whether the designed relational chains are logically clear, and measure the health of the data warehouse from the perspective of relational chains.

[0054] Furthermore, the out-degree of each directed edge, i.e. the reusability of each data unit as a parent data unit, can determine whether there are many child data units that need to depend on the parent data unit when each data unit is the parent data unit. This can be used to measure whether the designed parent data unit has any value, and from the perspective of downstream dependencies, the health of the data warehouse can be measured.

[0055] Furthermore, when each pair of data units is used as a child data unit, the degree of overlap between their respective parent data units can determine whether there is redundant design between parent data units with high overlap, thereby measuring whether the designed parent data units are redundant, and measuring the health of the data warehouse from the perspective of upstream dependency.

[0056] By conducting multi-faceted quality assessments of data warehouses, the health status of the data warehouse can be comprehensively measured, accurate quality assessment results can be obtained, and low-quality data warehouses with chaotic call relationships and redundant data units can be avoided from being put into use. Attached Figure Description

[0057] Figure 1 This is one application scenario of the method for evaluating a data warehouse provided in the embodiments of this application;

[0058] Figure 2 A flowchart illustrating a method for evaluating a data warehouse provided in an embodiment of this application;

[0059] Figure 3A A schematic diagram of the principle of the method for evaluating a data warehouse provided in the embodiments of this application. Figure 1 ;

[0060] Figure 3BA schematic diagram of the principle of the method for evaluating a data warehouse provided in the embodiments of this application. Figure 2 ;

[0061] Figure 4A A schematic diagram three illustrating the principle of a method for evaluating a data warehouse provided in an embodiment of this application;

[0062] Figure 4B A schematic diagram four illustrating the principle of a method for evaluating a data warehouse provided in an embodiment of this application;

[0063] Figure 5 A schematic diagram of the principle of the method for evaluating a data warehouse provided in the embodiments of this application. Figure 5 ;

[0064] Figure 6 A schematic diagram of the principle of the method for evaluating a data warehouse provided in the embodiments of this application. Figure 6 ;

[0065] Figure 7A A schematic diagram seven illustrating the principle of a method for evaluating a data warehouse provided in an embodiment of this application;

[0066] Figure 7B A schematic diagram of the principle of the method for evaluating a data warehouse provided in the embodiments of this application. Figure 8 ;

[0067] Figure 7C A schematic diagram of the principle of the method for evaluating a data warehouse provided in the embodiments of this application. Figure 9 ;

[0068] Figure 8 A schematic diagram of the structure of an apparatus for evaluating a data warehouse provided in an embodiment of this application. Figure 1 ;

[0069] Figure 9 A schematic diagram of the structure of an apparatus for evaluating a data warehouse provided in an embodiment of this application. Figure 2 . Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0071] The following explanations of some terms used in the embodiments of this application are provided to facilitate understanding by those skilled in the art.

[0072] (1) Data Warehouse:

[0073] A data warehouse is a strategic collection of data that supports decision-making processes at all levels within an enterprise. It is a single data store created for analytical reporting and decision support purposes. For enterprises that require business intelligence, it provides guidance for business process improvement, monitoring of time, cost, quality, and control.

[0074] (2) Directed graph, nodes, and directed edges:

[0075] A graph is used to describe the structure of a set of objects, where certain pairs of objects are "related" in some sense. These objects are mathematical abstractions called nodes, and each related pair of nodes is connected by an edge (also called a link or line). A directed graph is a graph with a direction, consisting of a set of vertices and a number of directed edges.

[0076] (3) Data warehouse layering specifications:

[0077] Data warehouse layering refers to the hierarchical division of a data warehouse. Developers can divide the data warehouse into different layers based on experience or business needs to facilitate clear processes for calling or querying data. Standardization for data warehouse layering includes naming conventions for table fields, specifying the design of table names, task names, and field contents for each layer. These conventions are largely business-related, easy to understand, easy to implement, intuitive, and easy to monitor. Data warehouse layering also includes calling conventions, defining the calling rules between different layers. These conventions are less directly related to business needs and are the core of data warehouse layering. Once the layering is clearly defined, changes in calling conventions are minimal, and differences between different business processes are minimal.

[0078] The embodiments of this application relate to cloud computing technology design, mainly including cloud storage and cloud security, and can be applied to fields such as artificial intelligence, blockchain, smart transportation, assisted driving, or mapping.

[0079] Cloud computing refers to the delivery and usage model of IT infrastructure, meaning obtaining necessary resources in an on-demand and easily scalable manner through a network. In a broader sense, cloud computing refers to the delivery and usage model of services, meaning obtaining necessary services in an on-demand and easily scalable manner through a network. These services can be IT and software related, internet-related, or other services. Cloud computing is a product of the development and integration of traditional computer and network technologies such as grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, and load balancing.

[0080] With the development of the internet, real-time data streams, and the diversification of connected devices, as well as the demands for search services, social networks, mobile commerce, and open collaboration, cloud computing has rapidly developed. Unlike previous parallel distributed computing, cloud computing will fundamentally revolutionize the entire internet model and enterprise management model.

[0081] Cloud storage is a new concept that has been extended and developed from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that uses cluster applications, grid technology and distributed storage file systems to bring together a large number of storage devices of various types in the network (storage devices are also called storage nodes) to work together through application software or application interfaces to provide data storage and business access functions to the outside world.

[0082] Currently, the storage method of storage systems is as follows: Logical volumes are created. During the creation of a logical volume, physical storage space is allocated to each logical volume. This physical storage space may consist of a single storage device or the disks of several storage devices. Clients store data on a logical volume, which means storing the data on the file system. The file system divides the data into many parts, each part being an object. Each object contains not only the data but also additional information such as a data identifier (ID entity, ID). The file system writes each object to the physical storage space of that logical volume and records the storage location information of each object. Therefore, when a client requests access to data, the file system can allow the client to access the data based on the storage location information of each object.

[0083] The process by which a storage system allocates physical storage space to a logical volume is as follows: the physical storage space is pre-divided into strips according to the capacity estimate of the objects stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the grouping of Redundant Array of Independent Disks (RAID). A logical volume can be understood as a strip, thus allocating physical storage space to the logical volume.

[0084] Cloud security refers to the collective term for security software, hardware, users, organizations, and security cloud platforms based on cloud computing business models. Cloud security integrates emerging technologies and concepts such as parallel processing, grid computing, and the identification of unknown virus behavior. Through a large network of clients, it monitors abnormal software behavior on the network, obtains the latest information on Trojans and malware on the internet, sends it to the server for automatic analysis and processing, and then distributes solutions for viruses and Trojans to each client.

[0085] The main research directions in cloud security include: 1. Cloud computing security, which mainly studies how to ensure the security of the cloud itself and various applications on the cloud, including cloud computer system security, secure storage and isolation of user data, user access authentication, information transmission security, network attack protection, and compliance auditing; 2. Cloudification of security infrastructure, which mainly studies how to use cloud computing to build and integrate security infrastructure resources and optimize security protection mechanisms, including building a large-scale security event and information collection and processing platform through cloud computing technology to achieve the collection and correlation analysis of massive amounts of information and improve the ability to control network-wide security events and risks; 3. Cloud security services, which mainly studies various security services provided to users based on cloud computing platforms, such as antivirus services.

[0086] It should be noted that in the embodiments of this application, data related to data units contained in the data warehouse are obtained. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required each time data is obtained, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0087] The following is a brief introduction to the application areas of the data warehouse evaluation method provided in the embodiments of this application.

[0088] With the continuous development of technology, more and more devices can integrate raw data from multiple heterogeneous data sources into a data warehouse through data warehouse design. After integration, the devices can reorganize and analyze the raw data according to themes to provide different data processing services.

[0089] When designing a data warehouse, the first step is typically to name data units according to a naming convention, thus creating different data units that can be used to store different types of data. After establishing these data units, a data retrieval standard is followed to define the rules for retrieval between each pair of data units. These rules allow data to be retrieved from one data unit for processing, and the processed results are then stored in another data unit. Therefore, through the retrieval of different data units, the device can perform various data processing functions based on the raw data, providing diverse data processing services.

[0090] The naming conventions for data units in a data warehouse are relatively fixed and easy to maintain, while the data retrieval conventions are usually simple and fragmented. Although this makes it easier for developers to flexibly use the data retrieval conventions to design data warehouses with rich and diverse functions, the lack of a systematic evaluation standard makes it easy for equipment to put low-quality data warehouses with chaotic retrieval relationships and redundant data units into use, resulting in a large amount of unnecessary resource consumption in terms of storage and computing resources.

[0091] It is evident that, in addition to traditional naming conventions and data retrieval conventions, related technologies require an evaluation method for the designed data warehouse to avoid designing a data warehouse with high resource consumption, thereby improving the quality of the designed data warehouse.

[0092] To address the issue of low quality in designed data warehouses, this application proposes a method for evaluating data warehouses. After obtaining the logical relationships between the data units within the data warehouse to be evaluated, this method can, based on these logical relationships, obtain at least one target relationship chain that satisfies preset unit association conditions. Each logical relationship represents that, between two corresponding data units, the parent data unit is the data source for the child data unit. Simultaneously, based on these logical relationships, the method determines the reusability of each data unit as a parent data unit and the overlap between the parent data units of each pair of data units when they are child data units. Based on the obtained reusability, overlap, and the number of at least one target relationship chain, the data warehouse is evaluated to obtain the corresponding quality evaluation results.

[0093] In this embodiment of the application, by obtaining the data logical relationship between each data unit contained in the data warehouse to be evaluated, the lineage relationship between each data unit can be obtained, and the parent data unit is the data source of the child data unit, so that a directed edge can be formed between each pair of parent data units and child data units.

[0094] Based on the logical relationships between the data, each directed edge can form multiple directed paths, which are multiple relational chains composed of each data unit. By measuring the number of target relational chains that satisfy the preset unit association conditions among the multiple relational chains, we can measure whether the relational chains contained in the designed data warehouse are logically clear, and measure the health status of the data warehouse from the perspective of relational chains.

[0095] Furthermore, the out-degree of each directed edge, i.e. the reusability of each data unit as a parent data unit, can determine whether there are many child data units that need to depend on the parent data unit when each data unit is the parent data unit. This can be used to measure whether the parent data units contained in the designed data warehouse have any use value, and from the perspective of downstream dependencies, the health status of the data warehouse can be measured.

[0096] Furthermore, when each pair of data units is used as a sub-data unit, the degree of overlap between their respective parent data units can determine whether there is redundant design between parent data units with high overlap. This allows us to measure whether the parent data units contained in the designed data warehouse are redundant, and to measure the health of the data warehouse from the perspective of upstream dependencies.

[0097] By conducting multi-faceted quality assessments of data warehouses, the health status of the data warehouse can be comprehensively measured, accurate quality assessment results can be obtained, and low-quality data warehouses with chaotic call relationships and redundant data units can be avoided from being put into use.

[0098] The following describes the application scenarios of the data warehouse evaluation method provided in this application.

[0099] Please refer to Figure 1 This diagram illustrates an application scenario of the data warehouse evaluation method provided in this application. The application scenario includes a client 101 and a server 102. The client 101 and the server 102 can communicate with each other. The communication method can be wired, such as through a network cable or serial cable; or wireless, such as through Bluetooth or Wi-Fi. No specific limitation is imposed.

[0100] Client 101 generally refers to any device that can specify a data warehouse to be evaluated to server 102, or provide the logical relationships between the data units contained in the data warehouse to be evaluated. Examples include terminal devices, third-party applications accessible by the terminal devices, or web pages accessible by the terminal devices. Terminal devices include, but are not limited to, mobile phones, computers, intelligent transportation equipment, and smart appliances. Server 102 generally refers to any device capable of evaluating a data warehouse, such as terminal devices or servers. Servers include, but are not limited to, cloud servers, local servers, or associated third-party servers. Both client 101 and server 102 can utilize cloud computing to reduce the consumption of local computing resources; similarly, they can also utilize cloud storage to reduce the consumption of local storage resources.

[0101] As one embodiment, the client 101 and the server 102 can be the same device. For example, this embodiment can be implemented using only the client 101 or only the server 102, etc., without any specific limitation. In this embodiment, the client 101 and the server 102 are described as different devices.

[0102] The following is based on Figure 1 Taking server 102 as an example, the method for evaluating a data warehouse provided in this application embodiment will be described in detail. Please refer to... Figure 2 This is a flowchart illustrating a method for evaluating a data warehouse provided in an embodiment of this application.

[0103] S201, Obtain the logical relationships between the data units contained in the data warehouse to be evaluated.

[0104] Each data logical relationship represents the following: between two corresponding data units, the parent data unit is the data source for the child data unit. The server can receive evaluation instructions sent by the client. These instructions specify the data warehouse to be evaluated. Based on these instructions, the server accesses the data warehouse and retrieves the data logical relationships between the data units contained within it.

[0105] Each data warehouse can be set with an evaluation flag indicating whether it is to be evaluated. The server will use each data warehouse with an evaluation flag indicating that it has not been evaluated as the data warehouse to be evaluated, access the data warehouse to be evaluated, and obtain the data logical relationships between the data units contained in the data warehouse. There are no restrictions on the specific method of determining the data warehouse to be evaluated.

[0106] As one implementation, the server can obtain each data unit by sequentially accessing each data unit contained in the data warehouse to be evaluated. If the data warehouse has a lineage table, which represents the parent data unit corresponding to the data source of each data unit (see Table 1), the lineage identifier is used to uniquely identify each lineage relationship, the current level represents the data warehouse level associated with the current data unit, and the parent data unit level represents the data warehouse level associated with the parent data unit. The data warehouse levels will be described in detail later. The server can then obtain each data unit by traversing the lineage table, etc., without any specific limitations.

[0107] Table 1

[0108] Blood relation identification Current level Data Unit Parent data unit Parent data unit hierarchy 1 Level A Data Unit A Data Unit C Level C 2 Level B Data Unit B Data Unit D Level D

[0109] The server can determine the data source of each data unit based on the data retrieval strategy of the data warehouse, thereby determining the correspondence between each parent data unit and its corresponding child data unit, and obtaining the logical relationships between the data. If the data warehouse has a lineage table, the server can obtain the correspondence between each parent data unit and its corresponding child data unit from the lineage table, and obtain the logical relationships between the data, etc., without any specific restrictions.

[0110] The logical relationships between data units can be in the form of strings, lists, or graphs, etc., without any specific restrictions.

[0111] Taking a graph as an example, after the server obtains each data unit, it can treat each data unit as a node. After obtaining the logical relationships between the data, a directed edge can be established based on each logical relationship. The directed edge points from the parent node corresponding to the parent data unit to the child node corresponding to the child data unit, thus obtaining a directed graph that represents the logical relationships between the data units contained in the data warehouse.

[0112] Please refer to Figure 3A Each data unit includes data unit A, data unit B, data unit C, and data unit D. Data unit A is the data source for data unit B, and data unit B is the data source for data units C and D. Therefore, the directed graph contains four nodes: node A corresponds to data unit A, node B corresponds to data unit B, node C corresponds to data unit C, and node D corresponds to data unit D. There is a directed edge between node A and node B, pointing from node A to node B; a directed edge between node B and node C, pointing from node B to node C; and a directed edge between node B and node D, pointing from node B to node D.

[0113] Taking a list format as an example, after obtaining each data unit, the server can create a list of data units, as shown in Table 2. The data unit identifier uniquely identifies each data unit, the data unit name is the name of each data unit, and the data unit attributes describe the data unit.

[0114] Table 2

[0115] Data Unit Identifier Data Unit Name Data unit attributes 1 Data Unit A Data obtained through the transaction domain 2 Data Unit B Data obtained through promotional channels 3 Data Unit C Data obtained through promotional domains 4 Data Unit D Data obtained through merchant domain

[0116] After obtaining each parent data unit and its corresponding child data unit, a relationship list can be established based on the correspondence between each parent data unit and its corresponding child data unit to obtain the data logical relationship between each data unit. Please refer to Table 3.

[0117] Among them, the relation identifier can be used to uniquely represent each data logical relationship, the parent data unit is the data source of the child data unit, the data in the child data unit is generated by calling the data in the parent data unit for operation, and the relation attribute can be used to describe the data logical relationship.

[0118] Table 3

[0119] Relationship identifier Parent data unit Sub-data unit Relationship attributes 1 Data Unit A Data Unit B Calculated on a daily basis 2 Data Unit B Data Unit C Calculated in hours 3 Data Unit B Data Unit D Calculated on a monthly basis

[0120] S202, based on the obtained logical relationships of each data unit, obtain at least one target relationship chain that satisfies the preset unit association conditions.

[0121] After obtaining the logical relationships between the data units, the server can determine the candidate relationship chains composed of the data units based on these relationships, and select at least one target relationship chain from the candidate relationship chains that satisfies the preset unit association conditions. Each candidate relationship chain contains at least two data units, and there is a corresponding logical relationship between every two adjacent data units in each candidate relationship chain.

[0122] The server can use data units that do not have a corresponding parent data unit as the starting point of a candidate relationship chain and data units that do not have a corresponding child data unit as the ending point of a candidate relationship chain to obtain various candidate relationship chains; the server can also use data units that do not have a corresponding parent data unit as the starting point of a candidate relationship chain and each data unit as the ending point of a candidate relationship chain to obtain various candidate relationship chains, etc., without any specific restrictions.

[0123] The server can start with a data unit and perform a depth-first search to find the parent data unit when the data unit is a child data unit, and the parent data unit when the parent data unit is a child data unit, until no corresponding parent data unit exists, thus obtaining a candidate relationship chain.

[0124] For example, please refer to Figure 3B Let represent the logical relationships between the data units, which include data unit A, data unit B, data unit C, data unit D, and data unit E. When data unit A is the parent data unit and data unit B is the child data unit, there is a logical relationship between data unit A and data unit B; when data unit B is the parent data unit and data unit C is the child data unit, there is a logical relationship between data unit B and data unit C; when data unit E is the parent data unit and data unit C is the child data unit, there is a logical relationship between data unit E and data unit C; when data unit C is the parent data unit and data unit D is the child data unit, there is a logical relationship between data unit C and data unit D.

[0125] When the server uses data unit C as the endpoint, it performs a depth-first search to find the parent data units (data units B and E) when data unit C is a child data unit. Since it's a depth-first search, it further searches for the parent data unit (data unit A) when data unit B is a child data unit. Because data unit A does not have a corresponding parent data unit, a possible relationship chain is obtained: data unit A links to data unit B, and data unit B links to data unit C.

[0126] Preset unit association conditions are used to determine whether the data logic of candidate relationship chains contains logical errors, whether the logic is clear, and whether data decoupling has been achieved. There can be multiple preset unit association conditions; different conditions can be set depending on the use case. There are no specific restrictions; three examples are described below.

[0127] Preset unit association condition one:

[0128] The alternative relationship chains did not form a circular chain.

[0129] A circular chain indicates that the candidate relationship chain has no initiating or ending point. Each data unit can act as a sub-data unit, and each sub-data unit corresponds to a parent data unit, thus forming a circular chain. A circular chain can easily cause data to enter an infinite loop during the data retrieval process, resulting in the device continuously performing data retrieval and calculation operations, causing unnecessary resource consumption. Therefore, the preset unit association condition can include the absence of a circular chain in the candidate relationship chain to determine the health status of each data logical relationship.

[0130] The server can set an access identifier for each data unit and access each data unit sequentially, starting from the beginning of each candidate relationship chain. The access identifier of an unaccessed data unit defaults to a first specified value. The server updates the access identifier of the currently accessed data unit from the first specified value to a second specified value. When the server finishes accessing a data unit, it updates the access identifier of the accessed data unit from the second specified value to a third specified value.

[0131] When the server determines that the access identifier of the next data unit to be accessed is the third specified value, it can determine that the candidate relationship chain forms a circular chain; when the server determines that there is no next data unit to be accessed, the candidate relationship chain is judged and no circular chain is formed.

[0132] Please refer to Figure 4A (1) represents a candidate relation chain. When no access is performed, the access identifier `used` of each data unit in this candidate relation chain is 0. When determining whether the candidate relation chain is a circular chain, the server can first access data unit A. At this time, the access identifier `used` of data unit A is updated from 0 to 1. Please refer to [link / reference]. Figure 4A (2).

[0133] After accessing data unit A, update the access flag "used" of data unit A from 1 to -1. Please refer to [link / reference]. Figure 4B (1). After accessing data unit A, continue accessing data units B, C, D, and E. After accessing data unit E, the access flag `used` for data units A, B, C, D, and E is all -1. Please refer to [reference needed]. Figure 4B (2). Based on the alternative relationship chain, continue to access data unit A. At this time, the access identifier used = -1 of data unit A. Therefore, it can be determined that the alternative data chain is a circular chain and cannot be used as the target relationship chain.

[0134] Preset unit association condition two:

[0135] The number of the first data unit in the candidate relation chain is no greater than a preset threshold.

[0136] When the number of the first data units in a candidate relationship chain exceeds a preset threshold, it indicates that data units near the end of the chain require data from numerous other data units to be obtained. This causes data in one unit to involve many other data units, increasing the complexity of the calling relationships. This not only hinders the maintenance of calling relationships but also easily leads to data calling or processing errors. Furthermore, locating the erroneous data when errors occur is complex, requiring repeated checks of numerous data units for anomalies. Additionally, error correction must consider the impact on other data units. Therefore, a preset unit association condition could include ensuring that the number of the first data units in a candidate relationship chain does not exceed a preset threshold to assess the health of each data logical relationship.

[0137] As one embodiment, for candidate relationship chains with the same ending point, since the ending point is the same, only the candidate relationship chain with the largest number of first units containing data units needs to be evaluated to see if the number of first units exceeds a preset threshold, without evaluating other candidate relationship chains. If, for the candidate relationship chain with the largest number of first units containing data units, the number of first units exceeds the preset threshold, it indicates that the data units and data logic relationships related to that ending point need to be redesigned or optimized, etc. Therefore, it is no longer necessary to evaluate other candidate relationship chains corresponding to that ending point, thus improving evaluation efficiency.

[0138] For example, please refer to Figure 5 By using a depth-first search to find the parent data unit of data unit D when it is only a child data unit, three candidate relationship chains are obtained: data unit A links to data unit B, data unit B links to data unit C, and data unit C links to data unit D; data unit E links to data unit D; and data unit F links to data unit G, and data unit G links to data unit D. The three candidate relationship chains have the same endpoint, and the number of first units of the data units contained in each of the three candidate relationship chains are 4, 2, and 3 respectively. Therefore, the server can only determine whether the number of parent data units other than data unit D (3) in the candidate relationship chain of data unit A linking to data unit B, data unit B linking to data unit C, and data unit C linking to data unit D is greater than a preset threshold of 2. Since 3 is greater than 2, this candidate relationship chain cannot be used as the target relationship chain.

[0139] Preset unit association condition three:

[0140] There are no cross-level data warehouses between any two adjacent data units in the candidate relational chain.

[0141] A data warehouse can be structured with multiple ordered data warehouse levels. Each data unit is associated with a corresponding data warehouse level, and each level represents the data purpose of that data unit. This hierarchical structure simplifies complex problems by breaking down complex data computation tasks into multiple simple subtasks. Each data warehouse level handles only one simple subtask, allowing for accurate and quick location and troubleshooting of errors. Furthermore, many different computational results are based on the same intermediate computational values. With a data warehouse hierarchy, when obtaining different results, it's unnecessary to repeatedly calculate these intermediate values; instead, they only need to be calculated once and directly retrieved each time. This reduces computational complexity and avoids errors caused by inconsistencies in intermediate values ​​due to network issues or other external factors, improving data retrieval accuracy. Finally, a data warehouse hierarchy also isolates computational results from the original data, preventing accidental modification of the original data during computation.

[0142] A data warehouse can have multiple layers, such as: an Application Data Service (ADS) layer for storing personalized statistical data; a Common Data Model (CDM) layer for processing and integrating data; and an Operation Data Store (ODS) layer for storing raw, unprocessed data. The Common Data Model layer can further include: a Common Dimension layer for establishing consistent dimensions across the entire business based on dimensional modeling principles; a Common Aggregate Fact Layer for building aggregated metric facts at a common granularity, driven by the analytical subject and based on the metric requirements of upper-layer applications and products; and a Fine-Grained Fact Layer for building the finest-grained facts based on the business processes, tailored to the characteristics of each specific business process.

[0143] Depending on the level of the associated data warehouse, there are different specifications for whether each data unit can be used as a data source. Please refer to Table 4 for a specification on whether data can be called across levels of the data warehouse as a data source.

[0144] Table 4

[0145]

[0146] For example, a reference layer cannot be both a caller and a callee at the same time; that is, a reference layer cannot be the data source of a reference layer. If there is a data logical relationship between two data units, and they are respectively the parent data unit and the child data unit, and the data warehouse level associated with both data units is the reference layer, then the data warehouse level of these two data units is cross-level.

[0147] For example, the common summary granularity fact layer can act as the caller, and the reference layer as the callee. That is, the reference layer can act as the data source of the common summary granularity fact layer. In this case, there is a data logical relationship between the two data units. When they act as the parent data unit and the child data unit respectively, if the data warehouse level associated with the parent data unit is the reference layer and the data warehouse level associated with the child data unit is the common summary granularity fact layer, then the data warehouse levels of these two data units are not cross-level.

[0148] As one embodiment, the preset unit association conditions may include one or more of the above three conditions, or conditions other than the above three conditions, etc., and there is no specific limitation. In this embodiment of the application, the preset unit association conditions including the above three conditions are used as an example for description. After obtaining each candidate relationship chain, the server can select from each candidate relationship chain a candidate relationship chain that does not form a circular chain, and the number of first units of the data units contained therein is not greater than a preset number threshold, and the data warehouse levels corresponding to each two adjacent data units are not cross-level, as the target relationship chain, thereby obtaining at least one target relationship chain.

[0149] S203, based on the logical relationships of each data unit, determine the reusability of each data unit as a parent data unit, and determine the overlap between the parent data units of each pair of data units when they are child data units.

[0150] The server can determine the reusability of each data unit as a parent data unit based on the logical relationships between the data. The reusability characterizes the number of child data units corresponding to each data unit when it is a parent data unit. The more child data units there are, the higher the reusability of the data unit when it is a parent data unit, indicating that the design of the data unit is more reasonable and that the data in the data unit can be used by multiple other data units.

[0151] Taking a single data unit as an example, this section describes the calculation process for the reusability of a data unit. The calculation process for the reusability of each data unit is similar and will not be repeated here. The server can, based on the logical relationships between data units, count the number of second-generation child data units corresponding to a given data unit when it is used as the parent data unit. This counted number of second-generation child data units is then used as the reusability of one data unit.

[0152] The server can determine the reusability of each data unit based on logical data relationships in the form of a graph or a table. For example, in a graph-based relationship, the server can calculate the out-degree of each node and use that out-degree as the reusability of the corresponding data unit. Similarly, in a table-based relationship, the server can count the number of relationships in the relationship list that have a specific data unit as their parent, and use that count as the reusability of that data unit. The specific method is not limited.

[0153] The server can determine the degree of overlap between the parent data units of each pair of data units when they are used as child data units based on the logical relationships between the data. The degree of overlap indicates whether each pair of data units uses the same parent data unit as its data source. The higher the degree of overlap, the more common the parent data units are used as the data source for the two data units. In this case, there may be a design redundancy problem between the two data units. For example, the two data units record the same or similar data, but occupy two storage resources.

[0154] Taking a single data unit as an example, this section describes the calculation process for the overlap between two data units. The calculation process for the overlap between other data units is similar and will not be repeated here.

[0155] The server can, based on the logical relationships between data units, obtain at least one first relation chain when one data unit is solely a child data unit, and at least one second relation chain when another data unit is solely a child data unit. It then matches each first relation chain and each second relation chain to determine if there are any identical data units, and counts the number of third units corresponding to these identical data units. The counted number of third units is then used to determine the overlap between the corresponding parent data units when one data unit and another data unit are respectively considered as child data units.

[0156] As one embodiment, the server can obtain at least one relation chain when each data unit is only a sub-data unit, and establish a parent data unit set based on the data units contained in the at least one relation chain corresponding to each data unit. Please refer to Table 5. The data unit name represents the name of each data unit, and the parent data unit set represents the data units contained in the at least one relation chain corresponding to each data unit.

[0157] Table 5

[0158] Data unit name Parent data unit set Data Unit A {Data Unit B, Data Unit C} Data Unit D {Data Unit B, Data Unit E, Data Unit F}

[0159] For example, if data unit A is only a child data unit and there is at least one relation chain, and the other data units included are data units B and C, then the set of parent data units corresponding to data unit A is the set containing data units B and C. Similarly, if data unit D is only a child data unit and there is at least one relation chain, and the other data units included are data units B, E, and F, then the set of parent data units corresponding to data unit D is the set containing data units B, E, and F.

[0160] When data unit A and data unit D are respectively used as child data units, the corresponding parent data unit with the same data unit is data unit B, so the overlap is 1.

[0161] S204. Based on the obtained reusability, overlap and the number of chains of at least one target relation chain, the data warehouse is quality assessed and the corresponding quality assessment results are obtained.

[0162] After obtaining at least one target relation chain, the number of chains in the at least one target relation chain can be determined. Based on the obtained reusability, overlap and the number of chains in the at least one target relation chain, the data warehouse can be quality evaluated from multiple perspectives to obtain the corresponding quality evaluation results.

[0163] The server can use the number of chains for each degree of reusability, each degree of overlap, and at least one target relationship chain as the quality assessment result of the data warehouse; or it can convert and merge the number of chains for each degree of reusability, each degree of overlap, and at least one target relationship chain separately to obtain the quality assessment result of the data warehouse.

[0164] As one embodiment, after obtaining each reuse level, the server can calculate the ratio between the sum of each reuse level and the total number of data units, and normalize the obtained ratio, for example, by using the sigmoid function. The normalized result is then used as the reuse evaluation value S of the data warehouse. ReusedRate Please refer to formula (1).

[0165]

[0166] Among them, od i Let represent the reuse degree when the i-th data unit is used as the parent data unit, and N represent the total number of units in each data unit. This indicates that the variable x is normalized using the sigmoid function.

[0167] By normalizing the ratio between the sum of each reuse degree and the total number of data units, the value is mapped to the range of [0, 1]. This avoids large differences in the sum of reuse degrees or the total number of units due to different data warehouse sizes, ensuring that the obtained reuse evaluation values ​​are all calculated using a unified dimension.

[0168] As one implementation, after obtaining the overlap rates, the server can count the number of overlaps with overlap rates greater than a preset overlap threshold, and determine the overlap assessment value of the data warehouse based on this number of overlaps. If the overlap rate between the parent data units corresponding to two data units is greater than the preset overlap threshold, it indicates that the parent data units corresponding to the two data units are quite similar. For example, in the set of parent data units corresponding to two data units, there are more than two parent data units that are the same. These two or more identical parent data units may indicate design redundancy issues.

[0169] The server can count the number of combinations of every two data units in each data unit, and use the difference between a specified value and the ratio of the number of overlaps to the number of combinations as the overlap evaluation value S. Completeness Please refer to formula (2).

[0170]

[0171] Where the specified value is 1, parent(t) i ) indicates that the i-th data unit is used as a sub-data unit t i The set of parent data units corresponding to the time, parent(t) j ) indicates that the j-th data unit is used as a sub-data unit t. j The set of parent data units corresponding to the time, where 'a' is a preset overlap threshold, and 'count(·)' represents counting the corresponding number. This indicates the number of combinations of any two data units within each data unit.

[0172] As one embodiment, after determining the data warehouse reuse evaluation value based on the ratio between the sum of each reuse degree and the total number of data units, and determining the data warehouse overlap evaluation value based on the number of overlaps among the statistically calculated overlap degrees that exceed a preset overlap degree threshold, the server can further determine the data warehouse reasonable evaluation value S based on the ratio between the number of chains and the total number of units. Reasonableness Please refer to formula (3).

[0173]

[0174] Among them, FindCircle(t) i) = 1 indicates that the i-th data unit t i To form a circular chain for an alternative relation chain at the end, FindCircle(t) i ) = 0 indicates that the i-th data unit t i The alternative relationship chain at the end did not form a circular chain; MaxParentLevel(t i ) represents the i-th data unit t i The first unit of a candidate relation chain at the end is the number of data units contained in the first unit, and b is a preset number threshold; ParentLevelNodeInList(t i ) = 1 indicates that the i-th data unit t i In an alternative relation chain at the end, each pair of adjacent data units corresponds to a cross-level data warehouse hierarchy; ParentLevelNodeInList(t i ) = 0 indicates that the i-th data unit t i In an alternative relation chain with the end as the endpoint, there is no cross-level between the data warehouse levels corresponding to each two adjacent data units; count(·) represents counting the corresponding quantity, and N represents the total number of units in each data unit.

[0175] The server can determine the quality assessment result based on a weighted sum of reused assessment values, overlapping assessment values, and reasonable assessment values.

[0176] As one example, the server can use the weighted sum of the reused evaluation value, the overlapping evaluation value and the reasonable evaluation value as the quality score S of the data warehouse, please refer to formula (4).

[0177] S = w1 * S ReusedRate +w2*S Completeness +w3*S Reasonableness (4)

[0178] Where w1 represents the weighting coefficient of the reused evaluation value, w2 represents the weighting coefficient of the overlapping evaluation value, and w3 represents the weighting coefficient of the reasonable evaluation value.

[0179] The server can determine the target quality level corresponding to the target score range containing the quality score based on the preset mapping relationship between various score ranges and quality levels. Each quality level represents the proportion of data units and logical relationships to be optimized in the data warehouse relative to the total number of data units and logical relationships. The target quality level is used as the quality assessment result. Therefore, based on the target quality level, if the proportion of data units and logical relationships to be optimized in the data warehouse is high, the data warehouse needs to be redesigned; if the proportion is moderate, optimization can be performed on some data units or logical relationships; and if the proportion is low, the data warehouse can be put into use.

[0180] For example, score ranges can include 100%, [90%, 100%), [80%, 90%), and [0%, 80%); the corresponding quality levels are Excellent, Very Good, Good, and Unsatisfactory, respectively. When the target quality level is Excellent, the data warehouse can be put into use; when the target quality level is Very Good or Good, the data warehouse can be optimized in a targeted manner; when the target quality level is Unsatisfactory, the data warehouse needs to be redesigned, etc.

[0181] The following example, using a retail business scenario, illustrates the method for evaluating a data warehouse provided in this application.

[0182] Please refer to Figure 6 This is a data warehouse used in retail business scenarios, taking data tables as an example. The order information summary table represents the summary of information from each order in the retail business; the data warehouse layer associated with the order information summary table is the common data layer. The product activity information table represents the sales information of each product in the retail business; the data warehouse layer associated with the product activity information table is the common data layer. The order information summary tagging table represents the tagging summary of order information in the retail business; the data warehouse layer associated with the order information summary tagging table is the common data layer. The order product table represents the products in each order in the retail business; the data warehouse layer associated with the order product table is the common data layer. The order activity information table represents the promotional activity information of each order in the retail business; the data warehouse layer associated with the order activity information table is the common data layer.

[0183] The Store Terminal Table represents information about each terminal in a store, and its data warehouse layer is the data application layer. The Store Transaction Table represents information about each transaction in a store, and its data warehouse layer is the data application layer. The Store Activity Table represents information about each promotional activity in a store, and its data warehouse layer is the data application layer. The Store Activity Transaction Table represents transaction information corresponding to each promotional activity in a store, and its data warehouse layer is the data application layer. The Store Activity Channel Table represents the organizational channels for promotional activities, and its data warehouse layer is the data application layer.

[0184] The data warehouse contains a lineage table of each data table; please refer to Table 6.

[0185] Table 6

[0186]

[0187]

[0188] The server determines the data units contained in the data warehouse by traversing the lineage table of the data warehouse and establishes a list of data units, as shown in Table 7.

[0189] Table 7

[0190] Data Unit Identifier Data unit name Data unit attributes 1 Order Information Summary Table Transaction Domain 2 Product Activity Information Form Promotional domain 3 Order Information Summary and Tagging Table Transaction Domain 4 Order Product List Transaction Domain 5 Order Activity Information Form Promotional domain 6 Store Terminal Table Merchant side 7 Store Transaction Form Merchant side 8 Store Activity Schedule Merchant side 9 Store Activity Transaction Form Merchant side 10 Store Activity Channel List Merchant side

[0191] Among them, the data unit attributes are used to characterize the scenario in which the corresponding data unit is applied. For example, the order information summary table is a summary of the order information in the transaction domain; another example is the product activity information table, which is the promotional activity information of each product in the promotion domain; and yet another example is the store terminal table, which is the product or order information presented on the merchant's end, etc.

[0192] The server uses the lineage table to obtain the data retrieval rules of the data warehouse, and based on these rules, sequentially determines the data source for each data unit in the data unit list, obtaining each parent data unit and its corresponding child data units. A relationship list is then established between each parent data unit and its corresponding child data unit; please refer to Table 8 to obtain the logical data relationships between the data units.

[0193] Table 8

[0194] Relationship identifier Parent data unit Sub-data unit Relationship attributes 1 Order Information Summary Table Product Activity Information Form Tianyi 2 Order Information Summary Table Order Information Summary and Tagging Table Tianyi 3 Order Information Summary Table Order Product List Tianyi 4 Order Information Summary Table Order Activity Information Form Tianyi 5 Product Activity Information Form Order Activity Information Form Tianyi 6 Order Information Summary and Tagging Table Store Terminal Table Tianyi 7 Order Information Summary and Tagging Table Store Transaction Form Tianyi 8 Order Product List Store Transaction Form Tianyi 9 Order Information Summary and Tagging Table Store Activity Schedule Tianyi 10 Order Activity Information Form Store Activity Schedule Tianyi 11 Order Information Summary and Tagging Table Store Activity Transaction Form Tianyi 12 Order Activity Information Form Store Activity Transaction Form Tianyi 13 Order Information Summary and Tagging Table Store Activity Channel List Tianyi 14 Order Activity Information Form Store Activity Channel List Tianyi

[0195] Among them, the relational attribute is used to describe the logical relationship between the data, and "day-dependent" indicates that the data unit performs data calculations on a daily basis.

[0196] After obtaining the logical relationships of each data point, the server can calculate the reuse degree, overlap degree, and number of chains in the target relationship chain, and calculate the reuse evaluation value, overlap evaluation value, and reasonable evaluation value based on the reuse degree, overlap degree, and number of chains, which will be introduced below.

[0197] Reusability and reuse evaluation value:

[0198] Since the data table corresponding to the data application layer is the last level of data unit and there is no corresponding sub-data unit, the calculation of reuse degree and reuse evaluation value can be performed only on the data table corresponding to the data common layer.

[0199] The data tables corresponding to the common data layer include the order information summary table, the product activity information table, the order information summary tagging table, the order product table, and the order activity information table, a total of 5 tables, so N=5.

[0200] Based on the relationship list, each parent data unit is selected sequentially, and the number of child data units corresponding to each parent data unit is counted. It can be determined that when the order information summary table is used as the parent data unit, the corresponding child data units include the product activity information table, the order information summary tagging table, the order product table, and the order activity information table. Therefore, the number of second data units is 4, that is, the reuse rate is 4. Similarly, the number of second data units corresponding to the product activity information table is 1, that is, the reuse rate is 1; the number of second data units corresponding to the order information summary tagging table is 5, that is, the reuse rate is 5; the number of second data units corresponding to the order product table is 1, that is, the reuse rate is 1; and the number of second data units corresponding to the order activity information table is 3, that is, the reuse rate is 3.

[0201] According to formula (1), the data warehouse reuse evaluation value is 0.885.

[0202] Overlap ratio and overlap assessment value:

[0203] The total number of data units, N, is 10. Taking a preset overlap threshold of 2 as an example, if two data units are used as sub-data units, and there are two or more corresponding parent data units in the set of parent data units that are identical, then the statistical value is incremented by 1. When every pair of data units has been matched, the overlap assessment value of the data warehouse is determined by the ratio between the statistical value and the number of combinations of every pair of data units.

[0204] The server can sequentially select each data unit from the data unit list and determine the corresponding parent data unit when that data unit is used as a child data unit from the relationship list, thus creating a parent data unit set list (see Table 9). Then, based on the parent data unit set list, the server sequentially matches the parent data unit sets of every two data units.

[0205] Table 9

[0206]

[0207]

[0208] By matching the parent data unit sets corresponding to each pair of data units in Table 9, the intersection of the parent data unit sets of the store activity table, store activity transaction table, and store activity channel table contains two data units, with an overlap of 6. The number of combinations of 10 tables is 90. Therefore, according to formula (2), the overlap evaluation value of the data warehouse is 0.933.

[0209] Number of chains and reasonable valuation:

[0210] The total number of data units N is 10. Taking a preset number threshold of 5 as an example, the server can access each data unit sequentially according to the relationship table to determine whether each data unit forms a circular chain; and use a depth-first loop to find the parent data unit of each data unit to determine whether the number of the first unit of the parent data unit is greater than 5; and according to Table 4, determine whether there is a cross-level between the data warehouse levels corresponding to every two adjacent data units with data logical relationship.

[0211] By statistically analyzing the data, if no circular chain is formed, the number of the first unit is not greater than the preset number threshold, and the number of target relationship chains between the data warehouse levels corresponding to each two adjacent data units is 10, then according to formula (3), a reasonable evaluation value of 1 can be obtained.

[0212] Taking a weighted average of 1 / 3 for reuse evaluation values, 1 / 3 for overlap evaluation values, and 1 / 3 for reasonable evaluation values ​​as an example, the data warehouse quality score is calculated as 0.939 by weighting and summing the reuse evaluation values, overlap evaluation values, and reasonable evaluation values. This quality score falls within the range of [90%, 100%), corresponding to a very good target quality level. Targeted optimization of the data warehouse is possible to improve its design.

[0213] The embodiments of this application can accurately assess the quality of the data warehouse in both the design and data alarm monitoring stages. The method for assessing the data warehouse conforms to the principle of Mutually Exclusive Collectively Exhaustive (MECE), and provides a structured evaluation of the hierarchical health of the data warehouse.

[0214] From a reuse perspective, please refer to Figure 7A(1) Data unit A has four parent data units, data unit B has no corresponding parent data unit, data unit C has two parent data units, data unit D has no corresponding parent data unit, data unit E has two parent data units, data unit F has two parent data units, data unit A and data unit B, and data unit G has four parent data units, data unit B, data unit C, data unit D and data unit E.

[0215] Data Units B and D contain many inactive, zombie, or partially siloed data tables, indicating that resources are not being used effectively. In this case, the reuse evaluation value is 2 when it is not normalized.

[0216] Please refer to Figure 7A (2) Data unit A has four parent data units, data unit B has two parent data units, data unit C has two parent data units, data unit D has one parent data unit (data unit A), and data unit E has two parent data units (data units B and C). This reduces inactive data tables, zombie tables, or partially siloed data tables. The reuse evaluation value is 2.2 when not normalized.

[0217] From the perspective of overlap, please refer to Figure 7B (1) The intersection of the parent data set corresponding to data unit A and data unit B includes three data units, the intersection of the parent data set corresponding to data unit A and data unit C includes one data unit, and the intersection of the parent data set corresponding to data unit B and data unit C includes one data unit.

[0218] The data units in the data warehouse do not act as isolation layers. If the parent data units corresponding to data units A, B, and C need to be changed, all three data units need to be modified. The content of all three data units needs to be processed. At this time, the overlap evaluation value is 0.67.

[0219] Please refer to Figure 7B (2) The intersection of the parent data set corresponding to data unit A and data unit B includes one data unit. The intersection of the parent data set corresponding to data unit A and data unit C includes one data unit. The intersection of the parent data set corresponding to data unit B and data unit C includes one data unit. The intersection of data unit D with the parent data set corresponding to data unit A, data unit B and data unit C is empty.

[0220] If the parent data unit of data unit D needs to be changed, only data unit D needs to be modified. Data units A, B, and C do not need to be modified. Data units A, B, and C can directly use the data in data unit D. At this time, the overlap evaluation value is 1.

[0221] From a reasonable perspective, please refer to Figure 7C (1) The data warehouse level associated with data unit A is the common summary granularity fact layer, and the data warehouse level associated with the parent data unit of data unit A is the data import layer. According to Table 4, data unit A cannot call data from the data import layer.

[0222] Meanwhile, when searching for the parent data unit of data unit B using depth-first search, in a relationship chain, the four parent data units corresponding to data unit B, taking a preset overlap threshold of 3 as an example, the relationship chain of data unit B is relatively long and does not need to be maintained. At this time, the reasonable evaluation value is 0.78.

[0223] Please refer to Figure 7C (2) The data warehouse level associated with data unit A is the common summary granularity fact layer. The parent data unit of data unit A is data unit C. The data warehouse level associated with data unit C is the detailed granularity fact layer. The data warehouse level associated with the parent data unit of data unit C is the data import layer.

[0224] Meanwhile, when searching for the parent data unit of data unit B using depth-first search, in a relation chain, the three parent data units corresponding to data unit B do not have a long relation chain, so the reasonable evaluation value is 1.

[0225] The server can periodically perform quality assessments on the data warehouse. When the quality assessment results do not meet the preset requirements, an alarm will be issued to prompt maintenance operations on the data warehouse until the quality assessment results of the data warehouse are maintained to meet the preset requirements, such as when the reuse assessment value, the overlapping assessment value, and the reasonable assessment value are all 1.

[0226] Based on the same inventive concept, embodiments of this application provide an apparatus for evaluating a data warehouse, capable of performing the functions corresponding to the aforementioned method for evaluating a data warehouse. Please refer to... Figure 8 The device includes an acquisition module 801 and a processing module 802, wherein:

[0227] Acquisition module 801: used to acquire the data logical relationships between the data units contained in the data warehouse to be evaluated, wherein each data logical relationship represents that, between two corresponding data units, the parent data unit is the data source of the child data unit;

[0228] Processing module 802: Used to obtain at least one target relationship chain composed of each data unit that satisfies the preset unit association conditions based on the obtained logical relationships of each data unit;

[0229] The processing module 802 is also used to: determine the reusability of each data unit as a parent data unit based on the logical relationships of each data unit, and determine the overlap between the parent data units corresponding to each pair of data units when they are child data units;

[0230] The processing module 802 is also used to: perform a quality assessment of the data warehouse based on the obtained reusability, overlap and the number of chains of at least one target relation chain, and obtain the corresponding quality assessment results.

[0231] In one possible embodiment, the acquisition module 801 is specifically used for:

[0232] Iterate through all the data units contained in the data warehouse to be evaluated and create a list of data units;

[0233] Based on the data retrieval rules of the data warehouse, the data source of each data unit in the data unit list is determined sequentially to obtain each parent data unit and the child data units corresponding to each parent data unit.

[0234] Based on the correspondence between each parent data unit and its corresponding child data unit, a relationship list is established to obtain the data logical relationship between each data unit.

[0235] In one possible embodiment, the data warehouse is set up with multiple ordered data warehouse levels, each data unit is associated with a corresponding data warehouse level, and each data warehouse level represents the data purpose of the corresponding data unit.

[0236] The processing module 802 is specifically used for:

[0237] Based on the obtained logical relationships of each data unit, each candidate relation chain is obtained, wherein there is a corresponding logical relationship between two adjacent data units in each candidate relation chain;

[0238] From the candidate relation chains, select the candidate relation chains that do not form a circular chain, contain no more than a preset number threshold number of first units of data units, and do not cross levels between the data warehouse levels corresponding to any two adjacent data units, and obtain at least one target relation chain.

[0239] In one possible embodiment, the processing module 802 is specifically used for:

[0240] For each data unit, perform the following operations:

[0241] Based on the logical relationships between the data, count the number of second units of the corresponding child data unit when a data unit is used as the parent data unit;

[0242] The number of the second unit is counted as the reuse rate of a data unit.

[0243] In one possible embodiment, the processing module 802 is specifically used for:

[0244] For every two data units, perform the following operations:

[0245] Based on the logical relationships of each data unit, obtain at least one first relation chain when a data unit is only a sub-data unit, and at least one second relation chain when another data unit is only a sub-data unit.

[0246] Check whether there are identical data units between each first relation chain and each second relation chain, and count the number of third units with identical data units;

[0247] The overlap between the corresponding parent data units when the number of the third unit is counted as one data unit and another data unit is counted as a child data unit.

[0248] In one possible embodiment, the processing module 802 is specifically used for:

[0249] The reuse evaluation value of the data warehouse is determined by the ratio between the sum of each reuse degree and the total number of data units.

[0250] Based on the number of overlaps that are greater than the preset overlap threshold among the statistically obtained overlap degrees, the overlap assessment value of the data warehouse is determined.

[0251] A reasonable evaluation value for the data warehouse is determined based on the ratio between the number of chains and the total number of units.

[0252] The quality assessment result is determined by a weighted sum of the reused assessment value, the overlapping assessment value, and the reasonable assessment value.

[0253] In one possible embodiment, the processing module 802 is specifically used for:

[0254] Count the number of overlaps with overlaps greater than the preset overlap threshold among all overlap levels;

[0255] Count the number of permutations and combinations of any two data units in each data unit;

[0256] The difference between the specified value and the ratio of the number of overlaps to the number of combinations is used as the overlap assessment value.

[0257] In one possible embodiment, the processing module 802 is specifically used for:

[0258] The weighted sum of reused assessment values, overlapping assessment values, and reasonable assessment values ​​is used as the quality score of the data warehouse.

[0259] Based on the preset mapping relationship between each score range and each quality level, the target quality level corresponding to the target score range containing the quality score is determined. Each quality level represents the proportion of the amount of data units and data logical relationships to be optimized in the data warehouse to the total amount of each data unit and each data logical relationship.

[0260] The target quality level is used as the result of the quality assessment.

[0261] Please refer to Figure 9 The aforementioned apparatus for evaluating the data warehouse can run on a computer device 900. The current and historical versions of the data storage program, as well as the application software corresponding to the data storage program, can be installed on the computer device 900, which includes a processor 980 and a memory 920. In some embodiments, the computer device 900 may include a display unit 940, which includes a display panel 941 for displaying a user-interactive interface, etc.

[0262] In one possible embodiment, the display panel 941 may be configured in the form of a liquid crystal display (LCD) or an organic light-emitting diode (OLED).

[0263] The processor 980 is used to read a computer program and then execute the methods defined by the computer program. For example, the processor 980 reads a data storage program or file, thereby running the data storage program on the computer device 900 and displaying the corresponding interface on the display unit 940. The processor 980 may include one or more general-purpose processors, and may also include one or more DSPs (Digital Signal Processors) for performing related operations to implement the technical solutions provided in the embodiments of this application.

[0264] The memory 920 generally includes main memory and secondary storage. Main memory can be random access memory (RAM), read-only memory (ROM), and cache, etc. Secondary storage can be a hard disk, optical disk, USB flash drive, floppy disk, or magnetic tape drive, etc. The memory 920 is used to store computer programs and other data. The computer programs include applications corresponding to each client, and other data may include data generated after the operating system or applications are run, including system data (e.g., operating system configuration parameters) and user data. In this embodiment, program instructions are stored in the memory 920, and the processor 980 executes the program instructions in the memory 920 to implement any of the methods described in the preceding figures.

[0265] The aforementioned display unit 940 is used to receive input digital information, character information, or contact touch operations / non-contact gestures, and to generate signal inputs related to user settings and function control of the computer device 900. Specifically, in this embodiment, the display unit 940 may include a display panel 941. The display panel 941, for example, is a touch screen, which can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or on the display panel 941), and drive corresponding connection devices according to a pre-set program.

[0266] In one possible embodiment, the display panel 941 may include two parts: a touch detection device and a touch controller. The touch detection device detects the player's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller. The touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 980. It can also receive and execute commands from the processor 980.

[0267] The display panel 941 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the display unit 940, in some embodiments, the computer device 900 may also include an input unit 930. The input unit 930 may include an image input device 931 and other input devices 932, wherein the other input devices may include, but are not limited to, one or more of the following: a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick.

[0268] In addition to the above, the computer device 900 may also include a power supply 990 for powering other modules, an audio circuit 960, a near-field communication module 970, and an RF circuit 910. The computer device 900 may also include one or more sensors 950, such as an accelerometer, a light sensor, and a pressure sensor. The audio circuit 960 specifically includes a speaker 961 and a microphone 962, for example, the computer device 900 can use the microphone 962 to collect the user's voice and perform corresponding operations.

[0269] As one embodiment, the number of processors 980 can be one or more, and the processors 980 and the memory 920 can be coupled together or relatively independent.

[0270] As one example, Figure 9 The processor 980 in the middle can be used to implement, for example Figure 8 The functions of the acquisition module 801 and the processing module 802 in the process.

[0271] As one example, Figure 9 The processor 980 in the text can be used to implement the functions of the server or terminal devices discussed above.

[0272] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0273] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of software products, for example, through a computer program product. This computer program product is stored in a storage medium and includes several instructions to cause a computer device to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0274] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for evaluating a data warehouse, characterized in that, include: Obtain the logical relationships between the data units contained in the data warehouse to be evaluated, wherein each logical relationship represents that, between two corresponding data units, the parent data unit is the data source of the child data unit; Based on the obtained logical relationships of each data unit, at least one target relationship chain that satisfies the preset unit association conditions is obtained; Based on the aforementioned data logical relationships, the reuse degree of each data unit as a parent data unit is determined, and the overlap between the parent data units corresponding to each pair of data units when they are child data units is determined; wherein, the reuse degree represents the number of child data units corresponding to the corresponding data unit when it is a parent data unit. Based on the obtained reusability, overlap and number of chains of the at least one target relation chain, the data warehouse is quality assessed to obtain the corresponding quality assessment results. Among them, determining the overlap between the parent data units of each pair of data units when they are each child data units includes: For every two data units, perform the following operations: Based on the aforementioned data logical relationships, at least one first relation chain is obtained when a data unit is only a sub-data unit, and at least one second relation chain is obtained when another data unit is only a sub-data unit. Each first relation chain and each second relation chain is matched to see if there are any identical data units, and the number of third units with identical data units is counted. The number of the third unit is used as the overlap between the corresponding parent data units when the first data unit and the second data unit are respectively used as sub-data units.

2. The method according to claim 1, characterized in that, The process of obtaining the logical relationships between the data units contained in the data warehouse to be evaluated includes: Iterate through all the data units contained in the data warehouse to be evaluated and create a list of data units; Based on the data retrieval rules of the data warehouse, the data source of each data unit in the data unit list is determined sequentially to obtain each parent data unit and the child data units corresponding to each parent data unit. Based on the correspondence between each parent data unit and its corresponding child data unit, a relationship list is established to obtain the data logical relationship between each data unit.

3. The method according to claim 1, characterized in that, The data warehouse is set up with multiple orderly data warehouse levels. Each data unit is associated with a corresponding data warehouse level, and each data warehouse level represents the data purpose of the corresponding data unit. The step of obtaining at least one target relationship chain that satisfies the preset unit association conditions based on the obtained logical relationships of each data unit includes: Based on the obtained logical relationships of each data unit, each candidate relationship chain composed of the data units is obtained, wherein there is a corresponding logical relationship between two adjacent data units in each candidate relationship chain; From the candidate relation chains, select the candidate relation chain that does not form a circular chain, contains no more than a preset number threshold number of data units, and has no cross-level data warehouse hierarchy between any two adjacent data units as the target relation chain, and obtain the at least one target relation chain.

4. The method according to claim 1, characterized in that, The determination of the reusability of each data unit as a parent data unit based on the logical relationships between the data includes: For each of the data units, perform the following operations respectively: Based on the aforementioned data logical relationships, count the number of second units of the corresponding child data units when a data unit is used as a parent data unit; The number of the second unit is used as the reuse rate of the data unit.

5. The method according to any one of claims 1 to 4, characterized in that, The data warehouse is quality assessed based on the obtained reusability, overlap, and the number of chains in the at least one target relation chain, to obtain corresponding quality assessment results, including: The reuse evaluation value of the data warehouse is determined based on the ratio between the sum of the reuse values ​​and the total number of data units. Based on the number of overlaps that are greater than a preset overlap threshold among the statistically obtained overlap degrees, the overlap evaluation value of the data warehouse is determined. A reasonable evaluation value for the data warehouse is determined based on the ratio between the number of chains and the total number of units. The quality assessment result is determined based on the weighted sum of the reuse assessment value, the overlap assessment value, and the reasonable assessment value.

6. The method according to claim 5, characterized in that, The method of determining the overlap assessment value of the data warehouse based on the number of overlaps with overlaps greater than a preset overlap threshold among the statistically calculated overlap degrees includes: The number of overlaps with overlaps greater than a preset overlap threshold is counted among the various overlap degrees. Count the number of combinations of any two data units in each of the aforementioned data units; The difference between the specified value and the ratio of the number of overlaps to the number of combinations is used as the overlap evaluation value.

7. The method according to claim 5, characterized in that, The determination of the quality assessment result based on the weighted sum of the reuse assessment value, the overlap assessment value, and the reasonable assessment value includes: The weighted sum of the reuse evaluation value, the overlap evaluation value, and the reasonable evaluation value is used as the quality score of the data warehouse. Based on the preset mapping relationship between each score range and each quality level, the target quality level corresponding to the target score range containing the quality score is determined. Each quality level represents the proportion of the amount of data units and data logical relationships to be optimized in the data warehouse to the total amount of each data unit and each data logical relationship. The target quality level is used as the quality assessment result.

8. An apparatus for evaluating a data warehouse, characterized in that, include: Acquisition module: used to acquire the logical relationships between the data units contained in the data warehouse to be evaluated. Each logical relationship represents that, between two corresponding data units, the parent data unit is the data source of the child data unit. Processing module: used to obtain at least one target relationship chain that satisfies the preset unit association conditions, based on the obtained logical relationships of each data unit; The processing module is further configured to: determine the reusability of each data unit as a parent data unit based on the data logical relationships, and determine the overlap between the parent data units corresponding to each pair of data units when they are child data units; wherein, the reusability represents the number of child data units corresponding to the corresponding data unit when it is a parent data unit; The processing module is further configured to: perform a quality assessment on the data warehouse based on the obtained reusability, overlap and the number of chains of the at least one target relation chain, and obtain the corresponding quality assessment results; Specifically, the processing module is used to perform the following operations for every two data units: Based on the aforementioned data logical relationships, at least one first relation chain is obtained when a data unit is only a sub-data unit, and at least one second relation chain is obtained when another data unit is only a sub-data unit. Each first relation chain and each second relation chain is matched to see if there are any identical data units, and the number of third units with identical data units is counted. The number of the third unit is used as the overlap between the corresponding parent data units when the first data unit and the second data unit are respectively used as sub-data units.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the method as described in any one of claims 1 to 7.

10. A computer device, characterized in that, include: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute the method as described in any one of claims 1 to 7 according to the obtained program instructions.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the method as described in any one of claims 1 to 7.

Images