Index similarity screening method and device based on hierarchical index system

Abstract

The application provides a kind of index similarity screening method and device based on hierarchical index system, it is related to data index management technical field, the method steps include that external data source is collected to data warehouse ODS layer, and then processing forms DWD layer wide table, and records data blood relationship;Three-layer index system including atomic index, derived index and composite index is built, wherein the atomic index is the non-divisible measurement value of business activity, the derived index is generated based on atomic index association time period and modifier, and the composite index is generated by calculating derived index through operation rule;Graph structure is constructed with ODS layer table, DWD layer table, atomic index, derived index and composite index as node, and data flow relationship as edge;The similarity of index itself is calculated based on the node type in graph structure respectively, and the similarity of data source is calculated by combining the similarity of adjacent node of edge, to screen high similarity index.

Description

Technical Field

[0001] This invention relates to the field of data indicator management technology, and in particular to an indicator similarity screening method and apparatus based on a hierarchical indicator system. Background Technology

[0002] As enterprises deepen their digital transformation, various organizations generate massive amounts of data in their daily operations, and departments develop a large number of data metrics for analysis and decision support based on business needs.

[0003] However, the current industry standard for data indicator management is that each department independently develops data indicators based on basic data and their own business definitions. Each department defines indicators based on its own business understanding, directly extracting data from raw data tables or files to calculate indicators, without establishing a standardized data layering process. Because the meaning of indicators is related to the data source, it is difficult to identify the meaning of indicators. There are cases where indicators have the same name but inconsistent meanings, or vice versa. Therefore, there is currently a problem of redundant development of data indicators within organizations. Summary of the Invention

[0004] The purpose of this invention is to provide a method and apparatus for screening indicators based on a hierarchical indicator system, so as to solve the problem of redundant development of data indicators within organizations mentioned in the background art.

[0005] To achieve the above objectives, this invention provides the following technical solution: a method for screening indicators based on a hierarchical indicator system, comprising the following steps: collecting external data sources into the data warehouse ODS layer, then processing them to form a DWD layer wide table, and recording data lineage; constructing a three-layer indicator system including atomic indicators, derived indicators, and composite indicators, wherein the atomic indicators are indivisible metric values ​​of business activities, the derived indicators are generated based on the time period and modifiers associated with the atomic indicators, and the composite indicators are calculated and generated by the derived indicators through operational rules; constructing a graph structure with the ODS layer table, DWD layer table, atomic indicators, derived indicators, and composite indicators as nodes, and data flow relationships as edges; calculating the similarity of the indicators themselves based on the node types in the graph structure, and calculating the data source similarity by combining the similarity of adjacent nodes of the incoming edges, so as to screen indicators with high similarity.

[0006] Optionally, the atomic indicator includes indicator code, indicator name, indicator definition, indicator theme, DWD data table, measurement field, calculation method, and dimension; the derived indicator includes indicator code, indicator name, indicator definition, indicator theme, atomic indicator, time period, modifier, calculation method, and dimension; the composite indicator includes indicator code, indicator name, indicator definition, indicator theme, atomic indicator, derived indicator, calculation method, and dimension.

[0007] Optionally, the data lineage relationship includes the mapping relationship between the source system, source database, source table, source field and ODS layer table field, as well as the mapping relationship between ODS layer table and field and DWD layer table and field.

[0008] Optionally, the step of calculating the similarity of each index based on the node type in the graph structure specifically includes: calculating the index similarity between atomic indices, calculating the index similarity between derived indices, and calculating the index similarity between composite indices.

[0009] Optionally, the step of calculating the similarity of the index itself based on the node type in the graph structure and calculating the similarity of the data source in combination with the similarity of the adjacent nodes of the incoming edge further includes: when two nodes are the same, the similarity is 1; when two nodes are different and there are adjacent nodes of the incoming edge, the weighted sum of the similarity of the adjacent nodes of the incoming edge and the similarity of itself is calculated; when two nodes are different and there are no adjacent nodes of the incoming edge, only the similarity of itself is calculated.

[0010] Optionally, the self-similarity calculation steps specifically include: for ODS layer or DWD layer table nodes, calculating the average pooling vector similarity after embedding the table name and field name into a vectorized model; for atomic indicator nodes, calculating the weighted average pooling vector similarity after embedding the atomic indicator into a vectorized model; for derived indicator nodes, calculating the weighted average pooling vector similarity after embedding the derived indicator into a vectorized model; and for composite indicator nodes, calculating the weighted average pooling vector similarity after embedding the composite indicator into a vectorized model.

[0011] Optionally, the sum of the weights of the similarity between the adjacent nodes of the incoming edge and the similarity of itself is 1, and the weight of the similarity of itself is greater than the weight of the similarity between the adjacent nodes of the incoming edge.

[0012] On the other hand, the present invention also provides an indicator similarity screening device based on a hierarchical indicator system, comprising: a collection module for collecting external data sources into the data warehouse ODS layer, then processing them into a DWD layer wide table and recording data lineage; an indicator system construction module for constructing a three-layer indicator system including atomic indicators, derived indicators, and composite indicators, wherein the atomic indicators are indivisible metric values ​​of business activities, the derived indicators are generated based on the time period and modifiers associated with the atomic indicators, and the composite indicators are calculated and generated by the derived indicators through calculation rules; a graph structure construction module for constructing a graph structure with ODS layer table, DWD layer table, atomic indicators, derived indicators, and composite indicators as nodes and data flow relationships as edges; and an indicator similarity screening module for calculating the similarity of the indicators themselves based on the node types in the graph structure, and calculating the data source similarity by combining the similarity of adjacent nodes of the incoming edges, so as to screen highly similar indicators.

[0013] On the other hand, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described index similarity screening method based on a hierarchical index system.

[0014] On the other hand, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described index similarity screening method based on a hierarchical index system.

[0015] Compared with the prior art, the beneficial effects of the present invention are:

[0016] This application constructs a complete indicator similarity screening system for data indicator management systems. Through layered data collection and processing, the construction of a three-layer indicator system, graph structure representation, and similarity calculation, it effectively solves the problems of redundant indicator construction and difficulty in meaning identification within organizations. Through systematic indicator management and structured similarity calculation methods, it can automatically identify highly similar indicators, helping users clarify the relationships between indicators, reduce data redundancy, and improve the efficiency and accuracy of data indicator management. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the method steps of the present invention.

[0018] Figure 2 This is a flowchart of the method of the present invention.

[0019] Figure 3 This is a data flow diagram of the method of the present invention.

[0020] Figure 4 This is a schematic diagram of the system structure of the present invention.

[0021] In the diagram: 10 - Data collection module, 20 - Indicator system construction module, 30 - Graph structure construction module, 40 - Indicator similarity screening module. Detailed Implementation

[0022] The present invention will now be clearly and completely described in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be used interchangeably where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] Those skilled in the art will understand that, unless explicitly stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in the specification of this application means the presence of features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0025] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0026] It should be understood that the sequence number and size of each step in this embodiment do not imply the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application embodiment.

[0027] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0028] Please refer to Figures 1-3 This invention discloses a method for screening indicators based on a hierarchical indicator system. The steps include: collecting data from external data sources into the data warehouse ODS layer, processing it to form a DWD layer wide table, and recording data lineage; constructing a three-layer indicator system including atomic indicators, derived indicators, and composite indicators, wherein the atomic indicators are indivisible metrics of business activities, the derived indicators are generated based on the time period and modifiers associated with the atomic indicators, and the composite indicators are calculated and generated from the derived indicators using operational rules; constructing a graph structure with the ODS layer table, DWD layer table, atomic indicators, derived indicators, and composite indicators as nodes, and data flow relationships as edges; calculating the similarity of each indicator based on its node type in the graph structure, and combining the similarity of adjacent nodes with incoming edges to calculate the data source similarity, in order to screen highly similar indicators.

[0029] Specifically, the external data sources for this system are data tables or data files from various databases. The data acquisition unit collects the data tables or data files from the external data sources into the ODS layer of this system's data warehouse and records the data lineage. The data development unit joins the fact tables and dimension tables in the ODS layer to form a wide table, saves it to the DWD layer of this system's data warehouse, and records the data lineage.

[0030] Furthermore, the types of the fields marked in the DWD layer table are metric and dimension. Metrics represent calculable numerical data (such as number of credit card transactions, sales revenue), while dimensions represent categorical data used for analysis perspectives (such as gender, grade). If the type is dimension, select the dimension in dimension management. All analysis perspectives (such as gender, grade, course) are predefined through the dimension management module to form a standardized dimension catalog. A matching item must be selected from the predefined dimension library for binding. Creating new dimensions is prohibited. This provides standardized input for subsequent indicator similarity calculations.

[0031] Furthermore, a three-tiered indicator system is constructed, comprising atomic indicators, derived indicators, and composite indicators. Atomic indicators are indivisible, and their data source is the data tables in the Data Warehouse (DWD) layer. One atomic indicator can derive multiple derived indicators. The time period defined in the time period management module is selected based on the time field of the atomic indicator, followed by the aforementioned modifiers. Finally, the dimension of the derived indicator is selected from the dimension of the atomic indicator source. Composite indicators are processed by introducing atomic and derived indicators. The atomic indicators are indivisible measures of business activities, the derived indicators are generated based on the time period and modifiers associated with the atomic indicators, and the composite indicators are calculated and generated from the derived indicators using computational rules.

[0032] Furthermore, a graph structure is constructed with ODS layer table, DWD layer table, atomic indicators, derived indicators and composite indicators as nodes and data flow relationships as edges. The similarity of the indicators themselves is calculated based on the node types in the graph structure, and the similarity of the data source is calculated by combining the similarity of adjacent nodes of the incoming edges, so as to screen the indicators with high similarity.

[0033] This application constructs a complete indicator similarity screening system for data indicator management systems. Through layered data collection and processing, the construction of a three-layer indicator system, graph structure representation, and similarity calculation, it effectively solves the problems of redundant indicator construction and difficulty in meaning identification within organizations. Through systematic indicator management and structured similarity calculation methods, it can automatically identify highly similar indicators, helping users clarify the relationships between indicators, reduce data redundancy, and improve the efficiency and accuracy of data indicator management.

[0034] In some embodiments, the atomic indicator includes an indicator code, indicator name, indicator definition, indicator theme, DWD data table, measurement field, calculation method, and dimension; the derived indicator includes an indicator code, indicator name, indicator definition, indicator theme, atomic indicator, time period, modifier, calculation method, and dimension; the composite indicator includes an indicator code, indicator name, indicator definition, indicator theme, atomic indicator, derived indicator, calculation method, and dimension.

[0035] Specifically, atomic metrics include metric code, metric name, metric definition, metric theme, DWD data table, measure field, calculation method, and dimensions. An atomic metric is the smallest indivisible measure in the metric system. The data source for atomic metrics is the data tables in the DWD layer of the data warehouse. The aggregation method for the table fields is defined, and calculation methods include summation, count, maximum, minimum, and average. The dimensions of the atomic metric are defined, selecting a subset of dimensions from the DWD data tables. The atomic metric name directly reflects the business event behavior and does not contain modifiers; example: number of student card swipes.

[0036] Optionally, derived metrics include metric code, metric name, metric scope, metric theme, atomic metric, time period, modifiers, calculation method, and dimension. First, select the aforementioned atomic metric as the data source. One atomic metric can derive multiple derived metrics. Then, select the time period defined in the time period management module based on the time field in the atomic metric. Next, select the aforementioned modifiers. Finally, select the dimension of the derived metric from the dimensions of the data source atomic metric. The naming rules for derived metrics are as follows: Derived metric name = Time period (participating in naming) + Business scope (participating in naming) + Condition constraints (participating in naming) + Atomic metric (participating in naming) + Dimension (not participating in naming). Example of derived metric name: Number of computer science students swiping their cards in the library this week.

[0037] Optionally, composite indicators include indicator code, indicator name, indicator scope, indicator theme, atomic indicators, derived indicators, calculation expression, and dimension. Define the indicator scope, using natural language to describe the definition and calculation logic of the composite indicator. Define the calculation expression, referencing atomic and derived indicators, and using logical operators, aggregate functions, calculation operators, calculation functions, and conditional expressions to construct the composite indicator calculation expression. Indicator processing is performed by introducing atomic and derived indicators. Example of a composite indicator: Composite indicator name "Access Control Usage Percentage", statistical scope: percentage of all students using access control systems (number of access control users divided by the current number of students on campus), calculation expression: Access Control Usage Percentage = Number of Access Control Users / Current Number of Students on Campus.

[0038] This application establishes a standardized indicator description framework by clearly defining the specific components of atomic, derived, and composite indicators, making the meaning and origin of various indicators clearer and more traceable. This structured indicator definition method eliminates the problems of confusing indicator naming and ambiguous meanings, provides a unified data foundation for subsequent similarity calculations, and ensures the consistency and reliability of similarity assessment.

[0039] In some embodiments, the data lineage relationship includes the mapping relationship between the source system, source database, source table, source field and ODS layer table field, as well as the mapping relationship between the ODS layer table and field and the DWD layer table and field.

[0040] The technical solution in this application, which meticulously records data lineage, ensures complete traceability of data from its source to the final indicator, constructing a transparent path for data flow. This refined lineage mapping not only helps in understanding the true meaning of the indicator and the data source, but also accurately captures the similarity characteristics of the data source in similarity calculation, thereby improving the comprehensiveness and accuracy of similarity assessment.

[0041] In some embodiments, the step of calculating the similarity of the indicators based on the node types in the graph structure specifically includes: calculating the similarity between atomic indicators, calculating the similarity between derived indicators, and calculating the similarity between composite indicators.

[0042] Specifically, the ODS layer table, DWD layer table, atomic indicators, derived indicators, and composite indicators form a directional data flow relationship from left to right. Tables or indicators are nodes, and data flow relationships are edges, constructing a graph structure. The similarity between two atomic indicators is calculated. The similarity between two derived indicators is calculated. The similarity between two composite indicators is calculated.

[0043] This application implements a classification-based approach to index similarity calculation, employing specialized similarity calculation methods for different types of indicators. This classification method fully considers the differences in characteristics among various indicators, making similarity assessment more accurate and effectively identifying truly similar or duplicate indicators, thus avoiding misjudgments that may result from a single calculation method.

[0044] In some embodiments, the step of calculating the similarity of the index itself based on the node type in the graph structure and calculating the similarity of the data source in combination with the similarity of the adjacent nodes of the incoming edge further includes: when the two nodes are the same, the similarity is 1; when the two nodes are different and there are adjacent nodes of the incoming edge, the weighted sum of the similarity of the adjacent nodes of the incoming edge and the similarity of the index itself is calculated; when the two nodes are different and there are no adjacent nodes of the incoming edge, only the similarity of the index itself is calculated.

[0045] Specifically, a and b represent two nodes in the graph, and the similarity between nodes a and b is calculated using the following formula:

[0046] ;

[0047] In the formula: a and b represent two nodes in the graph. Indicates the similarity between two nodes. This represents the similarity between two nodes. and Each represents a set of nodes in another subset connected to the incoming edges of a and b. and represents the number of nodes in the set of nodes of another subset connected by the incoming edges of a and b, respectively. This represents the i-th node connected to the incoming edge 'a'. This represents the j-th node connected to the incoming edge b. and These are the weighting coefficients.

[0048] When nodes a and b are the same, the similarity is... The value is 1.

[0049] When nodes a and b are different, and a and b have adjacent nodes with incoming edges, the decomposition consists of the similarity between adjacent nodes with incoming edges and the similarity between the two nodes themselves. Weighted sum.

[0050] When nodes a and b are different, and a and b have no adjacent nodes with incoming edges, calculate the similarity between the two nodes. .

[0051] This application achieves flexibility and adaptability in similarity calculation by defining similarity calculation rules under three different conditions. The technical solution considers both the similarity of the nodes themselves and the influence of the similarity of the data sources. By using a weighted calculation method, it balances these two factors, making the similarity calculation results more comprehensive and reliable, and able to accurately reflect the true degree of similarity between indicators.

[0052] In some embodiments, the self-similarity calculation steps specifically include: for ODS layer or DWD layer table nodes, calculating the average pooling vector similarity after vectorizing the table name and field name using an embedding model; for atomic indicator nodes, calculating the weighted average pooling vector similarity after vectorizing the atomic indicator using an embedding model; for derived indicator nodes, calculating the weighted average pooling vector similarity after vectorizing the derived indicator using an embedding model; and for composite indicator nodes, calculating the weighted average pooling vector similarity after vectorizing the composite indicator using an embedding model.

[0053] Specifically, when nodes a and b are ODS layer table nodes or DWD layer table nodes, the node features are first converted into vectors using the bge-m3 embedding model, then average pooling is performed, and finally the similarity between the two nodes is calculated.

[0054] The array of table names for node 'a' and the mapping field names for adjacent nodes on the incoming edges is as follows: Embedded model After vectorization, we get ; Perform average pooling on A to obtain a vector .

[0055] The array of table names and mapped field names for node b is as follows: Embedded model After vectorization, we get Average pooling is performed on B to obtain the vector. .

[0056] Calculate the self-similarity between nodes a and b. . It is the dot product of two vectors. It is the product of the L2 norms of two vectors.

[0057] Optionally, when nodes a and b are atomic indicator nodes, the node features are first converted into vectors using an embedding model, then weighted average pooling is performed, and finally the similarity between the two nodes is calculated.

[0058] The indicator name, indicator definition, indicator theme, DWD data table, measure field, calculation method, and dimension array for node A are as follows: Embedded model After vectorization, we get Weighted average pooling is performed on A to obtain a vector. ;in , indicating that the sum of the weight coefficients of all 7 feature dimensions is 1.

[0059] The indicator name, indicator definition, indicator theme, DWD data table, measure field, calculation method, and dimension array of node b are as follows: Embedded model After vectorization, we get Weighted average pooling is performed on B to obtain the vector. ;in , indicating that the sum of the weight coefficients of all 7 feature dimensions is 1.

[0060] Calculate the self-similarity between nodes a and b. . It is the dot product of two vectors. It is the product of the L2 norms of two vectors.

[0061] Optionally, when nodes a and b are derived indicator nodes, the node features are first converted into vectors using an embedding model, then weighted average pooling is performed, and finally the similarity between the two nodes is calculated.

[0062] The indicator name, indicator definition, indicator theme, atomic indicator, time period, modifiers, calculation method, and dimension array of node A are as follows: Embedded model After vectorization, we get Weighted average pooling is performed on A to obtain a vector. ;in This indicates that the sum of the weight coefficients of all 8 feature dimensions is 1.

[0063] The indicator name, indicator definition, indicator theme, atomic indicator, time period, modifiers, calculation method, and dimension array of node b are as follows: Embedded model After vectorization, we get Weighted average pooling is performed on B to obtain the vector. ;in This indicates that the sum of the weight coefficients of all 8 feature dimensions is 1.

[0064] Calculate the self-similarity between nodes a and b. . It is the dot product of two vectors. It is the product of the L2 norms of two vectors.

[0065] Optionally, when nodes a and b are composite indicator nodes, the node features are first converted into vectors using an embedding model, then weighted average pooling is performed, and finally the similarity between the two nodes is calculated.

[0066] The indicator name, indicator definition, indicator theme, atomic indicator, derived indicator, calculation expression, and dimension array of node a are as follows: Embedded model After vectorization, we get Weighted average pooling is performed on A to obtain a vector. ;in , indicating that the sum of the weight coefficients of all 7 feature dimensions is 1.

[0067] The indicator name, indicator definition, indicator theme, atomic indicator, derived indicator, calculation method, and dimension array of node b are as follows: Embedded model After vectorization, we get Weighted average pooling is performed on B to obtain the vector. ;in , indicating that the sum of the weight coefficients of all 7 feature dimensions is 1.

[0068] Calculate the self-similarity between nodes a and b. . It is the dot product of two vectors. It is the product of the L2 norms of two vectors.

[0069] This application designs differentiated vector processing and similarity calculation methods for different types of node features. By employing embedding model vectorization and pooling operations, the semantic information of node features can be effectively captured and transformed into computable similarity values. This approach preserves the richness of the original information while achieving quantitative evaluation of similarity, greatly improving the accuracy of index similarity analysis.

[0070] In some embodiments, the sum of the weights of the similarity between the incoming edge neighboring nodes and the similarity of the self is 1, and the weight of the self-similarity is greater than the weight of the similarity between the incoming edge neighboring nodes.

[0071] This application, by rationally setting the weight allocation in similarity calculation, highlights the dominant role of the similarity of the indicators themselves, while appropriately considering the influence of data sources. This weight design ensures that the similarity calculation results focus more on the semantic and functional similarity of the indicators themselves, avoiding evaluation bias caused by over-reliance on data sources, and making the similarity screening results more in line with actual business needs.

[0072] Please refer to Figure 4 On the other hand, the present invention also provides an indicator similarity screening device based on a hierarchical indicator system, comprising: a collection module for collecting external data sources into the data warehouse ODS layer, then processing them into a DWD layer wide table and recording data lineage; an indicator system construction module for constructing a three-layer indicator system including atomic indicators, derived indicators, and composite indicators, wherein the atomic indicators are indivisible metric values ​​of business activities, the derived indicators are generated based on the time period and modifiers associated with the atomic indicators, and the composite indicators are calculated and generated by the derived indicators through calculation rules; a graph structure construction module for constructing a graph structure with ODS layer table, DWD layer table, atomic indicators, derived indicators, and composite indicators as nodes and data flow relationships as edges; and an indicator similarity screening module for calculating the similarity of the indicators themselves based on the node types in the graph structure, and calculating the data source similarity by combining the similarity of adjacent nodes of the incoming edges, so as to screen highly similar indicators.

[0073] On the other hand, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described index similarity screening method based on a hierarchical index system.

[0074] On the other hand, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described index similarity screening method based on a hierarchical index system.

[0075] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0076] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, database, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0077] The above are merely embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention's specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for screening indicators based on hierarchical indicator systems using indicator similarity, characterized by the following steps: include: External data sources are collected into the data warehouse ODS layer, then processed to form a DWD layer wide table, and the data lineage is recorded; A three-tiered indicator system is constructed, comprising atomic indicators, derived indicators, and composite indicators. The atomic indicators are indivisible metric values ​​for business activities. The derived indicators are generated based on the time period and modifiers associated with the atomic indicators. The composite indicators are calculated and generated by the derived indicators through calculation rules. The atomic indicators include indicator code, indicator name, indicator definition, indicator theme, DWD data table, measurement field, calculation method, and dimension; The derived indicators include indicator code, indicator name, indicator scope, indicator theme, atomic indicator, time period, modifiers, calculation method, and dimension; The composite indicator includes indicator code, indicator name, indicator scope, indicator theme, atomic indicator, derived indicator, calculation method, and dimension; Construct a graph structure with ODS layer table, DWD layer table, atomic indicators, derived indicators and composite indicators as nodes and data flow relationships as edges; The self-similarity of the indicators is calculated based on the node types in the graph structure. The specific steps for calculating the self-similarity include: For ODS or DWD layer table nodes, the table name and field name are embedded into the model vectorization process and then the average pooling vector similarity is calculated. For atomic index nodes, the atomic indexes are vectorized by embedding the model and then the weighted average pooling vector similarity is calculated. For derived indicator nodes, the derived indicators are embedded into the model vectorization process and then the weighted average pooling vector similarity is calculated. For composite indicator nodes, the composite indicator is vectorized by embedding the model and then the weighted average pooling vector similarity is calculated. The data source similarity is calculated by combining the similarity of the adjacent nodes of the incoming edge. When two nodes are different and have adjacent nodes of the incoming edge, the weighted sum of the similarity of the adjacent nodes of the incoming edge and the similarity of the node itself is calculated. The sum of the weights of the similarity of the adjacent nodes of the incoming edge and the similarity of the node itself is 1, and the weight of the similarity of the node itself is greater than the weight of the similarity of the adjacent nodes of the incoming edge, so as to filter high similarity indicators.

2. The index similarity screening method based on a hierarchical index system according to claim 1, characterized in that, The data lineage relationship includes the mapping relationship between the source system, source database, source table, source field and ODS layer table field, as well as the mapping relationship between ODS layer table and field and DWD layer table and field.

3. The index similarity screening method based on a hierarchical index system according to claim 1, characterized in that, The steps for calculating the similarity of each index based on the node type in the graph structure specifically include: The similarity between atomic indicators is calculated, the similarity between derived indicators is calculated, and the similarity between composite indicators is calculated.

4. The index similarity screening method based on a hierarchical index system according to claim 3, characterized in that, The steps of calculating the similarity of the indicators themselves based on the node types in the graph structure, and calculating the similarity of the data sources by combining the similarity of adjacent nodes of the incoming edges, further include: When two nodes are identical, the similarity is 1. When two nodes are different and have no adjacent nodes with incoming edges, only their own similarity is calculated.

5. A similarity screening device based on a hierarchical indicator system, characterized in that, include: The data acquisition module is used to collect data from external data sources into the data warehouse ODS layer, and then process it to form a wide table in the DWD layer, and record the data lineage. The indicator system construction module is used to construct a three-layer indicator system including atomic indicators, derived indicators, and composite indicators. The atomic indicators are indivisible metric values ​​of business activities, the derived indicators are generated based on the time period and modifiers associated with the atomic indicators, and the composite indicators are calculated and generated by the derived indicators through calculation rules. The atomic indicators include indicator code, indicator name, indicator definition, indicator theme, DWD data table, measurement field, calculation method, and dimension; The derived indicators include indicator code, indicator name, indicator scope, indicator theme, atomic indicator, time period, modifiers, calculation method, and dimension; The composite indicator includes indicator code, indicator name, indicator scope, indicator theme, atomic indicator, derived indicator, calculation method, and dimension; The graph structure construction module is used to construct a graph structure with ODS layer table, DWD layer table, atomic indicators, derived indicators and composite indicators as nodes and data flow relationships as edges. The indicator similarity filtering module is used to calculate the self-similarity of indicators based on the node type in the graph structure. The calculation steps for the self-similarity specifically include: For ODS or DWD layer table nodes, the table name and field name are embedded into the model vectorization process and then the average pooling vector similarity is calculated. For atomic index nodes, the atomic indexes are vectorized by embedding the model and then the weighted average pooling vector similarity is calculated. For derived indicator nodes, the derived indicators are embedded into the model vectorization process and then the weighted average pooling vector similarity is calculated. For composite indicator nodes, the composite indicator is vectorized by embedding the model and then the weighted average pooling vector similarity is calculated. The data source similarity is calculated by combining the similarity of the adjacent nodes of the incoming edge. When two nodes are different and have adjacent nodes of the incoming edge, the weighted sum of the similarity of the adjacent nodes of the incoming edge and the similarity of the node itself is calculated. The sum of the weights of the similarity of the adjacent nodes of the incoming edge and the similarity of the node itself is 1, and the weight of the similarity of the node itself is greater than the weight of the similarity of the adjacent nodes of the incoming edge, so as to filter high similarity indicators.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the indicator similarity screening method based on a hierarchical indicator system as described in any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the indicator similarity screening method based on the hierarchical indicator system as described in any one of claims 1 to 4.

Images