Data index consistency identification method and device, equipment, medium and product

Abstract

The invention discloses a data index consistency identification method and device, equipment, a medium and a product, and relates to the technical field of big data processing, and the method comprises the steps: obtaining index metadata; performing index grouping based on the index metadata to obtain a first candidate index group, the first candidate index group having two to-be-identified indexes; tracing the two indexes in the first candidate index group based on the index metadata to obtain first blood relationship information corresponding to the two indexes; and based on the first consanguinity information corresponding to the two indexes, carrying out homology and computational logic identification to obtain an index consistency identification result of the two indexes. According to the method, automatic identification of data index consistency can be realized, and the identification efficiency and accuracy are improved.

Description

Technical Field

[0001] This application relates to the field of big data processing technology, specifically to methods, apparatus, equipment, media, and products for identifying the consistency of data indicators. Background Technology

[0002] In big data management, data consistency is a core prerequisite for ensuring data quality and supporting business decisions. Especially as business complexity increases, a single business scenario often involves thousands to tens of thousands of data metrics, making consistency identification a significant challenge. Therefore, a method for identifying data metric consistency is needed to address this challenge. Summary of the Invention

[0003] In view of this, this application provides a method, apparatus, device, medium and product for identifying data indicator consistency, so as to solve the problem of identifying data indicator consistency.

[0004] Firstly, this application provides a method for identifying the consistency of data indicators, including: Obtain indicator metadata; Based on the indicator metadata, the indicators are grouped to obtain a first candidate indicator group, which has two indicators to be identified. For the two indicators in the first candidate indicator group, the source is traced based on the indicator metadata to obtain the first lineage information corresponding to the two indicators. Based on the first bloodline information corresponding to the two indicators, homology and calculation logic are identified to obtain the consistency identification result of the two indicators.

[0005] Secondly, this application provides a data indicator consistency identification device, comprising: The metadata acquisition module is used to acquire indicator metadata; The indicator grouping module is used to group indicators based on the indicator metadata to obtain a first candidate indicator group, wherein the first candidate indicator group has two indicators to be identified. The tracing module is used to trace the origin of two indicators in the first candidate indicator group based on the indicator metadata to obtain the first lineage information corresponding to the two indicators. The consistency identification module is used to identify the homology and calculation logic based on the first bloodline information corresponding to the two indicators, and to obtain the consistency identification result of the two indicators.

[0006] Thirdly, this application provides an electronic device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the data index consistency identification method described in the first aspect or any corresponding embodiment.

[0007] Fourthly, this application provides a computer-readable storage medium storing computer instructions for causing a computer to execute the data indicator consistency identification method described in the first aspect or any corresponding embodiment.

[0008] Fifthly, this application provides a computer program product, including computer instructions, which are used to cause a computer to execute the data indicator consistency identification method described in the first aspect or any corresponding embodiment.

[0009] The data indicator consistency identification method provided in this application involves: acquiring indicator metadata; grouping indicators based on the metadata to obtain a first candidate indicator group, with each first candidate indicator group containing two indicators to be identified; tracing the origins of each indicator within the first candidate indicator group based on its metadata to obtain the first lineage information corresponding to the two indicators; and identifying the homology and computational logic based on the first lineage information to obtain the indicator consistency identification result for the two indicators. This method performs preliminary grouping based on the acquired indicator metadata to obtain the first candidate indicator group. For the two indicators within the first candidate indicator group, through tracing, homology identification, and computational logic identification, it achieves the entire process of indicator data consistency identification, enabling automatic identification of data indicator consistency and improving identification efficiency and accuracy. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the specific embodiments or related technologies of this application, the drawings used in the description of the specific embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram illustrating an application scenario according to an embodiment of this application; Figure 2 This is a schematic diagram of the first process of a method for identifying the consistency of data indicators according to an embodiment of this application; Figure 3 This is a schematic diagram of a second process for identifying the consistency of data indicators according to an embodiment of this application; Figure 4This is a schematic diagram of the third process of the data indicator consistency identification method according to the embodiments of this application; Figure 5a This is a schematic diagram illustrating the collection of indicator metadata according to an embodiment of this application; Figure 5b This is a schematic diagram of index grouping according to an embodiment of this application; Figure 5c This is a traceability diagram based on an embodiment of this application; Figure 5d This is a schematic diagram illustrating the similarity and computational logic identification according to embodiments of this application; Figure 6 This is a structural block diagram of a data index consistency identification device according to an embodiment of this application; Figure 7 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0014] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0015] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0016] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0017] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0018] For ease of description below, the terms used in the embodiments of this application are explained as follows: A data warehouse is a subject-oriented, integrated, relatively stable collection of data that reflects historical changes and is used to support management decisions.

[0019] Metric definition: A standard specification for calculating and defining metrics in a data warehouse. It clarifies key elements such as the business meaning, calculation logic, data source, and data scope of the metrics, and is an important basis for ensuring the accuracy, consistency, and availability of data in the data warehouse.

[0020] Metadata: Data about data, providing information about the content, quality, source, relationships, and other characteristics of the data. Examples include data production lineage, table structure, and table production code.

[0021] Data lineage: Metadata used to record the rotation path and relationships of data throughout its entire lifecycle, covering the complete link and dependencies from the source of generation, processing and transformation to consumption scenarios, providing basic support for data traceability, quality control and other aspects.

[0022] Large Language Model: An artificial intelligence model built on deep learning technology. By training on a large amount of text data, it can understand the structure, semantics and grammar of language, thereby generating natural and fluent text content.

[0023] Indicator consistency: refers to the degree of matching between indicators of the same name or type in different scenarios or data links in terms of core elements such as data source, calculation logic and data scope. It is a key indicator for measuring data reliability and business synergy.

[0024] In related technologies, the identification of data consistency metrics is generally handled through manual comparison or simple rule engines. However, this approach suffers from low efficiency and insufficient accuracy. Specifically, the aforementioned solutions can only reflect a single layer of the dashboard during the data consistency identification process, failing to fully cover the core information in the data warehouse processing chain. Since the key calculation logic of metrics is often implemented in intermediate layer tables, deriving downstream HIVE tables alone cannot reflect the processing logic of the intermediate layer, resulting in one-sided comparisons and difficulty in accurately judging metric consistency.

[0025] Based on this, this application provides a method for identifying data indicator consistency, which involves obtaining indicator metadata; grouping indicators based on the indicator metadata to obtain a first candidate indicator group, wherein the first candidate indicator group has two indicators to be identified; for the two indicators in the first candidate indicator group, tracing their origins based on the indicator metadata to obtain the first lineage information corresponding to the two indicators; and identifying the homology and calculation logic based on the first lineage information corresponding to the two indicators to obtain the indicator consistency identification result of the two indicators.

[0026] This method uses a workflow of metadata collection, similar indicator grouping, and consistency identification. Based on the acquired indicator metadata, it performs preliminary grouping to obtain the first candidate indicator group. For the two indicators in the first candidate indicator group, it achieves the entire process of indicator data consistency identification through tracing, same-source identification, and calculation logic identification. This enables automatic identification of data indicator consistency, improving identification efficiency and accuracy.

[0027] Furthermore, this method combines end-to-end lineage, large-scale model, and business rule optimization strategies to achieve efficient and high-precision consistency identification of large-scale data warehouse metrics.

[0028] As one optional application scenario of this disclosure embodiment, such as Figure 1 As shown, the user configures the data warehouse to be identified through the data metric consistency identification application on terminal 101. Correspondingly, a data metric consistency identification method is deployed on server 102, and terminal 101 communicates with server 102. Terminal 101 can be a mobile terminal, a computer, etc. Server 102 can be deployed in the cloud, a data center, or an edge computing node, etc., depending on actual needs, and no limitations are imposed here.

[0029] The number of terminals 101 connected to server 102 is not limited to... Figure 1 As shown, the settings can be configured according to actual needs. It should be understood that the number of servers used for dataset detection is not limited to... Figure 1 As shown, it could be one or more, or it could be a deployed cluster, etc.

[0030] Server 102 identifies the consistency of data metrics in the deployed data warehouse, obtains the consistency identification result, and feeds the result back to terminal 101. Correspondingly, the identification result is displayed on the page of terminal 101. Furthermore, users can interact with the page of terminal 101 to correct the identification result, thereby improving the accuracy of data metric consistency identification.

[0031] According to an embodiment of this application, an embodiment of a method for identifying data indicator consistency is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0032] This embodiment provides a method for identifying data indicator consistency, which can be used in the aforementioned server. Figure 2 This is a flowchart of a data indicator consistency identification method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps: Step S201: Obtain indicator metadata.

[0033] For the same metric, it is generally displayed at the consumption layer through dashboards and charts. Dashboards and charts are usually associated with business lines. Accordingly, the metric metadata includes the metric name, the business line to which the metric belongs, and the structured query language (SQL) statement corresponding to the metric.

[0034] For example, when you open a metric in the user interface, the interface will display the metric name, the metric dimensions, and the dataset from which its data source originates. This dataset can be traced back to its upstream Hive table at the data warehouse level. Based on this, the metric metadata can include the metric name, aggregation fields, and analysis dimensions, etc., where the aggregation fields and analysis dimensions can be represented by the SQL code corresponding to the metric.

[0035] Indicator metadata can be represented in text form. For example, metadata belonging to the same indicator can be concatenated together to obtain the metadata representation of that indicator.

[0036] Step S202: Group the indicators based on the indicator metadata to obtain the first candidate indicator group.

[0037] The first candidate indicator group has two indicators to be identified.

[0038] Indicator grouping is used to initially group the acquired indicator metadata and filter out indicator groups that may have data indicator consistency; these are called the first candidate indicator groups. For example, for any two indicator metadata sets, the text similarity can be calculated, and the first candidate indicator group is determined based on the magnitude of the similarity. For instance, indicator groups with text similarity greater than a certain threshold are identified as the first candidate indicator groups. It should be understood that the threshold value can be set according to actual needs, and no limitations are imposed here.

[0039] In the first candidate indicator group, there are two indicators to be identified. For example, for a certain data warehouse, the similarity of the metadata of any pair of indicators in the data warehouse is calculated. If the similarity is greater than a certain threshold, it is regarded as the first candidate indicator group. The same indicator may exist in different first candidate indicator groups. After indicator grouping, the indicators in the data warehouse can be divided into multiple first candidate indicator groups.

[0040] It should be understood that differences in naming and other aspects of the same metric across different data sources may lead to low similarity in the metadata of the same metric. Therefore, before calculating the similarity of metric metadata, it is advisable to standardize the metric metadata. For example, this could involve replacing metric terms in the metric name and unifying synonyms to eliminate interference from differences in expression.

[0041] Step S203: For the two indicators in the first candidate indicator group, trace their origins based on the indicator metadata to obtain the first lineage information corresponding to the two indicators.

[0042] Source tracing of an indicator is used to trace the data source upon which the indicator depends. This is because indicators that originally belong to the same data source may exhibit different characteristics after going through different processing paths. Therefore, source tracing can identify indicators that belong to the same data source.

[0043] Specifically, tracing is performed based on the first candidate indicator group. That is, for two indicators in the first candidate indicator group, tracing is performed using their indicator metadata. For example, after obtaining the indicator metadata, it is necessary to transform the discrete indicator metadata into a traceable lineage relationship through syntax parsing combined with logical association. The core of this is parsing table-level lineage and field-level lineage to obtain the first lineage information corresponding to the two indicators.

[0044] The tracing of lineage can be done at a preset level (e.g., 5 levels) or by setting a stopping condition, such as stopping tracing upon encountering a data entry log table to avoid redundant upstream information. It should be understood that both the tracing level and the stopping condition are set according to actual needs and are not subject to any limitations here.

[0045] Step S204: Based on the first bloodline information corresponding to the two indicators, the homology and calculation logic are identified to obtain the indicator consistency identification result of the two indicators.

[0046] Homology identification, or common origin identification, involves analyzing the primary lineage information corresponding to two indicators to determine whether they originate from the same data source. This analysis can be achieved using a large model or by organizing the primary lineage information into a tree structure and comparing the node information of two lineage trees to determine if they share a common lineage. For a large model, the input includes the primary lineage information corresponding to the two indicators, and the output includes a conclusion on whether they are from the same source.

[0047] The identification of computational logic is used to determine whether the computational logic of two metrics is consistent, such as data sources, aggregation functions, and filtering conditions. During the identification process, only core computational logic can be identified, while non-core differences can be ignored. The distinction between core and non-core computational logic is related to the actual business scenario and is not limited here. For example, differences in non-core computational logic include, but are not limited to, unit conversion, dimension aggregation methods, period differences, and global filtering conditions.

[0048] It should be understood that the identification of common origins and the identification of computational logic can be performed in parallel; alternatively, the identification of computational logic can be performed after the two indicators are identified as having common origins, and so on. There are no restrictions on the order of identification of common origins and computational logic; the specific order can be set according to actual needs.

[0049] If two indicators share the same origin and have the same calculation logic, they can be considered consistent. That is, the consistency identification result of the two indicators is consistent; otherwise, the consistency identification result of the two indicators is inconsistent.

[0050] The data indicator consistency identification method provided in this embodiment performs preliminary grouping based on the obtained indicator metadata to obtain a first candidate indicator group. For two indicators within the first candidate indicator group, the method performs source tracing, same-source identification, and calculation logic identification to realize the entire process of indicator data consistency identification. This enables automatic identification of data indicator consistency, improving identification efficiency and accuracy.

[0051] This embodiment provides a method for identifying data indicator consistency, which can be used in the aforementioned server. Figure 3 This is a flowchart of a data indicator consistency identification method according to an embodiment of this application, such as... Figure 3 As shown, the process includes the following steps: Step S301: Obtain indicator metadata. See details below. Figure 2 Step S201 of the illustrated embodiment will not be described again here.

[0052] Step S302: Group the indicators based on the indicator metadata to obtain the first candidate indicator group.

[0053] The first candidate indicator group has two indicators to be identified.

[0054] Specifically, step S302 includes: Step S3021: Obtain the indicator name and the object name of the display object to which the indicator belongs from the indicator metadata.

[0055] The display object to which the indicator belongs is used to represent the charts, cards, and dashboards to which the indicator is displayed in the user interface. That is, the display object includes, but is not limited to, charts, cards, and dashboards.

[0056] The object names displayed are chart name, card name, and dashboard name, which can be obtained when retrieving indicator metadata.

[0057] Step S3022: Based on the weights corresponding to the indicator name and the object name, the indicator name and the object name are fused to obtain the comparison features.

[0058] Since the same metric name may exist in different display objects—for example, the metric name "sum" may appear in both chart 1 and chart 2, and chart 1 and chart 2 may correspond to different business scenarios—combining the chart name with metric similarity comparison can further differentiate business scenarios.

[0059] That is, the indicator name and the object name are merged. The merging method can be concatenation. For example, the concatenated feature is: object name - indicator name.

[0060] Furthermore, to highlight the importance of the indicator name, different weights can be assigned to the indicator name and the object name respectively, and the indicator name and object name can be merged by combining their respective weights. That is, the merged comparison feature can be represented as: object name * weight 1 - indicator name * weight 2.

[0061] Step S3023: Calculate the similarity between any two paired features.

[0062] After obtaining the comparative features of the indicators, the similarity between any two comparative features can be calculated. The method of similarity calculation can be set according to actual needs, and no restrictions are imposed here.

[0063] Step S3024: Obtain the first candidate index group based on the similarity of the compared features.

[0064] For any two contrasting features, a similarity score can be obtained. The similarity score is compared with a threshold. If it is greater than the threshold, it indicates that the indicators corresponding to the two contrasting features may have consistency. This is then used as the first candidate indicator group for further identification.

[0065] In some alternative implementations, step S3024 above includes: Step a1: The indicator group to which the comparison features with similarity greater than a preset threshold belong is taken as the second candidate indicator group.

[0066] Step a2: Filter the second candidate indicator group using preset field names to obtain the first candidate indicator group. The preset field names correspond to the target business, and the preset fields are used to represent indicator information that does not require consistent identification.

[0067] After comparing similarity with a preset threshold, the indicator groups that exceed the preset threshold are selected as the second candidate indicator groups. Based on the second candidate indicator groups, further grouping optimization can be performed by combining preset field names.

[0068] The preset field names represent indicator information that does not require consistency identification, such as a whitelist of assets to be decommissioned and indicators with the same name in different regions. For example, there may be some data tables in a data warehouse that are about to be decommissioned. It is not very meaningful to identify the consistency of indicators in these data tables. Therefore, by maintaining a whitelist of assets to be decommissioned, the indicator names in the second candidate indicator group can be further filtered by comparing them with the asset whitelist.

[0069] For example, it is not very meaningful to compare the same indicators in different regions. For instance, there is no meaning in comparing the total assets in region 1 with the total assets in region 2. Therefore, the preset field name can be used to represent the preset region field. That is, the indicator name and the object name of the displayed object in the second candidate indicator group are compared with the preset region field. If the indicator name or the object name of the displayed object contains the preset region field, then it is filtered.

[0070] Therefore, the first candidate indicator group was obtained by screening based on the second candidate indicator group.

[0071] It should be noted that the preset field names correspond to the target business, that is, to the specific business scenario. Accordingly, the preset field names are set according to the target business, and no restrictions are placed on them here.

[0072] For the second candidate indicator group with a similarity greater than a preset threshold, the second candidate indicator group is further filtered by the preset field name corresponding to the target business to avoid ungrouped indicators with the same indicator name in different partitions, and to exclude indicators that do not need to be identified, thereby reducing invalid grouping.

[0073] Step S303: For the two indicators in the first candidate indicator group, trace their origins based on the indicator metadata to obtain the first lineage information corresponding to the two indicators.

[0074] Specifically, step S303 includes: Step S3031: For the two indicators in the first candidate indicator group, the first structured query statement in the indicator metadata is trimmed to obtain the second structured query statement.

[0075] To reduce the amount of data processing required for subsequent tracing, SQL pruning is performed on the two indicators in the first candidate indicator group. For example, pruning rules can be set to describe the objects that need to be pruned, and code pruning tools can be used to prune the SQL, removing redundant logic and improving the parsing efficiency of subsequent lineage agents.

[0076] In some alternative implementations, step S3031 includes: Step b1: Based on preset rules, the first structured query statement is trimmed to obtain the second structured query statement.

[0077] Step b2: If the second structured query statement cannot find the corresponding data source, it indicates that the pruning of the first structured query statement has failed.

[0078] Step b3: Use the first large model to re-tail the first structured query statement to obtain the second structured query statement.

[0079] The pruning of the first structured query statement can be divided into two levels. First, pruning is performed using preset rules. If pruning fails, then pruning is performed using the first main model. That is, a dual SQL pruning mechanism is used to prune the first structured query statement.

[0080] The first structured query statement is trimmed according to preset rules to obtain the second structured query statement. However, the effectiveness of this trimming cannot be determined during this process. Further source tracing is needed to determine the validity of the trimmed second structured query statement. Specifically, if the trimmed second structured query statement cannot find the corresponding data source, it indicates that the trimming based on the preset rules has failed.

[0081] It should be understood that the query to find the corresponding data source described here needs to be combined with the processing in the subsequent step S3032 to determine whether it is possible to trace back to the data source at the next higher level, i.e., the first-level data source. If it is possible to trace back to the first-level data source, it indicates that the pruning based on the preset rules is effective; if it is not possible to trace back to the first-level data source, it indicates that the pruning based on the preset rules is ineffective.

[0082] It should be noted that the first level is the level above which the first structured query statement directly depends. The tracing process involves going up level by level until a preset level is reached or the tracing stops.

[0083] If the pruning based on the preset rules fails, the first structured query statement is input into the first large model, and the second structured query statement is obtained by pruning the first large model.

[0084] The first structured query statement is pruned first using preset rules. In scenarios where the first structured query statement fails, the first major model is used to prune the first structured query statement. In other words, the effectiveness of SQL statement pruning is improved through a dual pruning approach.

[0085] Step S3032: The bloodline intelligence agent traces the source of the second structured query statement to obtain the first bloodline information.

[0086] The lineage agent is used for tracing origins. Input includes SQL statements, and output includes corresponding lineage information. The tracing rules for the lineage agent include, but are not limited to, field filtering rules, local table priority principles, and cross-datacenter syntax adaptation. Specifically, field filtering rules include ignoring globally filtered fields and retaining only data fields used as calculation factors; the local table priority principle prioritizes tracing local datacenter tables, skipping redundant tables introduced without cross-datacenter definition, thus reducing tracing time; and cross-datacenter syntax adaptation means that for SQL statements in the region-database-table-field format, the region prefix is ​​removed, and only the lineage information in the database-table-field format is output.

[0087] In some alternative implementations, step S3032 includes: Step c1: Input the second structured query statement into the bloodline agent to obtain the candidate bloodline information at the first level.

[0088] Step c2: Filter multiple tasks based on the name of the interfering task to obtain the first task in the candidate lineage information.

[0089] Step c3: If there are multiple first tasks, then filter the multiple first tasks based on task priority to obtain the first level of bloodline information.

[0090] Step c4: The bloodline information of the first level is trimmed, and the trimmed result is input into the bloodline agent for iterative tracing to obtain the first bloodline information.

[0091] After inputting the second structured query statement into the bloodline agent, the first-level candidate bloodline information is obtained. Since there may be data writing for interfering tasks within the candidate bloodline information, the tasks in the candidate bloodline information are filtered based on the names of the interfering tasks, resulting in the first task in the candidate bloodline information.

[0092] For example, by maintaining a keyword list and matching task names, non-routine tasks in the candidate lineage information can be eliminated. Generally, a table has only one write task, but some tables may involve multiple partitions, with one task per partition. Other tables may have backtracking tasks, which might only update certain fields periodically. These tasks are not routine and therefore do not record the core logic of the metrics. Therefore, for the data tables in the candidate lineage information, it is necessary to filter out the truly daily routine tasks from among multiple tasks.

[0093] If multiple first tasks still exist after filtering by distracting task names, they can be further filtered based on task priority to obtain the first-level lineage information. The task priority can be the broadest coverage task, including tasks with MT keywords; followed by tasks without data center keywords and with the shortest names, to ensure the acquisition of valid SQL, i.e., the first-level lineage information.

[0094] It should be understood that the names and priorities of the interference tasks are set according to actual business needs, and no restrictions are imposed on them here.

[0095] After obtaining the first-level bloodline information, the first-level bloodline information is trimmed according to the description of step S3031 above, and the trimmed result is input into the bloodline intelligence agent for iterative tracing to obtain the first bloodline information.

[0096] It should be understood that the tracing process in this application is an iterative process, namely, SQL trimming - tracing - task filtering and optimization - SQL trimming - tracing - ... tracing - task filtering and optimization - SQL trimming. The first lineage information is an integration of the results after all tracing steps, task filtering and optimization, and SQL trimming.

[0097] After the bloodline agent outputs the first-level candidate bloodline information, the first task in the candidate bloodline information is filtered by the interference task name and task priority. On this basis, the first-level bloodline information is pruned to further reduce the number of tokens input to the bloodline agent.

[0098] Step S304 involves identifying homology and computational logic based on the first lineage information corresponding to the two indicators, thereby obtaining the consistency identification result of the two indicators. See details... Figure 2 Step S204 of the illustrated embodiment will not be described again here.

[0099] The data indicator consistency identification method provided in this embodiment, since the object names of displayed objects are generally business-related, can supplement business scenario information and improve the accuracy of indicator grouping by combining the indicator name and the object name of the displayed object revealed by the indicator name when grouping indicators. Before inputting the lineage agent, the first structured data in the indicator metadata is trimmed, which can reduce the number of tokens input to the lineage agent and improve the processing efficiency of the lineage agent.

[0100] This embodiment provides a method for identifying data indicator consistency, which can be used in the aforementioned server. Figure 4 This is a flowchart of a data indicator consistency identification method according to an embodiment of this application, such as... Figure 4 As shown, the process includes the following steps: Step S401: Obtain indicator metadata. See details below. Figure 2 Step S201 of the illustrated embodiment will not be described again here.

[0101] Step S402: Group the indicators based on the indicator metadata to obtain the first candidate indicator group.

[0102] The first candidate indicator group has two indicators to be identified. See details below. Figure 2 Step S202 of the illustrated embodiment, or, Figure 3 Step S302 of the illustrated embodiment will not be described again here.

[0103] Step S403: For the two indicators in the first candidate indicator group, trace their origins based on the indicator metadata to obtain the first lineage information corresponding to the two indicators. See details... Figure 2 Step S202 of the illustrated embodiment, or, Figure 3 Step S303 of the illustrated embodiment will not be described again here.

[0104] Step S404: Based on the first bloodline information corresponding to the two indicators, the homology and calculation logic are identified to obtain the indicator consistency identification result of the two indicators.

[0105] Specifically, step S404 includes: Step S4041: Input the first lineage information corresponding to the two indicators into the homologous intelligent agent to obtain the homologous identification results of the two indicators.

[0106] For two indicators in the first candidate indicator group, their first lineage information is input into a homology agent for homology identification, yielding the homology identification result. The homology agent compares the lineage data sources of the two indicators and outputs the homology judgment result. Furthermore, if they are homologous, homology level information is also obtained. Additionally, if they are homologous, the homology level is marked in the first lineage information of each indicator.

[0107] Step S4042: If the homology identification result indicates that the two indicators are homologous, then based on the first lineage information corresponding to the two indicators and the computational logic agent, the first lineage information is identified by computational logic to obtain the indicator consistency identification result.

[0108] The input to the computational logic agent includes first-lineage information and the identification result of indicator consistency. For example, the computational logic agent performs consistency identification based on core difference rules; that is, it identifies valid differences and ignores non-core differences. Valid differences characterize inconsistencies in core computational logic, while non-core differences include differences in unit transformation, dimension aggregation methods, periodicity, and global filtering conditions, etc.

[0109] It should be understood that the effective differences and non-core differences are set according to actual business needs, and no restrictions are imposed on them here.

[0110] After the first lineage information corresponding to the two indicators is processed by the computational logic agent, the consistent identification result of the two indicators is obtained.

[0111] In some alternative implementations, step S4042 includes: Step d1: Based on the homology level in the homology identification results, the first lineage information corresponding to the two indicators is cropped, retaining the homology level and the upstream level that the homology level depends on, to obtain the second lineage information.

[0112] Step d2: Input the second bloodline information into the computational logic agent to obtain the indicator consistency recognition result. The computational logic agent is configured to work based on the preset difference rules.

[0113] If the homology identification result indicates that the two first lineage information are homologous, that is, the homology level is recorded. In the case of homology, the first lineage information corresponding to the two indicators is pruned, including only the homology level and the upstream level on which the homology level depends, thereby obtaining the second lineage information.

[0114] The second lineage information is then input into the computational logic agent to identify the consistency of indicators, and the corresponding indicator consistency identification results are obtained.

[0115] After identifying the homology level, irrelevant information is removed from the first lineage information corresponding to the two indicators to improve the accuracy of subsequent calculation logic identification.

[0116] The data consistency identification method provided in this embodiment utilizes both source-related intelligent agents and computational logic intelligent agents to identify source and computational logic, thereby improving the efficiency of indicator consistency identification.

[0117] In some optional implementations, the above-mentioned method for identifying data indicator consistency further includes: using a summarizing agent to summarize the first lineage information, the identification results of homology, and the identification results of indicator consistency to obtain summary information on data indicator consistency.

[0118] The summary agent is used to summarize the execution process of each step from a technical perspective and output summary information on the consistency of data indicators. The execution of each step includes: tracing the first lineage information of the two fields, integrating hierarchical lineage (i.e., integrating the lineage information of each level of the two indicators), the execution result of the same-origin agent, lineage hierarchy pruning (i.e., if they are the same-origin, then only the same-origin level and the dependent upstream level are retained), and the execution result of the computational logic agent.

[0119] The execution process of each step is used as input to the summarizing agent to obtain summary information on the consistency of data indicators.

[0120] By summarizing the output information of each agent using a summarizing agent, a summary information that can be understood from a business perspective is obtained.

[0121] As a specific application embodiment of this application, such as Figure 5a As shown, starting with the metrics exposed at the consumption layer, the names of the metrics and the object names of the displayed objects are collected. That is, data is collected from multiple data sources and then the metric information is assembled. Metrics are filtered using a maintained whitelist, retaining only those metrics that have consistent identification significance, resulting in a wide table of metric information. This wide table includes information about the consumption side where the metric resides and the metric's definition.

[0122] like Figure 5b As shown, similarity calculations are performed based on the wide table of indicator information, and indicators with a similarity greater than or equal to 0.7 are identified as candidate indicator groups. Further, task optimization and SQL pruning can be performed based on these candidate indicator groups to obtain the first candidate indicator group.

[0123] like Figure 5cAs shown, for the two indicators in the first candidate indicator group, namely indicator 1 and indicator 2, SQL trimming is performed separately. If rule trimming fails, a code trimming agent is used to trim the code to obtain the trimmed code fragments. The trimmed code fragments are then traced using a lineage agent to obtain the lineage information of each layer. The lineage tracing is an iterative process, which stops after 5 iterations in this embodiment. Based on all the iteration results, the code and hierarchical information are integrated.

[0124] like Figure 5d As shown, the integrated code and hierarchical information are processed by a homogeneous agent to determine homogeneity. If homogeneous, the homogeneous code portion is retained. The homogeneous code portion is then input into a computational logic agent for computational logic determination, outputting whether the definitions are the same or different. If the definitions are the same, it indicates that indicator 1 and indicator 2 are consistent; otherwise, indicator 1 and indicator 2 are inconsistent.

[0125] This solution achieves accurate traceability of the entire chain of data by recalling metadata in a layered manner. Specifically, in the metadata collection stage, layered collection is carried out for the data warehouse consumption layer (e.g., business lines, dashboards, charts) and the data warehouse layer (e.g., [missing information]). Combined with the multi-dimensional traceability rules of the lineage agent, the solution accurately locates the complete processing chain of indicators from the underlying data source to the terminal consumption, especially capturing the calculation logic of the intermediate layer table, which solves the problem of one-sided comparison caused by incomplete coverage of link information.

[0126] Furthermore, this solution integrates the entire process of identifying consistency in business rule optimization, forming an end-to-end rule system. Specifically, in the metadata collection stage, task filtering rules (e.g., filtering of interfering tasks and priority selection strategies for target tasks) ensure the acquisition of valid SQL queries. In the grouping and reshuffling of similar metrics, standardized metric terminology, weighted similarity calculation, and preset field names are used to improve grouping accuracy.

[0127] In the consistency identification stage, false positives are reduced by calculating the core difference judgment rules of the logical intelligent agent. Through deep integration with business scenarios, the efficiency and accuracy of indicator consistency identification in large-scale scenarios are improved.

[0128] This embodiment also provides a data indicator consistency identification device, which is used to implement the above embodiments and preferred embodiments, and will not be repeated for details already described. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0129] This embodiment provides a data indicator consistency identification device, such as... Figure 6 As shown, it includes: The metadata acquisition module 601 is used to acquire indicator metadata.

[0130] The indicator grouping module 602 is used to group indicators based on indicator metadata to obtain a first candidate indicator group, which has two indicators to be identified.

[0131] The tracing module 603 is used to trace the origin of two indicators in the first candidate indicator group based on the indicator metadata to obtain the first lineage information corresponding to the two indicators.

[0132] The consistency identification module 604 is used to identify the homology and calculation logic based on the first bloodline information corresponding to the two indicators, and obtain the consistency identification result of the two indicators.

[0133] In some alternative implementations, the indicator grouping module 602 includes: The name retrieval unit is used to retrieve the indicator name and the object name of the display object to which the indicator belongs from the indicator metadata.

[0134] The fusion unit is used to fuse indicator names and object names based on the weights corresponding to the indicator names and object names to obtain comparison features.

[0135] The computational unit is used to calculate the similarity between any two paired features.

[0136] The candidate indicator group determination unit is used to obtain the first candidate indicator group based on the similarity of the compared features.

[0137] In some optional implementations, the candidate index group determination unit includes: The first screening subunit is used to select the indicator group to which the comparison features with similarity greater than a preset threshold belong as the second candidate indicator group.

[0138] The second filtering subunit is used to filter the second candidate indicator group using preset field names to obtain the first candidate indicator group. The preset field names correspond to the target business, and the preset fields are used to represent indicator information that does not require consistent identification.

[0139] In some alternative implementations, the traceability module 603 includes: The first trimming unit is used to trim the first structured query statement in the metadata of the two indicators in the first candidate indicator group to obtain the second structured query statement.

[0140] The first tracing unit is used to trace the second structured query statement through the bloodline intelligence agent to obtain the first bloodline information.

[0141] In some alternative implementations, the first trimming unit includes: The first trimming subunit is used to trim the first structured query statement based on preset rules to obtain the second structured query statement.

[0142] The failure in the unit indicates that the pruning of the first structured query statement has failed if the second structured query statement cannot find the corresponding data source.

[0143] The second trimming subunit is used to trim the first structured query statement using the first large model to obtain the second structured query statement.

[0144] In some alternative implementations, the first tracing unit includes: The first input subunit is used to input the second structured query statement into the bloodline agent to obtain the first level of candidate bloodline information.

[0145] The third filtering subunit is used to filter multiple tasks based on the name of the interfering task to obtain the first task in the candidate lineage information.

[0146] The fourth filtering subunit is used to filter multiple first tasks based on task priority if multiple first tasks exist, so as to obtain the first level of lineage information.

[0147] The third trimming subunit is used to trim the first-level bloodline information and input the trimmed result into the bloodline agent for iterative tracing to obtain the first bloodline information.

[0148] In some alternative implementations, the consistency identification module 604 includes: The second input unit is used to input the first lineage information corresponding to the two indicators into the homologous intelligent agent to obtain the homologous identification result of the two indicators.

[0149] The computational logic identification unit is used to identify the first lineage information based on the first lineage information corresponding to the two indicators and the computational logic agent if the identification result indicates that the two indicators are from the same source, so as to obtain the indicator consistency identification result.

[0150] In some optional implementations, the computational logic identification unit includes: The fourth trimming subunit is used to trim the first lineage information corresponding to the two indicators based on the lineage level in the lineage identification result, retaining the lineage level and the upstream level that the lineage level depends on, to obtain the second lineage information.

[0151] The computational logic recognition subunit is used to input the second lineage information into the computational logic agent to obtain the indicator consistency recognition result. The computational logic agent is configured to work based on preset difference rules.

[0152] In some alternative implementations, it also includes: The summary module is used to summarize the first lineage information, the identification results of homology, and the identification results of indicator consistency using the summary agent, so as to obtain summary information on the consistency of data indicators.

[0153] The data consistency identification device provided in this disclosure can execute the data consistency identification method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the above modules and units are the same as in the corresponding embodiments described above, and will not be repeated here.

[0154] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure.

[0155] The following is a detailed reference. Figure 7 The diagram illustrates a structural schematic suitable for implementing an electronic device according to embodiments of the present disclosure. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 701, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 702 or a program loaded from memory 708 into random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the electronic device. The processor 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0156] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic devices to exchange data via wireless or wired communication with other devices. Although Figure 7 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0157] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 709, or installed from a memory 708, or installed from a ROM 702. When the computer program is executed by the processor 701, it performs the functions defined in the data indicator consistency identification method of embodiments of this disclosure.

[0158] Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0159] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the data indicator consistency identification method shown in the above embodiments is implemented.

[0160] A portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0161] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and all such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A method for identifying the consistency of data indicators, characterized in that, include: Obtain indicator metadata; Based on the indicator metadata, the indicators are grouped to obtain a first candidate indicator group, which has two indicators to be identified. For the two indicators in the first candidate indicator group, the source is traced based on the indicator metadata to obtain the first lineage information corresponding to the two indicators. Based on the first bloodline information corresponding to the two indicators, homology and calculation logic are identified to obtain the consistency identification result of the two indicators.

2. The method according to claim 1, characterized in that, The step of grouping indicators based on the indicator metadata to obtain a first candidate indicator group includes: Retrieve the indicator name and the object name of the display object to which the indicator belongs from the indicator metadata; Based on the weights corresponding to the indicator names and the object names, the indicator names and object names are fused to obtain comparison features; Calculate the similarity between any two of the compared features; The first candidate index group is obtained based on the similarity of the comparison features.

3. The method according to claim 2, characterized in that, The first candidate index group is obtained based on the similarity of the comparison features, including: The indicator group to which the comparison features with similarity greater than a preset threshold belong is selected as the second candidate indicator group. The second candidate indicator group is filtered using preset field names to obtain the first candidate indicator group. The preset field names correspond to the target business, and the preset fields are used to represent indicator information that does not require consistency identification.

4. The method according to claim 1, characterized in that, For the two indicators in the first candidate indicator group, the source is traced based on the indicator metadata to obtain the first lineage information corresponding to the two indicators, including: For the two indicators in the first candidate indicator group, the first structured query statement in the indicator metadata is trimmed to obtain the second structured query statement. The first lineage information is obtained by tracing the second structured query statement through the lineage intelligence agent.

5. The method according to claim 4, characterized in that, The step of trimming the first structured query statement in the indicator metadata to obtain the second structured query statement includes: The first structured query statement is trimmed based on preset rules to obtain the second structured query statement; If the second structured query statement cannot find the corresponding data source, it indicates that the pruning of the first structured query statement has failed. The first structured query statement is re-trimmed using the first major model to obtain the second structured query statement.

6. The method according to claim 4, characterized in that, The step of tracing the second structured query statement through a bloodline intelligence agent to obtain the first bloodline information includes: Input the second structured query statement into the bloodline intelligence agent to obtain the first level of candidate bloodline information; Based on the names of the interfering tasks, the multiple tasks are filtered to obtain the first task among the candidate lineage information; If there are multiple first tasks, the multiple first tasks are filtered based on task priority to obtain the bloodline information of the first level; The bloodline information at the first level is cropped, and the cropped result is input into the bloodline intelligence agent for iterative tracing to obtain the first bloodline information.

7. The method according to claim 1, characterized in that, The identification of homology and calculation logic based on the first bloodline information corresponding to the two indicators to obtain the indicator consistency identification result of the two indicators includes: Input the first lineage information corresponding to the two indicators into the homologous intelligent agent to obtain the homologous identification results of the two indicators; If the homology identification result indicates that the two indicators are homologous, then based on the first lineage information corresponding to the two indicators and the computational logic agent, the computational logic identification is performed on the first lineage information to obtain the indicator consistency identification result.

8. The method according to claim 7, characterized in that, The step of using the first lineage information corresponding to the two indicators and the computational logic agent to identify the computational logic of the first lineage information and obtain the consistency identification result of the indicators includes: Based on the homology level in the homology identification result, the first lineage information corresponding to the two indicators is cropped, retaining the homology level and the upstream level that the homology level depends on, to obtain the second lineage information; The second bloodline information is input into the computational logic agent to obtain the consistency recognition result of the indicator. The computational logic agent is configured to work based on a preset difference rule.

9. The method according to claim 1, characterized in that, Also includes: The summary agent is used to summarize the first bloodline information, the identification results of the homology, and the identification results of the indicator consistency to obtain summary information on the consistency of data indicators.

10. A data indicator consistency identification device, characterized in that, include: The metadata acquisition module is used to acquire indicator metadata; The indicator grouping module is used to group indicators based on the indicator metadata to obtain a first candidate indicator group, wherein the first candidate indicator group has two indicators to be identified. The tracing module is used to trace the origin of two indicators in the first candidate indicator group based on the indicator metadata to obtain the first lineage information corresponding to the two indicators. The consistency identification module is used to identify the homology and calculation logic based on the first bloodline information corresponding to the two indicators, and to obtain the consistency identification result of the two indicators.

11. An electronic device, characterized in that, include: A memory and a processor are interconnected, the memory stores computer instructions, and the processor executes the computer instructions to perform the data indicator consistency identification method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the data indicator consistency identification method according to any one of claims 1 to 9.

13. A computer program product, characterized in that, Includes computer instructions for causing a computer to execute the identification method for data indicator consistency according to any one of claims 1 to 9.

Images