Method and apparatus for correcting service metadata constraint rule drift

Abstract

This application discloses a method and apparatus for correcting drift in business metadata constraint rules. The method includes: acquiring multiple pieces of business data for a target business within a preset time period, and metadata constraint rules for multiple business fields within each piece of business data; determining the true data distribution feature vector and the baseline data distribution feature vector for each business field based on the metadata constraint rules and the multiple pieces of business data, and determining the constraint confidence level corresponding to the metadata constraint rule for each business field based on the true data distribution feature vector and the baseline data distribution feature vector; and correcting the target metadata constraint rule for the target business field based on the true data distribution feature vector when the constraint confidence level of the target metadata constraint rule for the target business field is lower than a preset threshold. This application solves the technical problems of related methods in correcting business metadata constraint rules, such as slow response, high false alarm rate, and lack of global coordination.

Description

Technical Field

[0001] This application relates to the field of data governance technology, and more specifically, to a method and apparatus for correcting drift in business metadata constraint rules. Background Technology

[0002] In the actual operation of data-intensive industries such as finance, manufacturing, and e-commerce, business rules frequently evolve with policy adjustments, product iterations, or changes in the market environment. This causes predefined metadata constraint rules (such as field types, value ranges, enumeration lists, etc.) to gradually deviate from the actual data distribution, a phenomenon known as "business metadata constraint rule drift." To address this drift problem, existing technical solutions mainly adopt the following correction strategies:

[0003] I. Manual periodic inspections and manual updates: Data governance experts manually modify the metadata database after identifying rule failures through sampling analysis or log review. However, this method has a long response time, with an average cycle from detecting drift to completing repairs. During this period, the false alarm rate for data quality is high, seriously affecting the accuracy of business decisions.

[0004] Second, dynamic adjustment based on fixed thresholds, such as automatically expanding the boundary of a preset safety range when the value of a field exceeds a certain percentage. However, this method lacks adaptability and cannot distinguish between normal business fluctuations and genuine rule drift, making it prone to false alarms or missed alarms. At the same time, threshold setting relies on expert experience, making it difficult to quantify the current effectiveness of the rule.

[0005] Third, single-point rule optimization involves isolating and correcting a single detected anomalous metadata constraint rule without considering other related metadata constraint rules or the global data distribution. However, because this method severs the global connection between metadata constraint rules and business data distribution, the corrected metadata constraint rule may conflict with other metadata constraint rules, thus failing to fundamentally solve the systemic drift problem.

[0006] There is currently no effective solution to the above problems. Summary of the Invention

[0007] This application provides a method and apparatus for correcting drift in business metadata constraint rules, which at least solves the technical problems of related methods in correcting business metadata constraint rules, such as slow response, high false alarm rate and lack of global coordination.

[0008] According to one aspect of the embodiments of this application, a method for correcting drift in business metadata constraint rules is provided, comprising: acquiring multiple pieces of business data of a target business within a preset time period, and metadata constraint rules for multiple business fields within each piece of business data; determining the real data distribution feature vector and the benchmark data distribution feature vector for each business field based on the metadata constraint rules and the multiple pieces of business data, and determining the constraint confidence level corresponding to the metadata constraint rule of each business field based on the real data distribution feature vector and the benchmark data distribution feature vector; and correcting the target metadata constraint rule of the target business field based on the real data distribution feature vector of the target business field when the constraint confidence level of the target metadata constraint rule of the target business field is lower than a preset threshold.

[0009] According to another aspect of the embodiments of this application, a business metadata constraint rule drift correction device is also provided, comprising: an acquisition module, configured to acquire multiple business data of a target business within a preset time period, and metadata constraint rules of multiple business fields within each business data; a determination module, configured to determine the real data distribution feature vector and the benchmark data distribution feature vector of each business field based on the metadata constraint rules and the multiple business data, and determine the constraint confidence level corresponding to the metadata constraint rule of each business field based on the real data distribution feature vector and the benchmark data distribution feature vector; and a correction module, configured to correct the target metadata constraint rule of the target business field based on the real data distribution feature vector of the target business field when the constraint confidence level of the target metadata constraint rule of the target business field is lower than a preset threshold.

[0010] In an exemplary embodiment, the metadata constraint rules include at least a value range; the aforementioned apparatus is further configured to determine the true data distribution feature vector and the baseline data distribution feature vector of each business field based on the metadata constraint rules and multiple business data, using the following method: for each business field, dividing the value range of the current business field into multiple discrete intervals; counting the amount of data in each discrete interval that includes multiple target business data containing the current business field, and determining the true data occurrence probability of each discrete interval by combining the total amount of data in the multiple target business data, summarizing the true data occurrence probabilities of multiple discrete intervals to obtain the true data distribution feature vector of the current business field; determining the theoretical data occurrence probability of each discrete interval based on the value range of the current business field and the preset theoretical distribution type corresponding to the metadata constraint rules of the current business field, summarizing the theoretical data occurrence probabilities of multiple discrete intervals to obtain the baseline data distribution feature vector of the current business field.

[0011] In an exemplary embodiment, the above-described apparatus is further configured to divide the value range of the current business field into multiple discrete intervals by means of the following method: sorting each value in the value range of the current business field in ascending order, determining a first value at the 25th position and a second value at the 75th position in the sorting result; determining the difference between the second value and the first value, and determining the width of the interval by combining the total number of multiple target business data containing the current business field; dividing the value range of the current business field according to the width of the interval to obtain multiple discrete intervals.

[0012] In an exemplary embodiment, the above-described apparatus is further configured to determine the constraint confidence level corresponding to the metadata constraint rule of each business field based on the real data distribution feature vector and the benchmark data distribution feature vector by the following method: for each business field, determining the distribution difference between the real data distribution feature vector and the benchmark distribution feature vector of the current business field, and determining the constraint confidence level corresponding to the metadata constraint rule of the current business field based on the distribution difference, a preset difference reduction control factor and a preset adjustable drift threshold.

[0013] In an exemplary embodiment, the above-described apparatus is further configured to modify the target metadata constraint rules of the target business field based on the real data distribution feature vector of the target business field using the following method: determining a target subgraph containing the target graph node corresponding to the target business field and the multi-order neighbor nodes of the target graph node within a preset data lineage graph, wherein the types of the multiple graph nodes in the data lineage graph include at least one of the following: business, business field, metadata constraint rules, business data table, and the types of the multiple directed edges in the data lineage graph include at least one of the following: data flow edge, constraint dependency edge, data influence edge; encoding the real data distribution feature vector of the target business field and the target subgraph respectively, and fusing the two encoding results to obtain a joint representation vector; and decoding the joint representation vector to obtain new metadata constraint rules.

[0014] In an exemplary embodiment, the above-described apparatus is further configured to encode the real data distribution feature vector of the target business field and the target subgraph respectively by means of the following method, and to fuse the two encoding results to obtain a joint representation vector, including: processing the real data distribution feature vector of the target business field based on a multilayer perceptron to obtain a first representation vector; processing the target subgraph based on a graph neural network to obtain a second representation vector; mapping the first representation vector and the second representation vector to the same vector space, and processing the mapped first representation vector and the mapped second representation vector based on an attention fusion network to obtain corresponding attention weights, and using the attention weights to perform weighted fusion of the mapped first representation vector and the mapped second representation vector to obtain a joint representation vector.

[0015] In an exemplary embodiment, the above-described apparatus is further configured to decode the joint representation vector to obtain new metadata constraint rules by means of the following method: mapping the joint representation vector to the metadata constraint space corresponding to the target business field to obtain new metadata constraint rules, wherein the new metadata constraint rules include at least one of the following: upper and lower limits of value range, null value rate, and probability distribution vector of enumerated values.

[0016] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, which stores a computer program, wherein the computer-readable storage medium, when executed by a processor, performs the steps in any of the above method embodiments.

[0017] According to another aspect of the embodiments of this application, a computer program product or computer program is also provided, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, causing the computer device to perform the steps in any of the method embodiments described above.

[0018] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the steps of any of the above method embodiments through the computer program.

[0019] In this embodiment, based on multiple business data points of the target business within a preset time period and the metadata constraint rules corresponding to multiple business fields within each business data point, the true data distribution feature vector and the benchmark data distribution feature vector of each business field are determined. This is used to calculate the constraint confidence of the metadata constraint rules corresponding to the business fields. When the constraint confidence of a metadata constraint rule for a certain business field is lower than a preset threshold, the rules are corrected based on the true distribution. This technical means reduces the drift detection response time from hours to minutes, significantly lowers the false alarm rate of data quality, and reduces the manpower cost of rule maintenance. It achieves the goal of solving the problem of outdated metadata and quality misjudgment caused by business rule drift, thereby solving the technical problems of delayed response, high false alarm rate, and lack of global coordination in related methods for correcting business metadata constraint rules. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0021] Figure 1This is a flowchart illustrating an optional business metadata constraint rule drift correction method according to an embodiment of this application;

[0022] Figure 2 This is a schematic diagram of an optional process for determining the real data distribution feature vector and the benchmark data distribution feature vector of a business field according to an embodiment of this application;

[0023] Figure 3 This is a flowchart of an optional target metadata constraint rule for modifying a target business field according to an embodiment of this application;

[0024] Figure 4 This is a flowchart of an optional method for determining a joint representation vector according to an embodiment of this application;

[0025] Figure 5 This is a structural block diagram of an optional business metadata constraint rule drift correction device according to an embodiment of this application;

[0026] Figure 6 This is a structural block diagram of a computer terminal for implementing a business metadata constraint rule drift correction method according to an embodiment of this application. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first," "second," etc., used in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] According to an embodiment of this application, a method for correcting drift in business metadata constraint rules is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0030] Figure 1 This is a flowchart illustrating a method for correcting drift in business metadata constraint rules according to an embodiment of this application. Figure 1 As shown, the method includes the following steps:

[0031] Step S102: Obtain multiple business data entries for the target business within a preset time period, as well as metadata constraint rules for multiple business fields within each business data entry.

[0032] In this application embodiment, the aforementioned target business is a specific business entity involved in the enterprise data governance scenario, such as financial transaction business, e-commerce order business, manufacturing production business, etc.

[0033] In this embodiment of the application, the aforementioned preset time period is a time window used to collect data distribution characteristics for drift detection. It can be a 1-minute sliding window, a 5-minute extended window, or a random sampling time period determined based on water tank sampling.

[0034] In this embodiment of the application, the aforementioned multiple business data are a set of data instances that are acquired in real time or processed in batches within a preset time period. These data can be table records in a relational database, real-time streaming data in a Kafka message queue, and structured and unstructured text in a log file.

[0035] In the embodiments of this application, the aforementioned business fields are the various attributes or columns that constitute the business data. For example, the business fields in e-commerce order data may be product ID, price, user ID, transaction time, amount, and status identifier, etc.

[0036] In this application embodiment, the above-mentioned metadata constraint rules are static or baseline rules that define the legality, value range, and association relationship of business fields, such as field data type (e.g., integer, variable-length string), value range (e.g., the price value range can be between 0 and 1000), non-null constraint (i.e., a data integrity rule that forces business fields to contain valid data values ​​when stored and prohibits the storage of null values), uniqueness constraint (i.e., a rule that ensures that the value of a business field cannot be repeated in a specific data set or the entire data set and must remain absolutely unique), enumerated value list (i.e., a rule that restricts business fields to only select one from a predefined set of discrete and fixed value sets), and foreign key association rules with other fields (i.e., a constraint mechanism that establishes a reference integrity relationship between the current business field and other tables in the dataset or other fields in the current table).

[0037] Step S104: Determine the real data distribution feature vector and the benchmark data distribution feature vector for each business field based on the metadata constraint rules and multiple business data, and determine the constraint confidence level corresponding to the metadata constraint rules for each business field based on the real data distribution feature vector and the benchmark data distribution feature vector.

[0038] In this embodiment, the aforementioned real data distribution feature vector is a data distribution status indicator obtained by statistically analyzing the corresponding business fields within multiple pieces of actual collected business data.

[0039] In this embodiment of the application, the aforementioned baseline data distribution feature vector is the desired data distribution state defined by metadata constraint rules.

[0040] In the embodiments of this application, the above-mentioned constraint confidence is an indicator that quantifies the current validity of the metadata constraint rules of the business field. Its range is between 0 and 1, and the lower the constraint confidence, the higher the risk of drift of the metadata constraint rules of the business field.

[0041] Step S106: If the constraint confidence of the target metadata constraint rule of the target business field is lower than the preset threshold, the target metadata constraint rule is modified according to the real data distribution feature vector.

[0042] In this embodiment, the aforementioned preset threshold is a confidence threshold for determining whether the metadata constraint rules of the business field have drifted and triggered a correction action. It can be a drift threshold set based on expert experience, a drift threshold dynamically and adaptively adjusted based on historical false positive and false negative rates, or a drift threshold configured differently according to the business importance of the target business. This embodiment does not impose specific restrictions on the setting of the preset threshold.

[0043] In this application embodiment, the above-mentioned modification is a process of automatically adjusting, updating or regenerating metadata constraint rules based on the actual data distribution feature vector of the target business field.

[0044] Based on the scheme defined in steps S102 to S106 above, it can be understood that in this embodiment of the application, based on multiple business data of the target business within a preset time period and the metadata constraint rules corresponding to multiple business fields in each business data, the real data distribution feature vector and the benchmark data distribution feature vector of each business field are determined. In this way, the constraint confidence of the metadata constraint rule corresponding to the business field is calculated. When the constraint confidence of the metadata constraint rule of a certain business field is lower than a preset threshold, the rule is corrected according to the real distribution. This technical means has achieved the technical effect of reducing the drift detection response time from the hour level to the minute level, greatly reducing the false alarm rate of data quality and the manpower cost of rule maintenance. It has achieved the purpose of solving the problem of metadata aging and quality misjudgment caused by business rule drift, and thus solved the technical problems of related methods having slow response, high false alarm rate and lack of global coordination in correcting business metadata constraint rules.

[0045] The following describes each step of the business metadata constraint rule drift correction method provided in the embodiments of this application, with reference to the specific implementation process.

[0046] As an optional solution, for each business field, it can be determined based on the metadata constraint rules of that business field and multiple business data records containing that business field, through... Figure 2 The process shown is to determine the true data distribution feature vector and the baseline data distribution feature vector for this business field, including:

[0047] Step S11: Divide the value range of the current business field into multiple discrete intervals.

[0048] Optionally, the technical solution provided in step S11 above can be implemented in the following way: the value range of the current business field is evenly or according to a strategy divided into a preset number of discrete spaces so as to discretize the continuous data distribution.

[0049] Optionally, the technical solution provided in step S11 above can also be implemented as follows: sort the values ​​in the current business field in ascending order, and determine the first value at the 25th position and the second value at the 75th position in the sorting result; determine the difference between the second value and the first value, and determine the interval width by combining the total number of target business data containing the current business field; divide the value field of the current business field according to the interval width to obtain multiple discrete intervals.

[0050] The principle behind the above partitioning scheme is to adaptively determine the width of the partitioning interval based on the actual value distribution of the current business field across multiple target business data sets. Then, the value range of the current business field is divided according to this partitioning interval width to obtain multiple discrete intervals. Specifically, the interquartile range (IQR) is used to adapt to the density and dispersion of the data distribution, thereby avoiding statistical bias caused by fixed binning. The first value mentioned above is the first quartile (Q1) in the data sorting result from smallest to largest, reflecting the specific numerical position of the lower quartile in the data distribution; the second value is the third quartile (Q3) in the data sorting result from smallest to largest, reflecting the specific numerical position of the upper quartile in the data distribution. Therefore, the above difference is the difference between the first and third quartiles, i.e., the interquartile range (IQR), which is used to quantify the dispersion of the middle 50% of the data. The width of the interval is calculated based on the interquartile range and the total number of business data entries containing the current business field. This allows the span of each discrete interval to be adaptively adjusted according to the statistical characteristics of the business data itself, resulting in several continuous, non-overlapping sub-value domains, i.e., multiple discrete intervals.

[0051] For example, the current business field is "product price," with a value range of 0 to 1000 yuan. First, the system extracts all actual values ​​for this field from multiple business data records acquired within a preset time period (e.g., 100 records containing this field), and sorts these values ​​from smallest to largest. Next, if the first value at the 25th percentile (lower quartile Q1) is 200 yuan, and the second value at the 75th percentile (upper quartile Q3) is 600 yuan, then the interquartile range (IQR) is determined based on the second and first values: IQR = 600 - 200 = 400 yuan. Combining this with the total number of business data records (n = 100), the width of the interval is determined using the following formula. Finally, the system uses 17.2 yuan as the interval width and starts from the starting point 0 of the value range to divide the range into multiple discrete intervals, namely [0,17.2), [17.2,34.4), [34.4,51.6), ..., [997.6,1000] until the entire value range is covered, thus discretizing the continuous commodity price distribution into multiple statistical intervals.

[0052] In addition to the implementation schemes listed above, based on the basic concept of this invention, those skilled in the art can also divide discrete intervals using other technical solutions. For example, based on the total number n of multiple target business data items containing the current business field, the number of intervals can be calculated using the following formula. Then, divide the value range of the current business field into k discrete intervals; or, based on the total number n of multiple target business data containing the current business field and the standard deviation of the current business field within the multiple target business data. Determine the width of the interval Then, based on the width of the divided interval The value range of the current business field is divided into multiple discrete intervals. Any modifications to the above implementation scheme made by those skilled in the art should also be within the scope of protection of this invention.

[0053] Step S12: Count the amount of data in each discrete interval of multiple target business data containing the current business field, and determine the probability of the actual data appearing in each discrete interval by combining the total amount of data of multiple target business data. Summarize the probability of the actual data appearing in multiple discrete intervals to obtain the feature vector of the actual data distribution of the current business field.

[0054] For example, step S12 above is a statistical method based on frequency estimation probability. By discretizing a continuous or large range of values ​​into several mutually exclusive and complete intervals, and counting the number of data samples falling into each interval, an empirical probability distribution of the current business field is constructed. This distribution reflects the actual state of the current business field within a preset time window. Specifically: iterate through multiple target business data containing the current business field, determine which pre-defined discrete interval the actual value of the current business field in each target business data falls into, and accumulate the amount of business data in each discrete interval. Then, divide the amount of business data in each discrete interval by the total number of multiple target business data containing the current business field to obtain the empirical probability value of each discrete interval; finally, arrange the empirical probability values ​​of all discrete intervals according to the order of the discrete intervals to form a vector, which is the true data distribution feature vector of the current business field.

[0055] Step S13: Determine the theoretical data occurrence probability of each discrete interval based on the value range of the current business field and the preset theoretical distribution type corresponding to the metadata constraint rules of the current business field, summarize the theoretical data occurrence probabilities of multiple discrete intervals, and obtain the baseline data distribution feature vector of the current business field.

[0056] In this embodiment, the aforementioned preset theoretical distribution type is a probability distribution model defined by the metadata constraint rules of the current business field, which is expected to be followed by the data. This includes, but is not limited to, uniform distribution, normal distribution, exponential distribution, or a specific probability distribution preset according to business logic. For example, the value range of the "product price" field is 0 to 1000 yuan, and the corresponding preset theoretical distribution type could be a uniform distribution, meaning the expected data appears uniformly within the range [0, 1000]. Conversely, the value range of the "user age" field is 18 to 60 years old, and the corresponding preset theoretical distribution type could be a normal distribution, meaning the expected data appears uniformly within the range [0, 1000]. They are concentrated in the vicinity.

[0057] For example, step S13 above involves integrating or summing the continuous theoretical distribution function over discrete intervals to calculate the expected probability of data falling into each discrete interval under the condition of no business drift and data perfectly conforming to preset rules. This establishes a standard, static reference system, i.e., a baseline data distribution feature vector. Specifically: based on the metadata constraints of the current business field (such as field type, value range, and business logic), the probability distribution model that the field should ideally follow is determined. For example, if the metadata constraints stipulate that the "payment amount" field is uniformly distributed between [a, b], then its theoretical distribution type is uniform distribution, and the corresponding theoretical probability density function is... If a certain "user age" field is specified to have an approximate average value between 18 and 60 years old... Standard deviation is If the distribution is normal, then its theoretical distribution type is also normal, and the corresponding theoretical probability density function is... For the i-th discrete interval obtained in step S11 The probability of the theoretical data appearing in the i-th discrete interval is... Finally, the occurrence probabilities of all theoretical data in the discrete intervals are arranged in the order of the discrete intervals to form a vector, which is the baseline data distribution feature vector of the current business field.

[0058] Based on the scheme defined in steps S11 to S13 above, the technical effect of discretizing the continuous data distribution to adapt to streaming statistical computing and quantifying the actual business data distribution and the ideal business data distribution is achieved.

[0059] As an optional approach, for each business field, the constraint confidence level corresponding to the metadata constraint rule of that business field is determined based on the real data distribution feature vector and the benchmark data distribution feature vector. This includes: determining the distribution difference between the real data distribution feature vector and the benchmark distribution feature vector of the current business field, and determining the constraint confidence level corresponding to the metadata constraint rule of the current business field based on the distribution difference, the preset difference reduction control factor, and the preset adjustable drift threshold.

[0060] In this embodiment, the aforementioned distribution difference is a numerical indicator that quantifies the degree of deviation between the actual data distribution feature vector of the current business field and the baseline distribution feature. Its expression can be written as: In the formula, k represents the total number of discrete spaces; Index variables representing discrete space; This represents the probability density (i.e., the value of the i-th element in the feature vector of the real data distribution) of the current business field within the business data at time t, where the value falls within the i-th discrete interval. This represents the probability density (i.e., the value of the i-th element in the feature vector of the baseline data distribution) that the data should theoretically fall within the i-th discrete interval under the theoretical distribution type corresponding to the metadata constraint rules of the current business field; log represents the logarithmic function, which is usually based on the natural constant e (approximately equal to 2.71828).

[0061] It should be noted that in the above calculation process, if If the denominator is zero, direct calculation will result in a denominator of 0, leading to numerical overflow or calculation errors. Similarly, if If it is zero, calculate directly. This would result in negative infinity. Therefore, to improve the numerical stability of the above floating-point operations, a smoothing factor can be added to the calculation formula. Correspondingly, the expression for the distributional dissimilarity mentioned above can also be written as: Smoothing factor It is usually a very small positive constant (e.g.) or ), used to perform additive smoothing on probability values.

[0062] In this embodiment, the constraint confidence level is a probabilistic indicator or score that quantifies the effectiveness of the metadata constraint rules for the current business field in the current data environment, and its expression can be written as: In the formula, This represents the constraint confidence level of the metadata constraint rule for the current business field, reflecting the degree of consistency between the real-time changing data distribution and the theoretical baseline distribution preset by the metadata. Therefore, a high constraint confidence level indicates that the metadata constraint rule is in a healthy state, while a low constraint confidence level indicates that the metadata constraint rule has experienced "business rule drift" and has become invalid or outdated. This indicates the distribution difference of the current business field; k represents the preset difference reduction control factor, which is usually a constant greater than 0 (such as 2), used to constrain the slope of the confidence level as the distribution difference increases; This represents the preset adjustable drift threshold, which is a constant (e.g., 0.7) used to determine the critical point at which the constraint confidence level changes from high to low; e represents the base of the natural logarithm (approximately 2.71828).

[0063] It should be noted that, in order to further adapt to changes in the business environment, It doesn't have to be a fixed value; instead, it can be dynamically adjusted based on a closed-loop feedback mechanism. In the formula, The learning rate is a constant that is greater than 0 and less than 1. It is used to control the step size of threshold adjustment to prevent the threshold from changing too quickly or too slowly due to word fluctuations. This indicates the current adjustable drift threshold, which is the value set before the previous adjustment or initially. This represents the updated adjustable drift threshold; FPR represents the historical false positive rate, which is the ratio of the number of normal metadata constraint rules incorrectly identified as drifting within a past time window to the total number of metadata constraint rules corresponding to the target service; FNR represents the historical false negative rate, which is the ratio of the number of metadata constraint rules that failed to detect actual drifting within a past time window to the total number of metadata constraint rules corresponding to the target service. Therefore, through this closed-loop feedback mechanism, false negatives and false negatives are automatically balanced, making the adjustable drift threshold... It always maintains near-optimal detection performance, thereby adapting to dynamically changing business environments.

[0064] Furthermore, the relationship between the constraint confidence of the metadata constraint rules for each business field and the preset threshold is determined, where:

[0065] If the constraint confidence of the target metadata constraint rule for the target business field is not lower than the preset threshold, it means that the metadata constraint rule for the target business field is still valid and can accurately reflect the current business data characteristics. There is no need to trigger the dynamic alignment correction mechanism, and the existing metadata constraint rule will remain unchanged.

[0066] If the constraint confidence of the target metadata constraint rule for the target business field is not lower than the preset threshold, it means that the metadata constraint rule for the target business field is outdated or inaccurate. Continuing to use the existing metadata constraint rule may lead to an increase in the false alarm rate of data quality or failure of data governance. Therefore, it is necessary to trigger the alignment correction mechanism at this time.

[0067] As an optional solution, the system can, based on the actual data distribution feature vector of the target business field, perform the following... Figure 3 The process steps shown modify the target metadata constraint rules for the target business fields, including:

[0068] Step S21: Determine the target subgraph within the preset data lineage graph, which includes the target graph node corresponding to the target business field and the multi-level neighbor nodes of the target graph node.

[0069] In this embodiment, the aforementioned data lineage graph is a directed graph using a graph database as its storage structure to describe the dependencies, flows, and constraints between various business entities in a data system. The graph nodes represent data assets or business entities, such as business operations (e.g., data extraction, transformation, and loading tasks), business fields (i.e., specific columns and rows in a business data table), metadata constraint rules (e.g., age constraint rules from 0 to 120 years old), and business data tables (i.e., physical tables or logical views storing multiple business data entries). Furthermore, the types of directed edges in the graph represent the relationships between graph nodes, such as data flow edges (i.e., data flows from a source field / table to a target field / table, such as the order amount field in table A → the total sales amount field in table B), constraint dependency edges (i.e., a certain business field or business data table depends on a certain metadata constraint rule for validation or constraint, such as the age field being constrained by a corresponding age constraint rule), and data influence edges (i.e., the change impact relationship between the aforementioned business data table / business field and downstream business data tables, such as the ID number field in table C affecting the user age field in table D).

[0070] In this embodiment of the application, the target subgraph is seeded by the first graph node and the second graph node corresponding to the target metadata constraint rule. The data lineage graph is traversed to extract the k-order neighbor nodes connected to the first graph node and the second graph node through directed edges, where k is a positive integer greater than or equal to 1.

[0071] For example, in an e-commerce order amount verification scenario, if the system detects a business drift in the metadata constraint rule (rule name: non-negative amount constraint) of the "order amount" field, it first identifies the first graph node Node1 corresponding to the "order amount" field and the second graph node Node2 corresponding to the "non-negative amount constraint" within the data lineage graph. These two graph nodes form the initial core of the target subgraph. Next, using Node1 and Node2 as centers, it searches the data lineage graph for edges directly connected to them, as well as their corresponding first-order and second-order neighbor nodes. Finally, it combines all the extracted neighbor nodes and the edges connecting them to form the final target subgraph.

[0072] Therefore, the target subgraph can be understood as follows: starting from the target metadata constraint rules that have drifted and the target business fields that directly constrain them, based on the data flow, constraint dependencies and data influence relationships in the data lineage graph, it expands outward layer by layer, extracting directly related first-order neighbor nodes and second-order neighbor nodes with deep semantic relationships, thereby constructing a local subgraph that includes both the current drifting data characteristics and upstream and downstream dependencies and business background knowledge, thus ensuring that the new metadata constraint rules generated based on the target subgraph conform to both real-time data distribution and global business logic.

[0073] Step S22: Encode the real data distribution feature vector and the target subgraph of the target business field respectively, and fuse the two encoding results to obtain a joint representation vector.

[0074] Optionally, the above joint representation vector can be obtained through Figure 4 The process steps shown are implemented as follows:

[0075] Step S31: Based on the multilayer perceptron, process the real data distribution feature vector of the target business field to obtain the first representation vector.

[0076] Specifically: The real data distribution feature vector of the target business field is used as the input to a multilayer perceptron. Through multiple fully connected layers within the multilayer perceptron (each fully connected layer is followed by a batch normalization layer and a ReLU activation layer), these high-dimensional / discrete statistical features are mapped to a low-dimensional, dense, continuous vector space, yielding computer-understandable data semantics—the first representation vector, denoted as... .

[0077] Step S32: Process the target subgraph based on the graph neural network to obtain the second representation vector.

[0078] Specifically: the target subgraph is used as input to a graph neural network. Information from each neighboring node within the target subgraph is aggregated using graph convolution or attention mechanisms to update the node representation of the current target business field, resulting in a second representation vector, denoted as... .

[0079] Step S33: Map the first representation vector and the second representation vector to the same vector space, and process the mapped first representation vector and the mapped second representation vector based on the attention fusion network to obtain the corresponding attention weights. Then, use the attention weights to perform weighted fusion on the mapped first representation vector and the mapped second representation vector to obtain the joint representation vector.

[0080] Specifically: due to the first representation vector Second characterization vector Since the semantic scales are different, direct fusion may lead to bias. Therefore, the embodiments of this application use two independent linear transformation matrices (or fully connected layers) to respectively transform the first representation vector. Second characterization vector Projecting onto a spatial vector of the same dimension yields the first representation vector after mapping. and the second representation vector after mapping Then, the mapped first representation vector and the second representation vector after mapping The features are concatenated, and the concatenated combined vector is then input into an attention fusion network (which includes one or more fully connected layers and activation functions, with each fully connected layer containing a learnable weight matrix). and bias terms By performing this operation, we can obtain an attention weight output by the attention fusion network. Therefore, the expression for this attention weight can be written as: In the formula Let represent the Sigmoid activation function, used to map any real number to the interval [0,1] to obtain the corresponding probability value; finally, the first and second mapped representation vectors are weighted and fused using attention weights to obtain the joint representation vector. Therefore, the expression for the joint representation vector can be written as: It should be noted that if the above attention weight is close to 1, it indicates that the data distribution characteristics are dominant (e.g., business rules are missing or unclear, and adjustments are mainly made based on the current data situation); while if the above attention weight is close to 0, it indicates that the business semantic characteristics are dominant (e.g., there are strict business constraints such as "the amount is not negative", and even if the data is abnormal, business common sense should be followed first).

[0081] Step S23: Decode the joint representation vector to obtain new metadata constraint rules.

[0082] Optionally, the above-mentioned new metadata constraint rules can be obtained by mapping the joint representation vector to the metadata constraint space corresponding to the target business field to obtain the new metadata constraint rules, wherein the new metadata constraint rules include at least one of the following: upper and lower limits of the value range, null value rate, and probability distribution vector of enumerated values.

[0083] This is because the aforementioned joint representation vector Z is a general semantic vector. The decoder first needs to determine the classification system in which the target business field's metadata constraint rules belong in terms of business logic, such as value range rules (rules that limit the range or distribution of field values), non-empty / completeness rules (rules that limit the proportion or probability threshold of allowed null or missing values ​​in a field), and enumeration / classification rules (rules that limit the field value to belong to a certain item in a predefined set). Then, Z is mapped to the metadata constraint space corresponding to this classification system. Since different types of metadata constraint rules are defined by different parameters, the decoder can use conditional decoding or a multi-task head mechanism to implement this, as follows:

[0084] If the metadata constraint rule is a range-based rule, the decoder can map the joint representation vector Z into two scalar parameters. This refers to the upper and lower limits of the value range. For example, if the target business field is the price of a product, and the two scalar parameters output by the decoder are 10 and 500, then the new metadata constraint rule for the target business field is "the price must be between 10 and 500".

[0085] If the metadata constraint rule is a non-empty / integrity rule, the decoder can map the joint representation vector Z to a probability value or threshold. For example, if the target business field is a user's mobile phone number, and the probability value output by the decoder is 0.01, then the new metadata constraint rule for that target business field is "the proportion of empty mobile phone numbers shall not exceed 1%".

[0086] If the metadata constraint rule is an enumeration / classification rule, the decoder can map the joint representation vector Z into a probability distribution vector or a list of allowed values. For example, if the target business field is order status, and the decoder outputs a list of allowed values ​​of [paid, shipped, completed], then the new metadata constraint rule for that target business field is "the order status can only be one of paid, shipped, or completed".

[0087] Furthermore, to facilitate understanding and use by relevant personnel, the system can also convert the new metadata constraint rules output by the decoder into natural language descriptions. Specifically, the system fills the decoded new metadata constraint rules, time windows (i.e., preset time periods), and other information into a preset natural language output template, obtaining corresponding prompts—that is, because the data distribution of the [field name] within the [time window] has drifted from [a brief description of the original data distribution feature vector] to [a brief description of the data distribution feature vector], the metadata constraint rules have been automatically updated from "[old metadata constraint rules]" to "[new metadata constraint rules]". For example, because the data distribution of [product price] within the [last 24 hours] has drifted from [uniform distribution] to [concentrated in the 100-200 range], the metadata constraint rules have been automatically updated from "price >= 0" to "price >= 10 and price <= 500".

[0088] Finally, the system sends a preparation message to all downstream platforms (such as business intelligence platforms, data service application programming interfaces, etc.) that have subscribed to the metadata constraint rules. This preparation message carries the version number and content of the new metadata constraint rule. Downstream platforms check their own resources and compatibility. If resources are sufficient and there are no conflicts, they can send a confirmation response message to the system; otherwise, they return an abort response message.

[0089] Furthermore, if all downstream platforms respond with confirmation messages, the system sends a commit message, notifying all downstream platforms to formally apply the new metadata constraint rule. This process essentially employs a two-phase commit protocol, ensuring the atomicity of rule changes in a distributed environment. For example, the system can push rule change notifications to the business intelligence platform via the WebSocket protocol. Upon receiving this notification, the business intelligence platform triggers a dashboard reload without requiring a service restart. Similarly, the data service application programming interface (API) automatically refreshes its in-memory metadata constraint rule cache by listening for system-issued change events, again without requiring a service restart.

[0090] Conversely, if any downstream platform sends a stop response message, the system sends a stop message, ensuring that all downstream platforms maintain their original metadata constraint rules. This ensures the atomicity of rule changes in a distributed environment and avoids data pollution issues caused by inconsistent data quality check standards.

[0091] It's important to note that when the new metadata constraint rules are officially implemented on all downstream platforms, they will not take full effect immediately. Instead, a phased, low-volume canary release strategy will be adopted. Specifically: In the first phase, the new metadata constraint rules will be applied to 1% of the input data stream or a designated test business line, with real-time monitoring of the false positive and false negative rates. If the monitoring in the first phase shows that the false positive rate increases more than a preset threshold (e.g., 5%) after applying the new metadata constraint rules, the canary release will be paused and an alarm will be triggered. If the monitoring in the first phase shows that the false positive rate increases less than the preset threshold (e.g., 5%), the canary release ratio will gradually increase (e.g., from 1% to 10%, then to 50%, and finally to 100%). If abnormal fluctuations in the false positive or false negative rate are detected at any stage of the canary release process, the system will automatically roll back to the previous stable version and notify manual intervention for investigation. This canary release strategy ensures that the new metadata constraint rules are fully validated before full implementation, effectively avoiding production incidents caused by rule logic defects. Furthermore, after all the new metadata constraint rules have taken effect, the system continuously monitors the false alarm rate and the false negative rate to ensure the stability of the rules.

[0092] Through the above embodiments, a leap from "manual static maintenance" to "automatic dynamic correction" has been achieved, effectively solving the problems of outdated metadata and quality misjudgment caused by business rule drift. It greatly shortens the drift detection response time, reduces the false alarm rate of data quality and the manpower cost of rule maintenance, and ensures the consistency of rule updates and business continuity in a distributed environment by means of a transactional canary release mechanism.

[0093] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0094] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / random access memory (RAM), magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0095] According to another aspect of the embodiments of this application, a business metadata constraint rule drift correction apparatus for implementing the business metadata constraint rule drift correction method provided in the above embodiments is also provided; details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0096] Figure 5 This is a structural block diagram of an optional business metadata constraint rule drift correction device according to an embodiment of this application, such as... Figure 5 As shown, the business metadata constraint rule drift correction device includes at least:

[0097] The acquisition module 52 is used to acquire multiple business data of the target business within a preset time period, as well as the metadata constraint rules of multiple business fields in each business data.

[0098] The determination module 54 is used to determine the real data distribution feature vector and the benchmark data distribution feature vector of each business field based on the metadata constraint rules and multiple business data, and to determine the constraint confidence level corresponding to the metadata constraint rules of each business field based on the real data distribution feature vector and the benchmark data feature vector.

[0099] The correction module 56 is used to correct the target metadata constraint rules of the target business field based on the real data distribution feature vector of the target business field when the constraint confidence of the target metadata constraint rules of the target business field is lower than a preset threshold.

[0100] In an exemplary embodiment, the metadata constraint rules include at least a value range; the aforementioned apparatus is further configured to determine the true data distribution feature vector and the baseline data distribution feature vector of each business field based on the metadata constraint rules and multiple business data, using the following method: for each business field, dividing the value range of the current business field into multiple discrete intervals; counting the amount of data in each discrete interval that includes multiple target business data containing the current business field, and determining the true data occurrence probability of each discrete interval by combining the total amount of data in the multiple target business data, summarizing the true data occurrence probabilities of multiple discrete intervals to obtain the true data distribution feature vector of the current business field; determining the theoretical data occurrence probability of each discrete interval based on the value range of the current business field and the preset theoretical distribution type corresponding to the metadata constraint rules of the current business field, summarizing the theoretical data occurrence probabilities of multiple discrete intervals to obtain the baseline data distribution feature vector of the current business field.

[0101] In an exemplary embodiment, the above-described apparatus is further configured to divide the value range of the current business field into multiple discrete intervals by means of the following method: sorting each value in the value range of the current business field in ascending order, determining a first value at the 25th position and a second value at the 75th position in the sorting result; determining the difference between the second value and the first value, and determining the width of the interval by combining the total number of multiple target business data containing the current business field; dividing the value range of the current business field according to the width of the interval to obtain multiple discrete intervals.

[0102] In an exemplary embodiment, the above-described apparatus is further configured to determine the constraint confidence level corresponding to the metadata constraint rule of each business field based on the real data distribution feature vector and the benchmark data distribution feature vector by the following method: for each business field, determining the distribution difference between the real data distribution feature vector and the benchmark distribution feature vector of the current business field, and determining the constraint confidence level corresponding to the metadata constraint rule of the current business field based on the distribution difference, a preset difference reduction control factor and a preset adjustable drift threshold.

[0103] In an exemplary embodiment, the above-described apparatus is further configured to modify the target metadata constraint rules of the target business field based on the real data distribution feature vector of the target business field using the following method: determining a target subgraph containing the target graph node corresponding to the target business field and the multi-order neighbor nodes of the target graph node within a preset data lineage graph, wherein the types of the multiple graph nodes in the data lineage graph include at least one of the following: business, business field, metadata constraint rules, business data table, and the types of the multiple directed edges in the data lineage graph include at least one of the following: data flow edge, constraint dependency edge, data influence edge; encoding the real data distribution feature vector of the target business field and the target subgraph respectively, and fusing the two encoding results to obtain a joint representation vector; and decoding the joint representation vector to obtain new metadata constraint rules.

[0104] In an exemplary embodiment, the above-described apparatus is further configured to encode the real data distribution feature vector of the target business field and the target subgraph respectively by means of the following method, and to fuse the two encoding results to obtain a joint representation vector, including: processing the real data distribution feature vector of the target business field based on a multilayer perceptron to obtain a first representation vector; processing the target subgraph based on a graph neural network to obtain a second representation vector; mapping the first representation vector and the second representation vector to the same vector space, and processing the mapped first representation vector and the mapped second representation vector based on an attention fusion network to obtain corresponding attention weights, and using the attention weights to perform weighted fusion of the mapped first representation vector and the mapped second representation vector to obtain a joint representation vector.

[0105] In an exemplary embodiment, the above-described apparatus is further configured to decode the joint representation vector to obtain new metadata constraint rules by means of the following method: mapping the joint representation vector to the metadata constraint space corresponding to the target business field to obtain new metadata constraint rules, wherein the new metadata constraint rules include at least one of the following: upper and lower limits of value range, null value rate, and probability distribution vector of enumerated values.

[0106] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0107] According to another aspect of the embodiments of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein the program executes the steps in any of the above method embodiments when it is run.

[0108] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, ROMs, RAMs, portable hard drives, magnetic disks, or optical disks.

[0109] Optionally, in this embodiment, the computer program in the above-described computer program product can be configured to perform the following steps when executed by a processor:

[0110] Step S1: Obtain multiple business data entries for the target business within a preset time period, as well as the metadata constraint rules for multiple business fields within each business data entry;

[0111] Step S2: Based on the metadata constraint rules and multiple business data, determine the real data distribution feature vector and the benchmark data distribution feature vector for each business field, and determine the constraint confidence level corresponding to the metadata constraint rules for each business field based on the real data distribution feature vector and the benchmark data distribution feature vector.

[0112] Step S3: If the constraint confidence of the target metadata constraint rule of the target business field is lower than the preset threshold, the target metadata constraint rule of the target business field is modified according to the real data distribution feature vector of the target business field.

[0113] According to another aspect of the embodiments of this application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor is configured to perform the steps of any of the method embodiments described above via the computer program. In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0114] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0115] According to another aspect of the embodiments of this application, a computer program product is also provided, the computer program product including a computer program / instructions containing program code for performing the method shown in the flowchart.

[0116] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0117] Step S1: Obtain multiple business data entries for the target business within a preset time period, as well as the metadata constraint rules for multiple business fields within each business data entry;

[0118] Step S2: Based on the metadata constraint rules and multiple business data, determine the real data distribution feature vector and the benchmark data distribution feature vector for each business field, and determine the constraint confidence level corresponding to the metadata constraint rules for each business field based on the real data distribution feature vector and the benchmark data distribution feature vector.

[0119] Step S3: If the constraint confidence of the target metadata constraint rule of the target business field is lower than the preset threshold, the target metadata constraint rule of the target business field is modified according to the real data distribution feature vector of the target business field.

[0120] As an alternative implementation, the above-mentioned electronic device may exist in the form of a mobile terminal, a computer terminal, or a similar computing device. Figure 6 A hardware block diagram of a computer terminal for implementing a method to correct drift in business metadata constraint rules is shown. Figure 6 As shown, the computer terminal 60 may include a processor 602, a memory 604 for storing data, and a transmission device 606 for communication functions. The processor 602 may include one or more processors. For ease of explanation, Figure 6 The diagram illustrates multiple processors, including a first processor 602a, a second processor 602b, ..., an nth processor 602n. Processor 602 may be, but is not limited to, a processing device such as a microprocessor (MCU) or a field-programmable gate array (FPGA).

[0121] In addition, the computer terminal 60 may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 6 The structure shown is for illustrative purposes only and does not limit the structure of the computer terminal described above. For example, computer terminal 60 may also include... Figure 6 The more or fewer components shown, or having the same Figure 6 The different configurations shown.

[0122] Processor 602 may include one or more processing units. For example, processor 602 may include processing devices such as central processing unit (CPU), graphics processing unit (GPU), digital signal processing (DSP) chip, microcontroller unit (MCU), field-programmable gate array (FPGA), neural network processing unit (NPU), tensor processing unit (TPU), artificial intelligence (AI) type processor, etc. Different processing units may be independent components or integrated into one or more processors.

[0123] It should be noted that the aforementioned one or more processors 602 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 60. As described in the embodiments of this application, the data processing circuits serve as processor control (e.g., selection of a variable resistor termination path connected to an interface).

[0124] The memory 604 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the business metadata constraint rule drift correction method in the embodiments of this application. The processor 602 executes various functional applications and data processing by running the software programs and modules stored in the memory 604, thereby implementing the aforementioned business metadata constraint rule drift correction method. The memory 604 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 604 may further include memory remotely located relative to the processor 602, and these remote memories can be connected to the computer terminal 60 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0125] The transmission device 606 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 60. In one example, the transmission device 606 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 606 may be a Radio Frequency (RF) module, used for data interaction with external networks or devices via wired and / or wireless network connections.

[0126] The display can be, for example, a liquid crystal display (LCD) or a touchscreen display (also known as a "touchscreen" or "touch display screen"). This LCD allows the user to interact with the user interface of the mobile terminal. Furthermore, the computer terminal 60 can be connected to a cursor control device and a keyboard via an input / output interface. The cursor control device (e.g., a mouse, trackball, or touchpad) is primarily used to transmit directional information and command selections to the terminal and control the movement of the cursor on the display; the keyboard is primarily used to receive letters, numbers, and other control commands input by the user, thereby enabling interaction between the user and the computer terminal 60.

[0127] It should be noted here that, in some optional embodiments, the above... Figure 6 The computer terminal shown may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that... Figure 6 This is only one instance of a specific particular instance, and is intended to illustrate the types of components that may exist in the aforementioned computer terminal.

[0128] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0129] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method for correcting drift in business metadata constraint rules, characterized in that, include: Obtain multiple business data entries for the target business within a preset time period, as well as metadata constraint rules for multiple business fields within each business data entry; Based on the metadata constraint rules and the multiple business data, determine the real data distribution feature vector and the benchmark data distribution feature vector for each business field, and determine the constraint confidence level corresponding to the metadata constraint rule for each business field based on the real data distribution feature vector and the benchmark data distribution feature vector; If the constraint confidence of the target metadata constraint rule of the target business field is lower than a preset threshold, the target metadata constraint rule of the target business field is modified according to the real data distribution feature vector of the target business field.

2. The method according to claim 1, characterized in that, The metadata constraint rules include at least: value range; Based on the metadata constraint rules and the multiple business data, the true data distribution feature vector and the baseline data distribution feature vector for each of the business fields are determined, including: For each of the aforementioned business fields, the value range of the current business field is divided into multiple discrete intervals; The data volume of multiple target business data containing the current business field falling within each discrete interval is counted, and the actual data occurrence probability of each discrete interval is determined by combining the total data volume of the multiple target business data. The actual data occurrence probabilities of the multiple discrete intervals are summarized to obtain the actual data distribution feature vector of the current business field. Based on the value range of the current business field and the preset theoretical distribution type corresponding to the metadata constraint rules of the current business field, the theoretical data occurrence probability of each discrete interval is determined, and the theoretical data occurrence probabilities of the multiple discrete intervals are summarized to obtain the baseline data distribution feature vector of the current business field.

3. The method according to claim 2, characterized in that, Divide the value range of the current business field into multiple discrete intervals, including: Sort the values ​​in the current business field in ascending order, and determine the first value at the 25th percentile and the second value at the 75th percentile in the sorted result; Determine the difference between the second value and the first value, and combine this with the total number of target business data entries containing the current business field to determine the width of the partitioned interval; The value range of the current business field is divided according to the width of the defined interval to obtain the multiple discrete intervals.

4. The method according to claim 1, characterized in that, Determining the constraint confidence level corresponding to the metadata constraint rule for each business field based on the real data distribution feature vector and the benchmark data distribution feature vector includes: For each of the business fields, the distribution difference between the real data distribution feature vector and the benchmark distribution feature of the current business field is determined, and the constraint confidence level corresponding to the metadata constraint rule of the current business field is determined based on the distribution difference, the preset difference reduction control factor and the preset adjustable drift threshold.

5. The method according to claim 1, characterized in that, The target metadata constraint rules for the target business field are modified based on the actual data distribution feature vector of the target business field, including: Within a preset data lineage graph, a target subgraph is determined that includes the target graph node corresponding to the target business field and the multi-order neighbor nodes of the target graph node. The types of the multiple graph nodes in the data lineage graph include at least one of the following: business, business field, metadata constraint rule, business data table. The types of the multiple directed edges in the data lineage graph include at least one of the following: data flow edge, constraint dependency edge, data influence edge. The real data distribution feature vector and target subgraph of the target business field are encoded respectively, and the two encoding results are fused to obtain a joint representation vector; The joint representation vector is decoded to obtain new metadata constraint rules.

6. The method according to claim 5, characterized in that, The real data distribution feature vector and target subgraph of the target business field are encoded separately, and the two encoding results are fused to obtain a joint representation vector, including: Based on the multilayer perceptron processing of the real data distribution feature vector of the target business field, a first representation vector is obtained; The target subgraph is processed using a graph neural network to obtain a second representation vector. The first representation vector and the second representation vector are mapped to the same vector space, and the mapped first representation vector and the mapped second representation vector are processed based on the attention fusion network to obtain the corresponding attention weights. The mapped first representation vector and the mapped second representation vector are then weighted and fused using the attention weights to obtain the joint representation vector.

7. The method according to claim 5, characterized in that, Decoding the joint representation vector yields new metadata constraint rules, including: The joint representation vector is mapped to the metadata constraint space corresponding to the target business field to obtain the new metadata constraint rule, wherein the new metadata constraint rule includes at least one of the following: upper and lower limits of value range, null value rate, and probability distribution vector of enumerated value.

8. A device for correcting drift in business metadata constraint rules, characterized in that, include: The acquisition module is used to acquire multiple pieces of business data of the target business within a preset time period, as well as the metadata constraint rules of multiple business fields in each piece of business data; The determination module is used to determine the real data distribution feature vector and the benchmark data distribution feature vector of each of the business fields based on the metadata constraint rules and the multiple business data, and to determine the constraint confidence level corresponding to the metadata constraint rule of each business field based on the real data distribution feature vector and the benchmark data distribution feature vector. The correction module is used to correct the target metadata constraint rules of the target business field based on the real data distribution feature vector of the target business field when the constraint confidence of the target metadata constraint rules of the target business field is lower than a preset threshold.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the device containing the computer-readable storage medium executes the business metadata constraint rule drift correction method according to any one of claims 1 to 7 by running the computer program.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the business metadata constraint rule drift correction method according to any one of claims 1 to 7 through the computer program.

Images