A Multi-Source Data Quality Assessment Method Based on Configurable Rules and Adaptive Sampling

Abstract

This invention relates to the field of computer data processing technology, and in particular to a multi-source data quality assessment method based on configurable rules and adaptive sampling, comprising the following steps: S1, constructing a configurable rule base containing various data quality verification logics, and establishing a dynamic reputation model that maps data source identifiers to historical quality scores, serving as a quantitative benchmark for subsequent sampling decisions. In this invention, by constructing a dynamic reputation model to quantitatively characterize the historical quality status of each heterogeneous data source, and calculating the corresponding dynamic sampling ratio in real time, the method can automatically increase the sampling density or even trigger full-scale detection for data sources with low reputation or abnormal tendencies, based on the ability to sensitively perceive real-time fluctuations in data source quality. This overcomes the shortcomings of existing static sampling strategies that cannot differentiate monitoring of specific risk sources, significantly reducing the missed detection rate of key quality issues and improving overall assessment efficiency under limited assessment cost constraints.

Description

Technical Field

[0001] This invention relates to the field of computer data processing technology, and in particular to a method for assessing the quality of multi-source data based on configurable rules and adaptive sampling. Background Technology

[0002] With the widespread application of big data technology, real-time access and processing of multi-source heterogeneous data has become a key link in enterprises building a digital foundation. Ensuring data quality is crucial for maintaining the accuracy of downstream business decisions. However, faced with massive and high-speed data flows, due to physical constraints such as computing resource costs and system throughput latency, it is often difficult to implement full real-time verification during the data integration stage. Therefore, adopting sampling detection methods based on statistical principles to obtain a general overview of data quality under controllable resource overhead has become a common practice in the current industry for large-scale data quality monitoring.

[0003] Existing static sampling strategies use a fixed proportion to perform indiscriminate testing on all data sources. This makes it difficult to dynamically adjust the sampling granularity according to the real-time quality fluctuations of the data sources. As a result, high-risk data sources are frequently missed due to insufficient sampling, while stable data sources are wasted resources due to over-testing, which seriously reduces the overall evaluation efficiency. Summary of the Invention

[0004] To overcome the above shortcomings, this invention provides a multi-source data quality assessment method based on configurable rules and adaptive sampling. It aims to improve the problem that existing static sampling strategies use a fixed proportion to perform indiscriminate testing on all data sources, making it difficult to dynamically adjust the sampling granularity according to the real-time quality fluctuations of the data sources.

[0005] This invention provides the following technical solution: a multi-source data quality assessment method based on configurable rules and adaptive sampling, comprising the following steps:

[0006] S1. Construct a configurable rule base containing various data quality verification logics, and establish a dynamic reputation model that maps data source identifiers to historical quality scores as a quantitative benchmark for subsequent sampling decisions.

[0007] S2. Access the multi-source data stream to be evaluated, extract the data distribution characteristics of the current batch in real time, and retrieve the corresponding historical quality score from the dynamic reputation model based on the data source identifier;

[0008] S3. Input the retrieved historical quality score and the data distribution characteristics into the preset sampling rate calculation model, and jointly calculate the dynamic sampling ratio of the current batch of data.

[0009] S4. Perform stratified sampling on the multi-source data stream according to the dynamic sampling ratio to extract the target sample set to be detected;

[0010] S5. Call the configurable rule base to perform rule matching and verification on the target sample set, generate anomaly detection results, and calculate the anomaly rate based on the results;

[0011] S6. Based on the anomaly rate, write back and update the historical quality score in the dynamic reputation model, and generate a full detection trigger instruction when the anomaly rate exceeds a preset threshold.

[0012] Preferably, in step S1, constructing a configurable rule base containing multiple data quality verification logics specifically includes:

[0013] A predefined single logical validation rule template is stored in the form of declarative configuration. The single logical validation rule template covers data format validation, value range validation, non-empty logic validation, and business consistency validation.

[0014] Based on the metadata structure of the multi-source data stream, the single logical verification rule template is instantiated and bound to a specific data field to generate a verification rule instance for a specific data object;

[0015] Configure a corresponding severity weight for the verification rule instance, and store the mapping relationship between the verification rule instance and the severity weight in the rule configuration database to form the configurable rule library.

[0016] Preferably, in step S1, establishing a dynamic reputation model that maps data source identifiers to historical quality scores specifically includes:

[0017] Initialize the reputation profile registry and establish a key-value mapping structure with the data source identifier as the primary key and the historical quality score as the status value.

[0018] Assign a preset initial reputation benchmark value to the newly connected data source identifier, and use it as the historical quality score of the data source at the beginning of its life cycle;

[0019] Configure model update parameters and associate them with the reputation profile registry. The model update parameters include at least a time decay factor for reducing the weight of long-term historical data and an error penalty coefficient for quantifying the magnitude of a single abnormal deduction.

[0020] Preferably, in step S2, retrieving the corresponding historical quality score from the dynamic reputation model based on the data source identifier specifically includes:

[0021] Parse the transmission protocol header or metadata information of the multi-source data stream, extract the string used to uniquely identify the data source attribute, and determine it as the data source identifier;

[0022] Using the data source identifier as the retrieval key, a retrieval operation is performed on the key-value database or memory cache storing the dynamic reputation model;

[0023] Determine if the search result exists. If it exists, directly read and output the associated historical quality score. If it does not exist, trigger the initialization program to generate a preset default score and output it as the historical quality score.

[0024] Preferably, in step S3, the joint calculation of the dynamic sampling ratio of the current batch of data specifically includes:

[0025] A preset risk mapping function is invoked to map the historical quality score to a basic sampling ratio, wherein the lower the historical quality score, the higher the basic sampling ratio generated by the mapping.

[0026] Calculate the variance or standard deviation index in the data distribution characteristics to generate a volatility correction coefficient, which is used to characterize the stability of the current data stream;

[0027] The base sampling ratio is weighted and corrected using the volatility correction coefficient to calculate the corrected sampling ratio.

[0028] The modified sampling ratio is compared and truncated with the preset minimum monitoring threshold and maximum load threshold to finally output the dynamic sampling ratio.

[0029] Preferably, in step S4, performing stratified sampling on the multi-source data stream according to the dynamic sampling ratio specifically includes:

[0030] Based on a time window or data batch ID, the multi-source data stream is divided into several consecutive logical hierarchical units.

[0031] For the logical hierarchical unit, the total amount of data within the unit is multiplied by the dynamic sampling ratio to calculate the target number of samples that the unit should extract;

[0032] Using a pseudo-random number generation algorithm or a hash modulo algorithm, random sampling is performed within the logical hierarchical unit until the number of extracted data entries reaches the target sample size, and all extracted data entries are aggregated to form the target sample set.

[0033] Preferably, in step S5, calling the configurable rule base to perform rule matching and verification on the target sample set specifically includes:

[0034] Traverse the sample data in the target sample set to identify its business type and the data fields it contains;

[0035] Based on the business type and data fields, retrieve and load the bound valid rule set from the configurable rule base;

[0036] Each rule in the set of valid rules is applied sequentially to perform a Boolean logic operation on the sample data. If the result of the operation is false, the sample data is determined to be abnormal data.

[0037] Record the detailed information of the abnormal data and the corresponding rule severity weights, generate the anomaly detection results, and calculate the ratio of the total amount of the abnormal data to the total amount of the target sample set as the anomaly rate.

[0038] Preferably, in step S6, updating the historical quality score in the dynamic reputation model based on the anomaly rate specifically includes:

[0039] The historical quality score at the current moment is read from the dynamic reputation model and used as the basis for calculation;

[0040] The anomaly rate is compared with a preset tolerance threshold. If the anomaly rate is higher than the tolerance threshold, a penalty function is called to calculate the reputation deduction value; otherwise, a recovery function is called to calculate the reputation recovery value.

[0041] Perform an arithmetic operation on the base value by subtracting the reputation deduction value or adding the reputation recovery value to generate the latest quality score;

[0042] The latest quality score is written into the dynamic reputation model, overwriting the original historical quality score, thus completing the update of the reputation status of the data source identifier.

[0043] The present invention has the following beneficial effects:

[0044] 1. In this invention, a dynamic reputation model is constructed to quantitatively characterize the historical quality status of each heterogeneous data source, and a matching dynamic sampling ratio is calculated in real time based on this model. On the basis of being able to keenly perceive the real-time fluctuations in the quality of data sources, the sampling density of data sources with low reputation or abnormal tendencies is automatically increased or even full detection is triggered. This overcomes the shortcomings of existing static sampling strategies that cannot perform differentiated monitoring for specific risk sources. Under the constraint of limited assessment costs, the missed detection rate of key quality issues is significantly reduced and the overall assessment efficiency is improved.

[0045] 2. In this invention, a double-buffered queue combined with consistent hashing or random scrambling algorithm is used to realize logical sharding of multi-source data streams and statistically representative random sampling. Combined with a rule verification engine based on memory caching and chain of responsibility mode, the invention solves the problems of concurrency conflicts and processing delays when large-scale data access is carried out while ensuring sample randomness and detection coverage. This significantly improves the execution efficiency from data access to anomaly judgment process.

[0046] 3. In this invention, a closed loop of reputation evolution based on nonlinear reward and punishment logic is established. By performing rapid deduction and circuit breaker protection on data sources with an anomaly rate exceeding the tolerance threshold, data quality fluctuations can be fed back in a timely manner and continuous anomalies can be suppressed. At the same time, a smooth reputation recovery mechanism is used to avoid drastic fluctuations in evaluation results. This effectively solves the problem that it is difficult to balance sudden anomaly response and long-term quality measurement under the traditional static monitoring mode, and promotes the data quality to be maintained at a stable level. Attached Figure Description

[0047] Figure 1 This is a flowchart of the multi-source data quality assessment method based on configurable rules and adaptive sampling proposed in this invention. Detailed Implementation

[0048] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] This invention provides a method for assessing the quality of multi-source data based on configurable rules and adaptive sampling, such as... Figure 1 As shown, it includes the following steps:

[0050] S1. Construct a configurable rule base containing various data quality verification logics, and establish a dynamic reputation model that maps data source identifiers to historical quality scores as a quantitative benchmark for subsequent sampling decisions.

[0051] Furthermore, in step S1, constructing a configurable rule base containing various data quality verification logics specifically includes:

[0052] A predefined single logical validation rule template is stored in the form of declarative configuration. The single logical validation rule template covers data format validation, value range validation, non-empty logic validation and business consistency validation.

[0053] Based on the metadata structure of multi-source data streams, a single logical verification rule template is instantiated and bound to a specific data field to generate a verification rule instance for a specific data object;

[0054] Configure the corresponding severity weight for each validation rule instance, and store the mapping relationship between validation rule instances and severity weights in the rule configuration database to form a configurable rule library.

[0055] Furthermore, in step S1, establishing a dynamic reputation model that maps data source identifiers to historical quality scores specifically includes:

[0056] Initialize the reputation profile registry and establish a key-value mapping structure with the data source identifier as the primary key and historical quality scores as status values.

[0057] Assign a preset initial reputation benchmark value to the identifier of the newly connected data source, and use it as the historical quality score of the data source at the beginning of its life cycle;

[0058] Configure model update parameters and associate them with the reputation profile registry. The model update parameters should include at least a time decay factor for reducing the weight of long-term historical data and an error penalty coefficient for quantifying the magnitude of a single abnormal deduction.

[0059] Specifically, to build a configurable rule base containing various data quality verification logics, system developers or administrators first predefine a set of single-logic verification rule templates stored in a declarative format such as JSON or YAML in the configuration center. These templates decouple the verification logic from the business code and cover basic types such as data format verification (e.g., verifying whether a date string conforms to a specific year, month, and day standard), value range verification (e.g., determining whether a value falls within a preset closed interval), non-empty logic verification (e.g., checking whether key fields have empty values), and business consistency verification (e.g., comparing whether the end timestamp is later than the start timestamp). Each rule template is defined as a data structure containing a unique rule code, algorithm type, and parameter placeholders. Subsequently, the system connects to the metadata repository of the data governance platform to read the metadata definitions of the multi-source data streams to be evaluated, such as reading data... Based on the definition statements of the database tables or the structure files of the message queue, for a specific data object in the data stream, such as the transaction amount field in the order table, the system calls the corresponding value range validation template for instantiation, takes the transaction amount field as the input object, and fills the parameter placeholders with specific boundary values ​​allowed by the business, such as setting the minimum value to zero and the maximum value to one million, thereby generating a specific validation rule instance. In this process, the system also configures a corresponding severity weight for each generated validation rule instance. This weight is a floating-point number between zero and one, used to quantify the impact on downstream business when the field has an error. Finally, the system combines the field's unique identifier, the configuration parameters of the validation rule instance, and the corresponding severity weight into a configuration record, and persists it to the rule configuration database in a key-value pair manner, completing the construction of the configurable rule base.

[0060] Simultaneously, to establish a quantitative benchmark for subsequent sampling decisions, the system builds a dynamic reputation model that maps data source identifiers to historical quality scores in a key-value database or memory cache. This process first involves the initialization of the reputation profile registry. The system constructs an efficient key-value mapping structure, where the key is used to store the unique identifier of the data source, such as an IP address, service name, or topic ID, and the value is used to store the current reputation status data of the data source. For each newly connected data source identifier, the system automatically triggers a registration event and assigns it a preset initial reputation benchmark value. This value is usually set to the model's maximum score or a high trust value, serving as the historical quality score of the data source at the beginning of its lifecycle. This ensures that existing high-quality data sources are not misjudged, while allowing new data sources to quickly enter the normal sampling state.

[0061] Building upon this, the system further configures model update parameters to control the dynamic evolution of scores, and associates these parameters with configuration attributes in the reputation profile registry as global constants or specific data sources. To accurately quantify reputation decay over time and the severity of penalties for anomalies, the system defines a set of model parameters, denoted as follows: The specific parameters are defined as follows:

[0062] ;

[0063] in:

[0064] This represents the time decay factor, and its value range satisfies... This parameter is used to reduce the weight of the influence of distant historical data on the current credit score in subsequent calculations, reflecting the timeliness of data quality, that is, the older the historical record, the smaller its contribution to the current credit score.

[0065] This represents the error penalty coefficient, and its value range satisfies... This parameter is used to quantify the extent to which the reputation score is deducted when abnormal data is detected in a single instance.

[0066] Meanwhile, the initialized reputation model state set At the initial moment It can be represented as:

[0067] ;

[0068] in:

[0069] Representing the Each access data source identifier is used to uniquely distinguish different data input streams;

[0070] This represents a preset initial reputation benchmark value, for example, 0.8, which serves as a trust metric for the starting point of the assessment.

[0071] This represents the total number of currently connected data sources.

[0072] By establishing a quantitative reputation model that includes decay and penalty mechanisms, an accurate mathematical benchmark is provided for subsequent reputation-driven adaptive stratified sampling. This ensures that evaluation resources can be tilted towards low-reputation data sources, thereby maximizing computational efficiency while ensuring detection coverage.

[0073] S2. Access the multi-source data stream to be evaluated, extract the data distribution characteristics of the current batch in real time, and retrieve the corresponding historical quality score from the dynamic reputation model based on the data source identifier;

[0074] Furthermore, in step S2, retrieving the corresponding historical quality score from the dynamic reputation model based on the data source identifier specifically includes:

[0075] Parse the transport protocol header or metadata information of multi-source data streams, extract the string used to uniquely identify the data source attribute, and determine it as the data source identifier;

[0076] Use the data source identifier as the retrieval key to perform retrieval operations on the key-value database or memory cache that stores the dynamic reputation model;

[0077] Determine if the search results exist. If they do, directly read and output the associated historical quality score. If they do not exist, trigger the initialization process to generate a preset default score and output it as the historical quality score.

[0078] Specifically, firstly, a connection is established with the multi-source data stream transmission channel. A data consumption group is created using a streaming computing engine to capture the data stream to be detected in real time. As the data stream passes through the memory computing node, the system does not immediately perform line-by-line verification. Instead, it performs statistical analysis on the currently defined micro-batch of data. The system uses a sliding window algorithm to calculate the statistical moments of the numerical data in the current batch, including but not limited to mean, variance, and skewness. These statistical indicators are then aggregated to generate the data distribution feature vector for the current batch. Let this data distribution feature vector be... Its formal definition is as follows:

[0079] ;

[0080] in:

[0081] This represents the arithmetic mean of the current batch of data, used to reflect the central tendency of the data;

[0082] This represents the standard deviation of the current batch of data, used to quantify the dispersion of the data.

[0083] This represents the skewness coefficient of the current batch of data, used to characterize the asymmetry of the data distribution.

[0084] Meanwhile, to obtain the historical reputation status corresponding to this batch of data, the system executes retrieval logic based on the data source identifier. First, the system parses the transmission protocol headers or accompanying metadata information of the multi-source data streams. The parser scans predefined protocol fields, such as the specific source field in the HTTP request header or the publisher tag in the message queue metadata attributes, and extracts the string used to uniquely identify the data source attribute. The system determines this string as the data source identifier for this query, denoted as [identifier]. .

[0085] Subsequently, the system uses the parsed data... As a retrieval key, a fast retrieval operation is performed on the key-value database storing the dynamic reputation model or the high-performance memory cache. The system sends an existence query command to the database to check whether the key-value already exists in the key set of the reputation profile registry. Based on the Boolean state of the retrieval result, the system executes branch processing logic: if the retrieval result shows that the key-value exists, the system directly reads and outputs the value associated with the key-value, that is, the effective historical quality score accumulated by the data source; if the retrieval result shows that the key-value does not exist, it means that the data stream is a new data source that is being connected to the system for the first time. At this time, the system immediately triggers the initialization program, generates a preset default score as the initial state of the data source, and outputs it as the historical quality score. At the same time, the registration of new users is completed asynchronously in the background.

[0086] The above retrieval and decision-making logic can be implemented using piecewise functions. It is expressed as follows:

[0087] ;

[0088] in:

[0089] This represents the historical quality score of the final output for this data source;

[0090] This represents the key-value mapping table structure for storing the dynamic reputation model;

[0091] Represents the key in the mapping table The corresponding stored value;

[0092] This represents the set of all registered data source identifiers in the current reputation model;

[0093] This indicates the preset default score, such as the initial trust value of 0.8 set by the system.

[0094] This facilitates accurate backtracking of the historical performance of existing data sources, while also enabling the automatic acceptance and initialization of new incremental data sources, providing indispensable input parameters for dynamically adjusting the sampling intensity based on credit rating in subsequent steps.

[0095] S3. Input the retrieved historical quality scores and data distribution characteristics into the preset sampling rate calculation model, and jointly calculate the dynamic sampling ratio of the current batch of data.

[0096] Furthermore, in step S3, the joint calculation of the dynamic sampling ratio for the current batch of data specifically includes:

[0097] Call the preset risk mapping function to map the historical quality score to the basic sampling ratio, where the lower the historical quality score, the higher the basic sampling ratio generated by the mapping.

[0098] Calculate the variance or standard deviation indices in the data distribution characteristics to generate volatility correction coefficients, which are used to characterize the stability of the current data stream;

[0099] The base sampling proportion is weighted and corrected using a volatility correction factor to obtain the corrected sampling proportion.

[0100] The corrected sampling ratio is compared and truncated with the preset minimum monitoring threshold and maximum load threshold to finally output the dynamic sampling ratio.

[0101] Specifically, the system first executes the risk mapping logic, calling a preset risk mapping function to perform a reverse mapping using the input historical quality score as the core independent variable to determine the base sampling ratio. The mathematical model of this function employs exponential decay inversion logic, aiming to achieve non-linear penalty for low-reputation data sources; that is, the lower the reputation score, the steeper the slope of the sampling rate increase. Let the input historical quality score be... Its value range has been normalized to a closed interval between zero and one. Let the theoretical maximum basic sampling rate defined by the system be... For example, if the value is 1.0, let the curvature adjustment factor of the mapping function be... This parameter determines the curvature of the penalty curve; in this embodiment, it is set to 2.0 to enhance sensitivity to low-scoring users. The base sampling ratio... The calculation formula is defined as follows:

[0102] ;

[0103] in:

[0104] This represents the initial theoretical sampling ratio derived from historical reputation, and is a double-precision floating-point number.

[0105] This represents the historical quality score obtained in step S2, which indicates the creditworthiness of the data source.

[0106] This represents an exponentially increasing sensitivity coefficient, used to accelerate the increase in sampling rates in low-reputation segments;

[0107] This represents the base scaling factor, used to limit the numerical magnitude of the base mapping.

[0108] While calculating the basic sampling ratio, the system analyzes the data distribution characteristics of the current batch and extracts the variance or standard deviation to assess the internal stability of the data stream. To eliminate absolute value differences caused by data with different dimensions (such as monetary value and age), the system uses logarithmic smoothing to calculate the volatility correction coefficient. Let the extracted standard deviation of the current batch data be... The system's preset fluctuation sensitivity constant is For example, a value of 0.05 represents the volatility correction coefficient. The calculation formula is as follows:

[0109] ;

[0110] in:

[0111] This represents a dimensionless volatility correction coefficient, with an initial value of 1 that monotonically increases as volatility increases.

[0112] This represents the standard deviation of the current data batch. Please ensure that it is a non-negative real number before inputting it.

[0113] This represents the volatility sensitivity gain factor, used to control the amplification weight of the standard deviation on the final sampling rate;

[0114] This represents a logarithmic operation with the natural constant e as the base, used to suppress excessive corrections for data with extremely large variances.

[0115] Subsequently, the system performs weighted correction and boundary constraint operations. It uses the calculated volatility correction coefficient to multiply and weight the base sampling ratio to obtain the corrected sampling ratio. Then, to ensure system stability, this ratio must be constrained within a controllable range; therefore, the system presets a minimum monitoring threshold. For example, 5% is set to prevent missed detections; a maximum load threshold is also preset. For example, 40% to prevent overload, the final output dynamic sampling ratio. The specific calculation logic is as follows, determined by nested extremum functions:

[0116] ;

[0117] in:

[0118] This represents the dynamic sampling ratio that is ultimately output to the stratified sampling module, and its value is strictly within... Within the range;

[0119] This indicates a hard constraint on the lower limit of the sampling rate set by the system.

[0120] This indicates a hard constraint on the upper limit of the sampling rate set by the system.

[0121] The function is used to perform upper limit truncation by taking the smaller of the corrected result and the upper limit value;

[0122] The function is used to take the larger of the above result and the lower limit value, and perform a lower limit guarantee.

[0123] The algorithm achieves key monitoring of low-reputation sources through exponential functions, and is compatible with data streams of different fluctuation ranges through logarithmic correction mechanisms. Finally, the boundary truncation ensures from an engineering perspective that the algorithm output is always within the safe zone of system resources, achieving a perfect balance between accurate monitoring and system performance.

[0124] S4. Perform stratified sampling on the multi-source data stream according to the dynamic sampling ratio to extract the target sample set to be detected.

[0125] Furthermore, in step S4, performing stratified sampling on the multi-source data stream according to the dynamic sampling ratio specifically includes:

[0126] Based on time windows or data batch IDs, multi-source data streams are divided into several consecutive logical hierarchical units;

[0127] For logically hierarchical units, the total amount of data within the unit is multiplied by the dynamic sampling ratio to calculate the target number of samples that should be extracted from the unit.

[0128] Using pseudo-random number generation algorithms or hash modulo algorithms, random sampling is performed within the logical hierarchical unit until the number of extracted data entries reaches the target sample size, and all extracted data entries are aggregated to form the target sample set.

[0129] Specifically, the streaming data fragmentation processing program is first started. This program uses a double-buffered queue mechanism in memory to achieve lock-free, high-throughput data access. The system initializes two memory blocks of fixed capacity, marked as the active write area and the frozen processing area, respectively. The system employs a dual-constraint triggering strategy to divide the logical hierarchical units. The first constraint is a time window constraint. The system starts a high-precision timer, setting a time threshold of [value missing]. For example, 500 milliseconds; the second constraint is capacity counting, where the system maintains an atomic counter with a set threshold. For example, with 5,000 records, when multiple data streams enter the system, the data is first written to the active write area. Each time a record is written, the atomic counter increments by one. Once the current time has elapsed... Or the value of the atomic counter reaches The system immediately triggers a buffer swap operation, first locking the active write area and marking its state as a frozen processing area. Simultaneously, it activates another previously idle memory area as a new active write area, resetting the counter and timer. This achieves seamless segmentation of the data stream. The frozen memory area constitutes an independent logical hierarchical unit. Let the total amount of data actually captured within this unit be... This value precisely corresponds to the total number of physically existing candidate samples within the current micro-batch, providing a definite base for subsequent calculations.

[0130] Subsequently, for this logical hierarchical unit, the system calculates the number of target samples that the unit should extract based on the dynamic sampling ratio output in step S3. To ensure that the calculation result is an executable integer, the system uses an algorithm with a rounding function to convert the product of the total amount of data in the unit and the dynamic sampling ratio into the smallest integer. This ensures that even with an extremely low sampling ratio, at least one sample can be extracted for monitoring. Let the input dynamic sampling ratio be... The number of target samples that the logical hierarchical unit should extract. The calculation formula is defined as follows:

[0131] ;

[0132] in:

[0133] This indicates the total number of samples that need to be extracted in the current batch, and its value is at least 1.

[0134] This indicates the total number of original data entries within the current logical hierarchical unit;

[0135] This represents the dynamic sampling ratio output by step S3, and its value ranges from 0 to 1.

[0136] This indicates an up-rounding operation, ensuring that even with extremely low sampling rates, the minimum number of monitored samples can still be retained.

[0137] After determining the specific sampling size, the system selects to execute either an in-memory scrambling algorithm or a distributed hash sorting algorithm, depending on the operating environment, to ensure the randomness and fairness of the sampling. In a single-machine memory scenario, the system uses an improved Fischer-Yets shuffle algorithm to perform sampling. The system first creates a shuffle algorithm in memory with a length of... An integer index array, in its initial state, the array's index... Store values ​​in each location. The system sets an iteration variable. Starting from zero and increasing until reaching... Up to this point, in each iteration, the system uses the Mason twitch algorithm to generate a pseudo-random integer. The range of values ​​for this random number is strictly limited to a closed interval. Subsequently, the system performs an atomic swap operation, changing the position in the index array. The value and position The values ​​are swapped, and when the loop ends, the first element in the index array is replaced. The value stored at each location is the physical offset of the selected data in the original buffer. The system directly performs a memory copy based on these offsets to extract the corresponding data object.

[0138] In distributed or streaming computing scenarios, the system employs a consistent hashing selection algorithm based on MurmurHash. The system iterates through each data record in the logical hierarchical unit, extracts its unique primary key or all binary content as input, and calls the MurmurHash3 algorithm to calculate and generate a 32-bit or 64-bit hash integer value. The system maintains a fixed capacity. The system uses a min-heap structure, where for each piece of data, the system compares its hash value. If the heap is not full, the data is directly inserted into the heap using the hash value of the top element; otherwise, the data is not. If the hash value is greater than that of the top element of the heap, then ignore the data. If the heap is full and the current data is... If the hash value is less than that of the top element of the heap, then the top element is popped and the current data is inserted into the heap. After traversal, the data is retained in the min-heap. Each data entry is a random sample with the most uniform distribution in the hash space. Finally, the system will aggregate all data entries extracted by any of the above methods to construct an independent data packet containing batch header information and sample body, i.e., the target sample set, and then serialize it and pass it to the subsequent rule verification module.

[0139] While ensuring the statistical representativeness of the samples, the computational load of subsequent detection steps is strictly constrained, enabling the system to maintain targeted monitoring capabilities even under high-traffic, high-concurrency scenarios, thus avoiding the risk of resource collapse caused by full-scale detection.

[0140] S5. Call the configurable rule base to perform rule matching and verification on the target sample set, generate anomaly detection results, and calculate the anomaly rate based on the results;

[0141] Furthermore, in step S5, the specific steps of calling the configurable rule base to perform rule matching and verification on the target sample set include:

[0142] Traverse the sample data in the target sample set to identify its business type and the data fields it contains;

[0143] Based on the business type and data fields, retrieve and load the bound valid rule sets from the configurable rule base;

[0144] Each rule in the valid rule set is applied sequentially to perform Boolean logic operations on the sample data. If the result of the operation is false, the sample data is determined to be abnormal data.

[0145] Record detailed information about abnormal data and the corresponding rule severity weights, generate anomaly detection results, and calculate the ratio of the total amount of abnormal data to the total amount of the target sample set as the anomaly rate.

[0146] Specifically, the system first deserializes the input target sample set, converting it into a list of data objects in memory. Then, the system creates a thread-safe context container and initializes a concurrent list for storing exception details and an atomic counter for counting the number of exception samples. The system then starts an iterator to traverse each sample data in the target sample set. For the currently traversed single data object, the system uses reflection or a path resolver to dynamically read its metadata attributes, accurately identifying its business type identifier and the list of all data field names contained in the object. Based on the combination key logic of the business type identifier and field names, the system constructs a unique index key and initiates a retrieval request to the rule execution container residing in the local cache. This container preloads a valid set of rules that has been compiled into executable closures or expression trees, avoiding the performance loss caused by parsing rule strings at runtime.

[0147] Subsequently, the system enters the rule execution phase, sequentially injecting sample data and its corresponding field values ​​into the context variables of the rule execution engine. The system employs a chain-of-responsibility design pattern to sequentially activate each rule validator in the valid rule set. Internally, each validator calls a high-performance expression engine to execute predefined Boolean logic operations. For example, for a value range validation rule, the engine substitutes the actual value of the current field into the variable for evaluation. If the Boolean operation result is true, the chain continues execution; if the result is false, the system immediately triggers a short-circuit handling mechanism, determining the sample data as anomalous and terminating the execution of subsequent rules to conserve computing power. After capturing an anomalous state, the system, based on the currently triggered rule... The system encodes the corresponding rule severity weights from the rule metadata and instantiates a standardized anomaly object. The data structure of this object includes the unique identifier of the sample that caused the anomaly, the code of the violated rule, the captured original anomaly value, and the severity weight configured for that rule. The system appends this object to the anomaly detection result list as the basis for subsequent root cause analysis. At the same time, the system calls the auto-increment method of the atomic counter to count the samples confirmed as anomalies. To ensure the rigor of the statistical caliber, the system adopts a deduplication counting logic, that is, for the same sample object, no matter how many rules it triggers, the counter is only incremented once, indicating that the sample as a whole is a defective product.

[0148] Finally, after completing the traversal and verification of the entire target sample set, the system calculates the anomaly rate of the current batch based on the statistical results. This indicator objectively reflects the quality level of the data stream within the current time window. Let the total number of samples in the target sample set be... This value is obtained directly by querying the size attribute of the set; let the number of unique samples that are determined to be abnormal be... This value is determined by the final value of the atomic counter, and the anomaly rate. The calculation formula is defined as follows:

[0149] ;

[0150] in:

[0151] This represents the calculated anomaly rate, a double-precision floating-point number, rounded to two decimal places.

[0152] This represents the total number of abnormal data samples in the target sample set that triggered at least one validation rule, and is a non-negative integer.

[0153] This indicates the total number of original data entries contained in the target sample set extracted in step S4, and this value is greater than zero.

[0154] Abstract business rules are transformed into specific memory instruction execution processes. By using caching technology and an expression engine, the contradiction between rule dynamism and execution efficiency is resolved. This not only generates micro-diagnostic reports containing detailed weight information, but also outputs accurate macro-anomaly rate statistics, providing a unique quantitative input for subsequent updates to the dynamic reputation model.

[0155] S6. Write back and update the historical quality score in the dynamic reputation model based on the anomaly rate, and generate a full detection trigger instruction when the anomaly rate exceeds a preset threshold.

[0156] Furthermore, in step S6, the process of writing back and updating the historical quality score in the dynamic reputation model based on the anomaly rate specifically includes:

[0157] The historical quality score at the current moment is read from the dynamic reputation model and used as the basis for calculation;

[0158] The anomaly rate is compared with a preset tolerance threshold. If the anomaly rate is higher than the tolerance threshold, the penalty function is called to calculate the reputation deduction value; otherwise, the recovery function is called to calculate the reputation recovery value.

[0159] Perform arithmetic operations on the base value, subtracting the reputation deduction or adding the reputation recovery value, to generate the latest quality score;

[0160] The latest quality score is written into the dynamic reputation model, overwriting the original historical quality scores, thus completing the update of the reputation status of the data source.

[0161] Specifically, the system first initiates a reputation update transaction, establishing a high-speed connection to the key-value database storing the dynamic reputation model. Using the currently processed data source identifier as a unique retrieval key, the system reads the historical quality score of that data source at the current moment from the database. To ensure data consistency in a distributed environment, the system employs a version-controlled reading mechanism, temporarily storing the read score value and its corresponding version number in local memory, and defining this score value as the base value for calculation. Let this base value be... Its value ranges from zero to one.

[0162] Subsequently, the system executes the core reputation evolution algorithm, comparing the current batch anomaly rate output in step S5 with the system's preset tolerance threshold. The system has a preset tolerance threshold. For example, if the abnormality rate of the current batch is set to 5%, If the value exceeds the tolerance threshold, it indicates a significant deterioration in the quality of the data source. The system then calls a preset penalty function to calculate the reputation deduction value. The penalty function uses an exponential growth model to achieve a rapid response to high-quality incidents. The system reads the configured error penalty coefficient. Calculate credit score deduction Its calculation formula is defined as follows:

[0163] ;

[0164] in:

[0165] This indicates the amount of reputation points that should be deducted in this update;

[0166] This represents the error penalty coefficient, used to adjust the baseline range of the penalty;

[0167] This indicates the actual anomaly rate of the current batch;

[0168] This indicates the upper limit of the anomaly rate that the system is allowed to tolerate;

[0169] It is a natural constant used to produce a more severe non-linear deduction effect as the anomaly rate exceeds the threshold.

[0170] Conversely, if the anomaly rate is lower than or equal to the tolerance threshold, it indicates that the data source is operating stably. The system calls the recovery function to calculate the reputation recovery value. To prevent inflated reputation, the recovery process is designed as a linear incremental model, and the system introduces a preset recovery factor. Calculate the credit recovery value The calculation formula is as follows:

[0171] ;

[0172] in:

[0173] This indicates the amount of reputation points that should be awarded in this update;

[0174] This represents the recovery factor, which is usually set to a very small positive number to ensure that reputation building is a long-term process.

[0175] After calculating the change value, the system performs an arithmetic update on the base value. If the system is in a penalty state, the penalty value is subtracted from the base value; if the system is in a recovery state, the recovery value is added to the base value. The system performs boundary truncation on the calculation result to ensure the final result is accurate. It falls strictly within the closed interval of the valid reputation range defined by the system.

[0176] Next, the system performs a status write-back operation, constructing a status containing the latest quality score. The update request uses comparison and exchange atomic instructions to write the new score into the dynamic reputation model, overwriting the original historical quality score, thereby completing the real-time refresh of the reputation of the data source.

[0177] Specifically, in the above comparison process, once the anomaly rate is determined... Above the tolerance threshold In addition to performing the deduction operation, the system will also instantiate a full detection trigger command object. This command is a control message containing the target data source identifier, command type identifier, and effective time window parameters. The system will immediately send the command to the front-end sampling controller through the control bus. After receiving the command, the sampling controller will forcibly bypass the sampling rate calculation logic of step S3 within the specified effective time window, and lock the sampling ratio of the data source to 100%, thereby achieving immediate circuit breaker defense.

[0178] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-source data quality assessment method based on configurable rules and adaptive sampling, characterized in that, Includes the following steps: S1. Construct a configurable rule base containing various data quality verification logics, and establish a dynamic reputation model that maps data source identifiers to historical quality scores as a quantitative benchmark for subsequent sampling decisions. S2. Access the multi-source data stream to be evaluated, extract the data distribution characteristics of the current batch in real time, and retrieve the corresponding historical quality score from the dynamic reputation model based on the data source identifier; S3. Input the retrieved historical quality score and the data distribution characteristics into the preset sampling rate calculation model, and jointly calculate the dynamic sampling ratio of the current batch of data. S4. Perform stratified sampling on the multi-source data stream according to the dynamic sampling ratio to extract the target sample set to be detected; S5. Call the configurable rule base to perform rule matching and verification on the target sample set, generate anomaly detection results, and calculate the anomaly rate based on the results; S6. Based on the anomaly rate, write back and update the historical quality score in the dynamic reputation model, and generate a full detection trigger instruction when the anomaly rate exceeds a preset threshold.

2. The multi-source data quality assessment method based on configurable rules and adaptive sampling according to claim 1, characterized in that, In step S1, constructing a configurable rule base containing various data quality verification logics specifically includes: A predefined single logical validation rule template is stored in the form of declarative configuration. The single logical validation rule template covers data format validation, value range validation, non-empty logic validation, and business consistency validation. Based on the metadata structure of the multi-source data stream, the single logical verification rule template is instantiated and bound to a specific data field to generate a verification rule instance for a specific data object; Configure a corresponding severity weight for the verification rule instance, and store the mapping relationship between the verification rule instance and the severity weight in the rule configuration database to form the configurable rule library.

3. The multi-source data quality assessment method based on configurable rules and adaptive sampling according to claim 1, characterized in that, In step S1, establishing a dynamic reputation model that maps data source identifiers to historical quality scores specifically includes: Initialize the reputation profile registry and establish a key-value mapping structure with the data source identifier as the primary key and the historical quality score as the status value. Assign a preset initial reputation benchmark value to the newly connected data source identifier, and use it as the historical quality score of the data source at the beginning of its life cycle; Configure model update parameters and associate them with the reputation profile registry. The model update parameters include at least a time decay factor for reducing the weight of long-term historical data and an error penalty coefficient for quantifying the magnitude of a single abnormal deduction.

4. The multi-source data quality assessment method based on configurable rules and adaptive sampling according to claim 1, characterized in that, In step S2, retrieving the corresponding historical quality score from the dynamic reputation model based on the data source identifier specifically includes: Parse the transmission protocol header or metadata information of the multi-source data stream, extract the string used to uniquely identify the data source attribute, and determine it as the data source identifier; Using the data source identifier as the retrieval key, a retrieval operation is performed on the key-value database or memory cache storing the dynamic reputation model; Determine if the search result exists. If it exists, directly read and output the associated historical quality score. If it does not exist, trigger the initialization program to generate a preset default score and output it as the historical quality score.

5. The multi-source data quality assessment method based on configurable rules and adaptive sampling according to claim 1, characterized in that, In step S3, the joint calculation of the dynamic sampling ratio of the current batch of data specifically includes: A preset risk mapping function is invoked to map the historical quality score to a basic sampling ratio, wherein the lower the historical quality score, the higher the basic sampling ratio generated by the mapping. Calculate the variance or standard deviation index in the data distribution characteristics to generate a volatility correction coefficient, which is used to characterize the stability of the current data stream; The base sampling ratio is weighted and corrected using the volatility correction coefficient to calculate the corrected sampling ratio. The modified sampling ratio is compared and truncated with the preset minimum monitoring threshold and maximum load threshold to finally output the dynamic sampling ratio.

6. The multi-source data quality assessment method based on configurable rules and adaptive sampling according to claim 1, characterized in that, In step S4, performing stratified sampling on the multi-source data stream according to the dynamic sampling ratio specifically includes: Based on a time window or data batch ID, the multi-source data stream is divided into several consecutive logical hierarchical units. For the logical hierarchical unit, the total amount of data within the unit is multiplied by the dynamic sampling ratio to calculate the target number of samples that the unit should extract; Using a pseudo-random number generation algorithm or a hash modulo algorithm, random sampling is performed within the logical hierarchical unit until the number of extracted data entries reaches the target sample size, and all extracted data entries are aggregated to form the target sample set.

7. The multi-source data quality assessment method based on configurable rules and adaptive sampling according to claim 1, characterized in that, In step S5, calling the configurable rule base to perform rule matching and verification on the target sample set specifically includes: Traverse the sample data in the target sample set to identify its business type and the data fields it contains; Based on the business type and data fields, retrieve and load the bound valid rule set from the configurable rule base; Each rule in the set of valid rules is applied sequentially to perform a Boolean logic operation on the sample data. If the result of the operation is false, the sample data is determined to be abnormal data. Record the detailed information of the abnormal data and the corresponding rule severity weights, generate the anomaly detection results, and calculate the ratio of the total amount of the abnormal data to the total amount of the target sample set as the anomaly rate.

8. The multi-source data quality assessment method based on configurable rules and adaptive sampling according to claim 1, characterized in that, In step S6, updating the historical quality score in the dynamic reputation model based on the anomaly rate specifically includes: The historical quality score at the current moment is read from the dynamic reputation model and used as the basis for calculation; The anomaly rate is compared with a preset tolerance threshold. If the anomaly rate is higher than the tolerance threshold, a penalty function is called to calculate the reputation deduction value; otherwise, a recovery function is called to calculate the reputation recovery value. Perform an arithmetic operation on the base value by subtracting the reputation deduction value or adding the reputation recovery value to generate the latest quality score; The latest quality score is written into the dynamic reputation model, overwriting the original historical quality score, thus completing the update of the reputation status of the data source identifier.

Images