A Master Data Management Method and System Based on Strategy Pattern and Rule Engine

Abstract

The application provides a kind of strategy mode and rule engine-based master data management method and system, related to data management field, including receiving master data and self-adapting flow limiting;Perform double check and calculate priority;Select processing rule set to construct workflow;Based on quality mark dynamic generation distribution rule;Master data is written into message queue and pushed to target system according to weight.The application improves master data processing efficiency and quality through differentiated processing strategy and adaptive data distribution, enhances the reliability and flexibility of data flow.

Description

Technical Field

[0001] This invention relates to data management technology, and more particularly to a master data management method and system based on strategy patterns and rule engines. Background Technology

[0002] As enterprises become increasingly information-driven, the importance of master data management in data consistency and quality control is becoming more and more prominent. Existing master data management systems mostly use fixed rules and static workflows for data processing and distribution, lacking the ability to dynamically adjust based on business priorities, data quality, and system operating status. When data traffic fluctuates significantly or its quality is inconsistent, static rules often lead to problems such as delayed processing of high-priority tasks, uneven resource allocation, and unstable data quality. Furthermore, the lack of real-time quality feedback mechanisms and dynamic optimization strategies during data distribution prevents timely adjustments to distribution paths and thread scheduling based on target system status, network environment, and node load, thus limiting the overall throughput and reliability of the system.

[0003] Therefore, it is necessary to provide a master data management method that combines strategy patterns and rule engines to support dynamic rate limiting, priority-driven multi-level verification and automatic correction, rule-based adjustable workflow, dynamic generation of distribution rules based on quality tags, and optimal path selection, so as to improve the real-time performance, accuracy and system resource utilization of master data processing and ensure that master data is efficiently and stably distributed to the target system. Summary of the Invention

[0004] This invention provides a master data management method and system based on strategy patterns and rule engines, which can solve the problems in the prior art.

[0005] A first aspect of this invention provides a master data management method based on strategy patterns and a rule engine, comprising:

[0006] Receive master data from the source system and perform load distribution. Perform adaptive rate limiting based on the master data timestamp and business priority, and allocate the master data to the parallel processing queue.

[0007] Perform field-level and record-level dual verification on the master data in the parallel processing queue, calculate the data processing priority based on the verification results, generate master data with priority identifiers, and trigger automatic data correction based on verification failure data;

[0008] Based on the priority identifier, the corresponding processing rule set is selected from the policy configuration library. The processing workflow is constructed according to the processing rule set. The processing workflow sets the node execution order according to the priority identifier and generates a main data stream with quality tags.

[0009] The main data stream with quality tags is input into the rule engine, and data distribution rules are dynamically generated based on the quality tags to determine the optimal distribution path;

[0010] The master data is written to the distributed message queue based on the optimal distribution path and persistently stored. The processing weight of the distribution thread group is set according to the quality tag. The distribution thread group is started to consume the master data in the queue in parallel according to the weight. The master data is pushed to the target system according to the data distribution rules.

[0011] In one alternative embodiment,

[0012] Receive master data from the source system and perform load balancing. Based on the master data's timestamp and service priority, perform adaptive rate limiting and allocate the master data to the parallel processing queue, including:

[0013] Receive master data from the source system, extract timestamps and service type identifiers from the master data, convert the timestamps into time decay values, convert the service type identifiers into corresponding service priorities, and combine the time decay values ​​with the service priorities to calculate the admission priority of the master data.

[0014] The system resource utilization rate is obtained and a resource utilization rate curve is constructed. When the resource utilization rate curve exceeds a preset utilization rate threshold, a rate limiting coefficient is calculated. The rate limiting coefficient is combined with the admission priority to calculate the admission probability. Based on the admission probability, it is determined whether the master data should enter the system for processing.

[0015] For master data that is allowed to enter the system for processing, the position of the master data feature value is calculated on a pre-constructed hash ring using a consistent hashing algorithm. Virtual nodes corresponding to parallel processing queues are distributed on the hash ring. The corresponding virtual node is determined according to the position of the feature value on the hash ring, and the master data is allocated to the parallel processing queue corresponding to the virtual node.

[0016] In one alternative embodiment,

[0017] Perform field-level and record-level dual validation on the master data in the parallel processing queue, calculate the data processing priority based on the validation results, generate master data with priority identifiers, and trigger automatic data correction based on validation failure data, including:

[0018] Field-level validation is performed on the master data in the parallel processing queue. Multi-dimensional joint validation is performed on the master data according to the field validation rules. The field quality score is calculated based on the weighted combination of the validation results of each dimension.

[0019] Based on the field quality score, the fields that pass the verification are determined and record-level verification is performed. The record integrity score is calculated according to the dynamic weight of the required fields, and the consistency score is calculated according to the deviation between the field value and the historical data. The integrity score and the consistency score are weighted and calculated to obtain the record-level score.

[0020] Data processing priority is calculated based on field quality scores and record-level scores. Priority identifiers are added to the master data according to the data processing priority to generate master data with priority identifiers.

[0021] For master data that fails validation, a correction priority sequence is determined based on the quality score of the fields that failed validation, and the corresponding correction strategy is selected according to the type of validation failure, triggering automatic correction of the master data.

[0022] The master data that has completed automatic correction will be re-validated at both the field level and the record level. If the validation passes, master data with a priority identifier will be generated. If the validation fails and the preset number of corrections has not been reached, automatic correction will continue to be triggered until the validation passes or the maximum number of corrections is reached, at which point it will be marked as pending manual processing.

[0023] In one alternative embodiment,

[0024] Based on priority identifiers, a corresponding processing rule set is selected from the policy configuration library. A processing workflow is constructed according to the processing rule set. The processing workflow sets the node execution order according to the priority identifiers and generates a main data stream with quality tags, including:

[0025] Build a strategy configuration library, divide the processing rules into different levels of processing rule sets according to data quality requirements, configure business verification rules and data transformation rules in each processing rule set, and establish a rule level mapping relationship;

[0026] Obtain the priority identifier and system resource status of the master data, determine the rule selection strategy based on the system resource status, and select the corresponding processing rule set from the policy configuration library based on the priority identifier;

[0027] The rules in the selected processing rule set are analyzed to extract the dependencies between the rules. A directed acyclic graph of rule execution is constructed based on the dependencies, and quality checkpoints are set at the node positions of the directed acyclic graph.

[0028] The directed acyclic graph is converted into a processing workflow. The execution order of nodes is set according to the priority identifier. Data status recording points are set between the nodes of the processing workflow. The data status recording points are used to obtain status information during the processing.

[0029] The master data is processed according to the execution order of the nodes in the processing workflow. The processing results are monitored through the quality checkpoints. When an exception occurs in the rule execution, the exception information and data status are recorded as quality marks, and the processing parameters of subsequent nodes are adjusted according to the quality marks.

[0030] The quality tags during the workflow execution process are summarized, a master data stream with quality tags is generated, and the quality tags are fed back to the policy configuration library for optimizing the corresponding processing rule set.

[0031] In one alternative embodiment,

[0032] The main data stream with quality tags is input into the rule engine, which dynamically generates data distribution rules based on the quality tags, and determines the optimal distribution path, including:

[0033] The main data stream with quality tags is input into the rule engine. Data quality features are extracted from the quality tags. The quality tags are graded and quantified to obtain a quality level matrix. The quality tag configuration in the rule engine is obtained. A weight adaptive function is constructed based on the quality level matrix to calculate the distribution quality score of the main data stream.

[0034] Collect the processing metrics of the target system and record them as the target system status. Obtain the data distribution rule template in the rule engine, build the target system feature library, establish a dynamically floating quality label threshold curve based on the target system status, fit and match the distribution quality score with the quality label threshold curve, and generate the matching degree of the data distribution rule.

[0035] Based on the matching degree of the data distribution rules, filter the rule templates, extract the distribution parameter configuration, generate self-contained data distribution rules and build a rule dependency graph, and complete the rule optimization in the rule engine;

[0036] The system acquires network status and node load data, calculates the transmission latency and processing capacity between nodes, uses the transmission latency and processing capacity as weighting coefficients for nodes and edges in the rule dependency graph, performs path planning to obtain alternative distribution paths, scores the alternative distribution paths, and determines the one with the highest score as the optimal distribution path.

[0037] In one alternative embodiment,

[0038] The master data is written to a distributed message queue based on the optimal distribution path and persisted. The processing weights of the distribution thread groups are set according to quality tags. The distribution thread groups are then started to consume the master data in the queue in parallel according to their weights. Finally, the master data is pushed to the target system according to the data distribution rules, including:

[0039] Analyze the processing capacity and network status of each node in the optimal distribution path, calculate the affinity between nodes to obtain the affinity matrix, construct the partition writing strategy for master data based on the affinity matrix, write the master data into the memory buffer of the distributed message queue according to the partition writing strategy, generate write-ahead logs and perform persistent storage when the buffer reaches the preset buffer threshold, and generate a persistent index containing partition information.

[0040] Extract the quality markers of the master data from the persistent index, construct the mapping relationship between quality features and processing weights, allocate processing quotas to the distribution thread group based on the processing weights, obtain control parameters according to the processing quota initialization message, and generate the processing weights of the distribution thread group.

[0041] Based on the data distribution rules, push conditions are set, and the distribution thread group is controlled to obtain master data from the message queue according to the processing weight. It is determined whether the push conditions are met, and the master data that meets the conditions is converted into a format according to the data distribution rules and then pushed to the target system.

[0042] During data processing and push, information on processing latency and resource consumption is collected. The collected status information is input into the rule engine to update the data distribution rules. Based on the updated data distribution rules, the processing weight of the distribution thread group is adjusted, node affinity is recalculated, and the partition write strategy is updated. The master data is then written to the message queue according to the updated partition write strategy.

[0043] A second aspect of this invention provides a master data management system based on strategy patterns and a rule engine, comprising:

[0044] The first unit is used to receive master data from the source system and perform load distribution. It performs adaptive rate limiting based on the timestamp of the master data and the business priority, and distributes the master data to the parallel processing queue.

[0045] The second unit is used to perform field-level and record-level dual verification on the master data in the parallel processing queue, calculate the data processing priority based on the verification results, generate master data with priority identifiers, and trigger automatic data correction based on the verification failure data.

[0046] The third unit is used to select the corresponding processing rule set from the policy configuration library based on the priority identifier, construct a processing workflow according to the processing rule set, and the processing workflow sets the node execution order according to the priority identifier and generates a main data stream with quality tags.

[0047] The fourth unit is used to input the main data stream with quality tags into the rule engine, dynamically generate data distribution rules based on the quality tags, and determine the optimal distribution path;

[0048] The fifth unit is used to write master data into a distributed message queue based on the optimal distribution path and perform persistent storage. It sets the processing weight of the distribution thread group according to the quality tag, starts the distribution thread group to consume the master data in the queue in parallel according to the weight, and pushes the master data to the target system according to the data distribution rules.

[0049] A third aspect of the present invention provides an electronic device, comprising:

[0050] processor;

[0051] Memory used to store processor-executable instructions;

[0052] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0053] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0054] In this embodiment, the combination of strategy patterns and a rule engine enables intelligent processing of master data management, effectively improving data processing efficiency and accuracy. An adaptive rate limiting mechanism based on timestamps and business priorities ensures stable system operation under high load, preventing performance degradation due to excessive system resource consumption. A dual verification mechanism at both the field and record levels, combined with automatic correction, significantly improves the quality and reliability of master data and reduces the data error rate. Through dynamic matching of priority identifiers and processing rule sets, the system can automatically select the most suitable processing strategy based on data characteristics, achieving flexible adaptation of the processing flow and efficient resource utilization. A distribution mechanism based on quality tags ensures priority distribution of high-quality data, meeting the differentiated needs of business systems for data timeliness and accuracy. The combined application of distributed message queues and weighted parallel processing enhances the system's scalability and fault tolerance, making the entire master data management process more efficient, reliable, and intelligent, effectively supporting the demand for high-quality master data during enterprise digital transformation. Attached Figure Description

[0055] Figure 1 This is a flowchart illustrating the master data management method based on strategy pattern and rule engine according to an embodiment of the present invention;

[0056] Figure 2 This is a flowchart illustrating the adaptive data distribution process based on node affinity according to an embodiment of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0059] Figure 1 This is a flowchart illustrating the master data management method based on strategy patterns and rule engines according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0060] Receive master data from the source system and perform load distribution. Perform adaptive rate limiting based on the master data timestamp and business priority, and allocate the master data to the parallel processing queue.

[0061] Perform field-level and record-level dual verification on the master data in the parallel processing queue, calculate the data processing priority based on the verification results, generate master data with priority identifiers, and trigger automatic data correction based on verification failure data;

[0062] Based on the priority identifier, the corresponding processing rule set is selected from the policy configuration library. The processing workflow is constructed according to the processing rule set. The processing workflow sets the node execution order according to the priority identifier and generates a main data stream with quality tags.

[0063] The main data stream with quality tags is input into the rule engine, and data distribution rules are dynamically generated based on the quality tags to determine the optimal distribution path;

[0064] The master data is written to the distributed message queue based on the optimal distribution path and persistently stored. The processing weight of the distribution thread group is set according to the quality tag. The distribution thread group is started to consume the master data in the queue in parallel according to the weight. The master data is pushed to the target system according to the data distribution rules.

[0065] In one optional implementation, receiving master data from the source system and performing load balancing, performing adaptive rate limiting based on the master data's timestamp and service priority, and allocating the master data to a parallel processing queue include:

[0066] Receive master data from the source system, extract timestamps and service type identifiers from the master data, convert the timestamps into time decay values, convert the service type identifiers into corresponding service priorities, and combine the time decay values ​​with the service priorities to calculate the admission priority of the master data.

[0067] The system resource utilization rate is obtained and a resource utilization rate curve is constructed. When the resource utilization rate curve exceeds a preset utilization rate threshold, a rate limiting coefficient is calculated. The rate limiting coefficient is combined with the admission priority to calculate the admission probability. Based on the admission probability, it is determined whether the master data should enter the system for processing.

[0068] For master data that is allowed to enter the system for processing, the position of the master data feature value is calculated on a pre-constructed hash ring using a consistent hashing algorithm. Virtual nodes corresponding to parallel processing queues are distributed on the hash ring. The corresponding virtual node is determined according to the position of the feature value on the hash ring, and the master data is allocated to the parallel processing queue corresponding to the virtual node.

[0069] In this implementation, master data transmitted from the source system is first received. This master data may come from multiple business systems and contains key information such as timestamps and business type identifiers. The timestamp reflects the point in time when the master data was generated, while the business type identifier is used to distinguish master data under different business scenarios. After extracting the timestamp from the master data, it needs to be converted into a time decay value. Specifically, the conversion method is to calculate the time difference between the current system time and the master data timestamp and apply a time decay function for conversion. For example, if the current system time is 2024-10-15 14:30:00 and the master data timestamp is 2024-10-15 14:25:00, the time difference is 5 minutes. Assuming a linear decay function is used, setting the decay to 0.05 per minute, the time decay value is 1 - 5 × 0.05 = 0.75.

[0070] When converting a business type identifier into a business priority, the system maintains a business type mapping table, with different business types corresponding to different priority values. For example, order-type master data might be assigned a priority value of 0.9, while inventory-type master data might have a priority value of 0.7, and customer information-type master data might have a priority value of 0.8. The admission priority of the master data is calculated by combining the time decay value with the business priority. The combination calculation method uses a weighted average. Assuming the weight of the time decay value is 0.6 and the weight of the business priority is 0.4, then for order-type master data with a time decay value of 0.75 and a business priority of 0.9, its admission priority is 0.75 × 0.6 + 0.9 × 0.4 = 0.81.

[0071] The system resource utilization monitoring module acquires metrics such as CPU utilization, memory utilization, and network bandwidth utilization in real time, constructing a resource utilization curve. The resource utilization sampling period is once per second, and the system retains the sampling data from the most recent 60 seconds to build a smooth curve. The preset utilization threshold is typically set at 80%. When the resource utilization curve exceeds this threshold, the system enters a rate-limiting state. The rate-limiting coefficient is calculated based on the difference between the resource utilization and the threshold; the higher the resource utilization, the larger the rate-limiting coefficient. For example, if the current resource utilization is 85% and the preset threshold is 80%, a linear calculation method can be used to obtain a rate-limiting coefficient of (85-80) / 20 = 0.25, meaning the system will reject 25% of requests.

[0072] The admission probability is calculated by combining the rate limiting coefficient and the admission priority. The calculation method is: Admission Probability = 1 - Rate Limiting Coefficient × (1 - Admission Priority). In the previous example, with a rate limiting coefficient of 0.25 and an admission priority of 0.81, the admission probability = 1 - 0.25 × (1 - 0.81) = 0.9525, meaning that the master data has a 95.25% probability of being accepted and processed by the system. A random number between 0 and 1 is generated. If the random number is less than the admission probability, the master data is allowed to enter the system for processing; otherwise, processing is rejected and a rate limiting warning is returned.

[0073] For master data allowed to enter the system for processing, it needs to be allocated to appropriate parallel processing queues to achieve load balancing. A consistent hashing algorithm is used to calculate the position of the master data's characteristic value on a pre-constructed hash ring. The master data characteristic value can be the hash value of the master data ID, business key, or a combination thereof. The hash ring is a ring space of integers from 0 to 2^32-1, and the hash function can be the MurmurHash3 algorithm, mapping the master data characteristic value to a specific position on the hash ring.

[0074] Virtual nodes corresponding to parallel processing queues are distributed on the hash ring. Each actual processing queue corresponds to multiple virtual nodes on the ring to avoid data skew. For example, the system is configured with 4 parallel processing queues, each with 50 virtual nodes, for a total of 200 virtual nodes distributed on the hash ring. The position of a virtual node on the hash ring is determined by hashing "queue identifier + virtual node index". For example, the position of the 3rd virtual node in processing queue 1 is hash("queue1#3").

[0075] After calculating the position of the master data feature value on the hash ring, the first virtual node encountered is searched clockwise along the hash ring. The actual processing queue corresponding to this virtual node is the target queue for the master data. For example, if the hash calculation result of the master data feature value is 12345678, the first virtual node encountered clockwise on the hash ring is 12400000, corresponding to processing queue 2. Therefore, the master data is assigned to processing queue 2.

[0076] In practical applications, when the system processing queues are expanded or shrunk, the consistent hashing algorithm ensures that only a small portion of the master data allocation changes, while the majority of master data remains allocated to the original queues, avoiding a global shuffle and improving system stability. Simultaneously, the introduction of virtual nodes solves the data skew problem, making the load on each processing queue more balanced.

[0077] This master data management approach, combining a strategy pattern with a rule engine, ensures that critical business data is processed first under high load through a combination of time decay and business priority calculations, along with an adaptive resource rate limiting mechanism. This improves overall system processing efficiency and business continuity. Simultaneously, the application of the consistent hashing algorithm results in more balanced master data distribution and stronger system scalability, making it suitable for large-scale distributed master data management scenarios.

[0078] In one optional implementation, dual validation at the field and record levels is performed on the master data in the parallel processing queue. Based on the validation results, data processing priorities are calculated, master data with priority identifiers is generated, and automatic data correction is triggered based on validation failure data. This includes:

[0079] Field-level validation is performed on the master data in the parallel processing queue. Multi-dimensional joint validation is performed on the master data according to the field validation rules. The field quality score is calculated based on the weighted combination of the validation results of each dimension.

[0080] Based on the field quality score, the fields that pass the verification are determined and record-level verification is performed. The record integrity score is calculated according to the dynamic weight of the required fields, and the consistency score is calculated according to the deviation between the field value and the historical data. The integrity score and the consistency score are weighted and calculated to obtain the record-level score.

[0081] Data processing priority is calculated based on field quality scores and record-level scores. Priority identifiers are added to the master data according to the data processing priority to generate master data with priority identifiers.

[0082] For master data that fails validation, a correction priority sequence is determined based on the quality score of the fields that failed validation, and the corresponding correction strategy is selected according to the type of validation failure, triggering automatic correction of the master data.

[0083] The master data that has completed automatic correction will be re-validated at both the field level and the record level. If the validation passes, master data with a priority identifier will be generated. If the validation fails and the preset number of corrections has not been reached, automatic correction will continue to be triggered until the validation passes or the maximum number of corrections is reached, at which point it will be marked as pending manual processing.

[0084] Exemplarily, when performing field-level verification on the master data in the parallel processing queue, the system retrieves the master data record from the master database, such as a customer information record, which contains fields such as customer number, customer name, contact phone number, address, and email. For each field, multi-dimensional joint verification is performed according to the preset field verification rules. For the contact phone number field, three-dimensional verification is performed: format verification (checking if it conforms to the "area code - number" format), length verification (checking if the number of digits is within a reasonable range), and validity verification (checking if the area code exists). Assuming the weight of format verification is 0.4, the weight of length verification is 0.3, and the weight of validity verification is 0.3, the contact phone number is verified: format verification passes (score 1.0), length verification passes (score 1.0), and validity verification passes (score 1.0). The calculated field quality score is 0.4×1.0 + 0.3×1.0 + 0.3×1.0 = 1.0, indicating that this field fully conforms to the verification rules.

[0085] For the customer name field, if the weight of special character verification is specified as 0.5 and the weight of length verification is 0.5, the system verifies the customer name "XX Technology Co., Ltd. #": special character verification fails (contains "#", score 0.0), length verification passes (score 1.0). The calculated field quality score is 0.5×0.0 + 0.5×1.0 = 0.5, indicating that this field partially conforms to the verification rules. Calculate the quality score for each field and determine whether the field verification passes according to the preset threshold (such as 0.7).

[0086] When determining the fields that pass the verification based on the field quality scores and performing record-level verification, first identify the fields with field quality scores higher than the threshold, such as the contact phone number (score 1.0) and address (assuming score 0.9). Then perform record-level verification and calculate the record integrity score. Assuming that in the customer information record, the customer number, contact phone number, and address are required fields, where the dynamic weight of the customer number is 0.5, the dynamic weight of the contact phone number is 0.3, and the dynamic weight of the address is 0.2. If the customer number is empty, then this required field is missing, and the record integrity score is calculated as 0.5×0 + 0.3×1 + 0.2×1 = 0.5.

[0087] Meanwhile, calculate the consistency score based on the deviation between the field value and the historical data. For example, if the contact phone number detected in this record is the same as that in the customer's historical record, the consistency score is 1.0; while there are slight differences in the address field compared to the historical record (such as the house number changing from "1001" to "1001 Room"), the consistency score is 0.8. Assuming the consistency weight of the contact phone number is 0.6 and the consistency weight of the address is 0.4, the consistency score is calculated as 0.6×1.0 + 0.4×0.8 = 0.92. If the integrity score weight is 0.6 and the consistency score weight is 0.4, the record-level score is calculated as 0.6×0.5 + 0.4×0.92 = 0.668.

[0088] When calculating the data processing priority based on the field quality score and the record-level score, comprehensively consider the field-level verification result and the record-level verification result. Assume that the proportion of the field quality score is set to 0.4 and the proportion of the record-level score is 0.6. For the aforementioned customer information record, take the average value of all field quality scores (assumed to be 0.8), then the data processing priority score is 0.4×0.8 + 0.6×0.668 = 0.7208. Set the priority identifier according to the priority score range: 0.9 - 1.0 is "High Priority", 0.7 - 0.9 is "Medium Priority", 0.5 - 0.7 is "Low Priority", and 0 - 0.5 is "To Be Processed". The priority identifier for this record is "Medium Priority", and the master data with the priority identifier generated will be placed in the corresponding processing queue.

[0089] For the master data with verification failures, determine the correction priority sequence based on the quality scores of the fields that failed the verification. For example, the customer name field (score 0.5) failed the verification because it contains special characters, and the email field (assumed score 0.3) failed the verification due to a formatting error. Sort the fields in descending order of their quality scores, first correct the customer name, and then correct the email. For the special character "#" in the customer name, the system adopts a preset correction strategy - remove the special character, and correct "XX Technology Co., Ltd.#" to "XX Technology Co., Ltd.". For the email with a formatting error, adopt the domain name completion strategy and correct it by querying the domain name suffix in the customer's historical record.

[0090] After automatic correction, field-level and record-level validations are re-executed. The corrected customer name "XXX Technology Co., Ltd." passes special character validation (score 1.0) and length validation (score 1.0), raising its field quality score to 1.0; the corrected email address passes format validation, raising its field quality score to 0.9. Record-level and priority scores are recalculated, assuming a priority score of 0.85, corresponding to the "medium priority" label. If any fields still fail validation after correction, and the preset correction limit (e.g., 3 times) has not been reached, the system will continue to trigger automatic correction. If the correction limit is reached and the record still fails validation, it is marked as "pending manual processing" and handed over to data management personnel for manual intervention.

[0091] In this embodiment, a combination of field-level and record-level dual verification enables multi-dimensional quality inspection and refined evaluation of master data, comprehensively identifying defects in data such as field integrity and consistency. Data processing priority is calculated based on quality scores, ensuring that high-quality, high-priority data is processed first, improving overall processing efficiency. An automatic correction mechanism based on quality scores is introduced, enabling rapid data error repair without manual intervention, and achieving closed-loop quality control through cyclical verification and correction. When automatic correction reaches its limit and still fails verification, the data is marked for manual processing, preventing erroneous data from flowing into subsequent processes. This significantly improves the accuracy, integrity, and reliability of master data, reducing system data processing latency and manual maintenance costs.

[0092] In one optional implementation, a corresponding processing rule set is selected from the policy configuration library based on a priority identifier, and a processing workflow is constructed according to the processing rule set. The processing workflow sets the node execution order according to the priority identifier and generates a main data stream with quality tags, including:

[0093] Build a strategy configuration library, divide the processing rules into different levels of processing rule sets according to data quality requirements, configure business verification rules and data transformation rules in each processing rule set, and establish a rule level mapping relationship;

[0094] Obtain the priority identifier and system resource status of the master data, determine the rule selection strategy based on the system resource status, and select the corresponding processing rule set from the policy configuration library based on the priority identifier;

[0095] The rules in the selected processing rule set are analyzed to extract the dependencies between the rules. A directed acyclic graph of rule execution is constructed based on the dependencies, and quality checkpoints are set at the node positions of the directed acyclic graph.

[0096] The directed acyclic graph is converted into a processing workflow. The execution order of nodes is set according to the priority identifier. Data status recording points are set between the nodes of the processing workflow. The data status recording points are used to obtain status information during the processing.

[0097] The master data is processed according to the execution order of the nodes in the processing workflow. The processing results are monitored through the quality checkpoints. When an exception occurs in the rule execution, the exception information and data status are recorded as quality marks, and the processing parameters of subsequent nodes are adjusted according to the quality marks.

[0098] The quality tags during the workflow execution process are summarized, a master data stream with quality tags is generated, and the quality tags are fed back to the policy configuration library for optimizing the corresponding processing rule set.

[0099] In this embodiment, when building the strategy configuration library, various processing rules are managed hierarchically according to the enterprise's data quality standards. Taking customer master data as an example, processing rules are divided into three levels: critical, important, and general. Critical rules include mandatory field validation rules (such as customer ID, name, and tax number must be filled in) and uniqueness constraint rules (such as customer ID being globally unique); important rules include data format validation rules (such as phone number format being area code-number) and field association consistency rules (such as postal code matching address); general rules include pinyin generation rules (generating pinyin codes based on Chinese names) and data standardization rules (such as address standardization). For each rule, its rule ID, rule type, rule description, execution priority, applicable business scenario, and triggering condition are configured. For example, for the "customer name required" rule, its ID is "CR001", its rule type is "mandatory validation", its priority is "high", it applies to "all customer data", and its triggering condition is "when customer name is empty". These rules are stored in the policy configuration library as key-value pairs, and a mapping table between rule levels and data priorities is established, so that the system can quickly locate the set of rules to be executed based on data priorities.

[0100] When obtaining the priority identifier and system resource status of master data, the system reads the priority label in the data packet header. For example, a batch of customer data contains the priority identifier "P1" (highest priority), and the system monitoring module detects that the current CPU utilization is 65%, the memory utilization is 70%, and the system load index is 0.7. Based on these parameters, the current resource status is determined to be "medium load," and the system selects a strategy table according to preset rules to decide whether to execute the complete rule set (including all rules at the critical, important, and general levels) on the P1-level data. If the data is "P3," which has a lower priority, under the same system resource status, only the critical and some important rules may be executed to balance processing efficiency and quality requirements. During the specific selection process, a pre-established mapping relationship table is queried. For example, P1-level data is mapped to rule set RS001 (containing 25 rules), while P3-level data is mapped to rule set RS003 (containing 10 rules).

[0101] When analyzing the selected set of processing rules, the dependencies between rules are extracted and a directed acyclic graph (DAG) is constructed. Taking customer master data processing as an example, the analysis reveals that the "address standardization" rule depends on the "address validity check" rule, and the "tax number format verification" rule depends on the "tax number not empty check" rule. Based on these dependencies, a DAG of rule execution is constructed, where each node represents a rule and edges represent dependencies. For example, node "CR001" (required customer name) points to node "CR008" (customer name format verification), indicating that the existence of the name must be verified before its format can be verified. In the constructed DAG, quality checkpoints are set at key node locations, such as setting a quality checkpoint QC001 after the "customer ID uniqueness check" node to monitor the pass rate of uniqueness verification.

[0102] When converting a directed acyclic graph into a processing workflow, the execution order of nodes is set according to priority identifiers. For P1 level data, all rules are executed in the order of dependencies, while for P3 level data, some non-critical nodes may be skipped. Data status log points are set between important nodes in the workflow. For example, a status log point (SR002, SR003) is set before and after the "customer address standardization" process to record the data status before and after the process, including data values, timestamps, and processing status. In specific implementation, the system generates a JSON format workflow definition file, which contains a list of nodes, execution order, conditional branches, and status log point configurations. For example: {"workflowId":"WF001","nodes":[{"nodeId":"N001","ruleId":"CR001","next":"N002 "}, {"nodeId": "N002", "ruleId": "CR008", "next": "N003", "stateRecordPoint": "SR001"}]}.

[0103] When processing master data according to the execution order of workflow nodes, the system applies rules one by one. For example, when executing the "Customer Tax ID Format Verification" rule, if a data entry's tax ID is found to be inconsistent with the specified format, an exception message will be recorded at the quality checkpoint: "dataId": "CUST10086", "ruleId": "CR015", "status": "failed", "reason": "Tax ID format error", "timestamp": "2025-09-15T10:30:25". Simultaneously, the processing parameters of subsequent nodes will be adjusted according to preset strategies, such as increasing the risk factor for this data entry or marking it as requiring manual review. For specific error patterns, the system may trigger automatic correction logic, such as automatically supplementing the area code or standardizing the address format.

[0104] When aggregating quality markers during workflow execution, a master data stream with these quality markers is generated. For each master data entry, a quality score and a quality tag are appended. For example, if a customer's data passes 20 rule validations and fails 2, its quality score is calculated as 91 points, and it is marked as "Good." Simultaneously, the quality markers are fed back to the policy configuration library to optimize the corresponding processing rule sets. For instance, if the "email format validation" rule frequently generates false positives, its parameters will be adjusted or its strictness reduced. Through this closed-loop optimization mechanism, the rules in the policy configuration library can continuously adapt to the actual characteristics of business data, improving the accuracy and efficiency of data processing. The generated master data stream with quality markers is output in JSON format, containing the original data, processing results, quality score, quality markers, and processing history, facilitating downstream systems to make appropriate processing decisions based on the quality status.

[0105] In this embodiment, by constructing a hierarchical strategy configuration library and dynamically selecting processing rule sets based on system resource status, adaptive optimization of the master data processing flow is achieved. Constructing a directed acyclic graph based on rule dependencies and setting quality checkpoints ensures the rationality of the rule execution order and the controllability of the results. Introducing data status recording points and a quality tagging mechanism enables real-time monitoring and anomaly tracking throughout the data processing process. Dynamic adjustment of subsequent processing parameters based on quality tags improves processing accuracy and stability. Feeding quality tags back to the strategy configuration library for rule optimization forms a closed-loop improvement mechanism, continuously enhancing the applicability of the processing rule set and the overall quality of data processing.

[0106] In one optional implementation, the main data stream with quality tags is input into the rule engine, and data distribution rules are dynamically generated based on the quality tags to determine the optimal distribution path, including:

[0107] The main data stream with quality tags is input into the rule engine. Data quality features are extracted from the quality tags. The quality tags are graded and quantified to obtain a quality level matrix. The quality tag configuration in the rule engine is obtained. A weight adaptive function is constructed based on the quality level matrix to calculate the distribution quality score of the main data stream.

[0108] Collect the processing metrics of the target system and record them as the target system status. Obtain the data distribution rule template in the rule engine, build the target system feature library, establish a dynamically floating quality label threshold curve based on the target system status, fit and match the distribution quality score with the quality label threshold curve, and generate the matching degree of the data distribution rule.

[0109] Based on the matching degree of the data distribution rules, filter the rule templates, extract the distribution parameter configuration, generate self-contained data distribution rules and build a rule dependency graph, and complete the rule optimization in the rule engine;

[0110] The system acquires network status and node load data, calculates the transmission latency and processing capacity between nodes, uses the transmission latency and processing capacity as weighting coefficients for nodes and edges in the rule dependency graph, performs path planning to obtain alternative distribution paths, scores the alternative distribution paths, and determines the one with the highest score as the optimal distribution path.

[0111] In this embodiment, the master data stream is tagged with quality information before entering the processing stage. This quality tagging covers multiple dimensions of data quality characteristics, including completeness, consistency, accuracy, and timeliness. For example, a master data set for an order might have the following quality tags: completeness 0.95, consistency 0.87, accuracy 0.92, and timeliness 0.98. These quality tags are generated during the data acquisition or preprocessing stage using a specific algorithm, reflecting the master data's performance across various quality dimensions. When the master data stream with these quality tags is input into the rule engine, the engine first extracts data quality features from the tags and performs hierarchical quantization on these features. Hierarchical quantization maps continuous quality values ​​to discrete grade ranges; for example, values ​​above 0.9 are mapped to grade A (value 4), 0.8-0.9 to grade B (value 3), 0.7-0.8 to grade C (value 2), 0.6-0.7 to grade D (value 1), and below 0.6 to grade E (value 0). For the order master data mentioned above, its quality grade matrix after hierarchical quantification is [4, 3, 4, 4].

[0112] The rules engine pre-configures the weight settings for quality markers, such as integrity weight 0.3, consistency weight 0.25, accuracy weight 0.25, and timeliness weight 0.2. When constructing the weight adaptive function based on the quality level matrix, the differentiated requirements of different business scenarios for data quality dimensions need to be considered. The weight adaptive function dynamically adjusts the weights of each dimension by analyzing the correlation between historical distribution success rate and quality level. For example, for transaction master data, historical data shows that accuracy is highly correlated with distribution success rate, and the system will automatically increase the weight of accuracy. Assuming the adjusted weights are integrity 0.25, consistency 0.2, accuracy 0.35, and timeliness 0.2, the distribution quality score of this order master data can be calculated as 4×0.25+3×0.2+4×0.35+4×0.2=3.8.

[0113] The target system processing metric acquisition module monitors key metrics such as processing capacity, response time, and error rate of each target system in real time and records them as the target system status. Assume there are three target systems with the following processing metrics: System 1 (processing capacity 1000 messages per second, response time 15ms, error rate 0.1%), System 2 (processing capacity 1500 messages per second, response time 25ms, error rate 0.2%), and System 3 (processing capacity 800 messages per second, response time 10ms, error rate 0.05%). The data distribution rule templates stored in the rule engine contain various distribution strategies, such as high-quality priority strategy, load balancing strategy, and shortest response time strategy. When constructing the target system feature library, the processing metrics of each system are standardized and multi-dimensional feature vectors are established to form the system feature library.

[0114] A dynamically floating quality label threshold curve is established based on the target system state, and the threshold is dynamically adjusted considering the current system load. For example, when a target system is heavily loaded, the quality threshold requirement for receiving master data will be increased; conversely, when the system load is light, the quality threshold will be appropriately decreased. Assuming that system 1 is currently loaded at 75%, according to the preset threshold adjustment function, its quality threshold curve will rise by 10%, meaning that master data that originally received a quality score of 3.0 or higher will now be required to have a quality score of 3.3 or higher. The distribution quality score is fitted and matched with the quality label threshold curve to calculate the matching degree of the data distribution rule. The fitting and matching process uses the point-to-surface distance calculation method. First, the quality feature of the master data is regarded as a point in a multi-dimensional space, and the quality label threshold curve is a surface in this space. The normalized distance from the point to the surface is calculated as the basic value of the matching degree. In specific implementation, the difference between the actual value and the threshold is calculated for each quality dimension, and then the overall matching degree is obtained by combining the differences of each dimension.

[0115] Suppose the quality metric for a certain order master data is: Integrity 0.95, Consistency 0.87, Accuracy 0.92, and Timeliness 0.98. The target system's current load is 75%, and the corresponding quality metric thresholds are: Integrity threshold 0.80, Consistency threshold 0.75, Accuracy threshold 0.85, and Timeliness threshold 0.90. Calculate the percentage exceeding the threshold for each dimension: Integrity (0.95-0.80) / 0.80 = 18.75%, Consistency (0.87-0.75) / 0.75 = 16%, Accuracy (0.92-0.85) / 0.85 = 8.24%, and Timeliness (0.98-0.90) / 0.90 = 8.89%. Next, based on the aforementioned weights for each dimension (completeness 0.25, consistency 0.2, accuracy 0.35, timeliness 0.2), the weighted average percentage exceeding the threshold is calculated as follows: 18.75% × 0.25 + 16% × 0.2 + 8.24% × 0.35 + 8.89% × 0.2 = 12.5%. This 12.5% ​​represents the matching degree of the master data to the distribution rules of the target system, indicating the extent to which the quality score of the master data exceeds the current required threshold of the system.

[0116] The matching score calculation also considers the balance of quality distribution. If the master data significantly exceeds the threshold in one dimension while other dimensions just reach the threshold, a penalty is applied through the variance term. For example, the standard deviation of the percentage exceeding the threshold for each dimension is calculated. If the standard deviation is too large, the final matching score is reduced. In the above case, the standard deviation of the percentage exceeding the threshold for each dimension is approximately 5.13%. Assuming the system's standard deviation penalty factor is set to 0.5, the final matching score is adjusted to 12.5% ​​- 5.13% × 0.5 = 9.94%.

[0117] After calculating the matching degree for multiple target systems, the system sorts them according to their matching degree and prioritizes the target systems with the highest matching degree as the distribution targets. Furthermore, the matching algorithm is continuously optimized using historical distribution data during the fitting and matching process. The system records the processing effect of each distribution and the target system, and adjusts the matching algorithm parameters through machine learning methods to improve matching accuracy. For example, by analyzing historical data, it was found that for order-type master data, the accuracy dimension has the greatest impact on the processing success rate, and the system automatically increases the weight of the accuracy dimension in the matching calculation.

[0118] Rule templates are filtered based on the matching degree of the data distribution rules; the higher the matching degree, the stronger the applicability of the rule template. After selecting the top three rule templates with the highest matching degree, their distribution parameter configurations are extracted, including target system priority, data batch size, timeout retry strategy, etc. A self-contained data distribution rule is then generated, which is a complete rule object containing attributes such as triggering conditions, execution actions, and priority. For example, for order master data, the generated self-contained rule might be: when the quality score > 3.5 and the data type is order, prioritize distribution to system 3, batch size is 200, timeout is 2 seconds, and retries are 3.

[0119] When constructing the rule dependency graph, the pre- and post-rules, mutual exclusion relationships, etc., between each rule are modeled as a directed graph. The rule optimization process includes rule conflict detection, redundant rule merging, and rule priority adjustment. For example, if there are two rules with overlapping applicable conditions but different execution actions, the system will resolve the conflict based on rule priority or more refined condition judgment.

[0120] The network status and node load data acquisition module obtains information such as network latency, packet loss rate, node CPU utilization, and memory utilization in real time. The transmission latency between nodes is calculated using a weighted average method, combining network latency and node processing capacity to form a comprehensive evaluation index. For example, if the network latency from the source node to System 1 is 5ms, and System 1's current CPU utilization is 75%, then the comprehensive transmission latency might be 5 × (1 + 75% × 0.5) = 6.88ms. Processing capacity calculation considers the node's nominal processing capacity and the current available resource ratio. For example, if System 1's nominal processing capacity is 1000 messages / second, and the current resource availability is 25%, then the actual processing capacity is 1000 × 25% = 250 messages / second.

[0121] Transmission latency and processing capacity are used as weighting coefficients for nodes and edges in the rule dependency graph, where node weights represent processing capacity and edge weights represent transmission latency. During path planning, an improved shortest path algorithm is employed to identify multiple alternative distribution paths from the source node to each target system. When scoring these alternative distribution paths, factors such as total path latency, processing capacity matching degree, and load balancing degree are comprehensively considered. For example, the path scoring formula can comprehensively consider the reciprocal of total latency (40%), processing capacity matching degree (35%), and load balancing degree (25%). Ultimately, the path with the highest score is determined as the optimal distribution path, and master data will be distributed to the target systems along this path.

[0122] This master data management method, which combines strategy patterns with a rule engine, enables the master data distribution process to dynamically adjust the distribution strategy based on data quality characteristics and the target system status. This achieves optimal matching between data quality and system processing capabilities, improves the success rate and efficiency of master data distribution, reduces system resource consumption, and enhances overall system performance.

[0123] like Figure 2 As shown, this embodiment demonstrates the adaptive data distribution process based on node affinity.

[0124] In one optional implementation, the master data is written to a distributed message queue based on the optimal distribution path and persistently stored. The processing weights of the distribution thread groups are set according to quality tags. The distribution thread groups are then started to consume the master data in the queue in parallel according to their weights. The master data is then pushed to the target system according to the data distribution rules, including:

[0125] Analyze the processing capacity and network status of each node in the optimal distribution path, calculate the affinity between nodes to obtain the affinity matrix, construct the partition writing strategy for master data based on the affinity matrix, write the master data into the memory buffer of the distributed message queue according to the partition writing strategy, generate write-ahead logs and perform persistent storage when the buffer reaches the preset buffer threshold, and generate a persistent index containing partition information.

[0126] Extract the quality markers of the master data from the persistent index, construct the mapping relationship between quality features and processing weights, allocate processing quotas to the distribution thread group based on the processing weights, obtain control parameters according to the processing quota initialization message, and generate the processing weights of the distribution thread group.

[0127] Based on the data distribution rules, push conditions are set, and the distribution thread group is controlled to obtain master data from the message queue according to the processing weight. It is determined whether the push conditions are met, and the master data that meets the conditions is converted into a format according to the data distribution rules and then pushed to the target system.

[0128] During data processing and push, information on processing latency and resource consumption is collected. The collected status information is input into the rule engine to update the data distribution rules. Based on the updated data distribution rules, the processing weight of the distribution thread group is adjusted, node affinity is recalculated, and the partition write strategy is updated. The master data is then written to the message queue according to the updated partition write strategy.

[0129] In this implementation, it is necessary to analyze the processing capacity and network conditions of each node in the optimal distribution path, and calculate the affinity between nodes to obtain an affinity matrix. Node processing capacity is comprehensively evaluated using indicators such as CPU utilization, memory usage, and disk I / O, while network conditions include key indicators such as bandwidth utilization, network latency, and packet loss rate. The affinity calculation between nodes considers geographical proximity, network connection quality, and historical interaction success rate, and uses a weighted summation method to obtain the affinity value. For example, for a distributed system with 5 nodes, a 5×5 affinity matrix can be generated. Assuming that the affinity between node 1 and node 2 is 0.85, it indicates that there is a high interaction efficiency between these two nodes. The entire affinity matrix may be as follows: 0.85 for node 1 and node 2, 0.62 for node 1 and node 3, 0.45 for node 1 and node 4, and 0.38 for node 1 and node 5; 0.76 for node 2 and node 3, 0.57 for node 2 and node 4, and 0.43 for node 2 and node 5; 0.83 for node 3 and node 4, and 0.69 for node 3 and node 5; and 0.92 for node 4 and node 5.

[0130] When constructing a partitioning strategy for master data based on an affinity matrix, an affinity clustering algorithm is used to group nodes with high affinity into the same partition group. For example, according to the affinity matrix above, nodes 4 and 5 can be divided into one partition group, nodes 1 and 2 into another partition group, and node 3 can be a separate partition group or a partition group with node 2, which has high affinity. The partitioning strategy also needs to consider the principle of data locality; highly correlated master data should be written to the same partition to reduce cross-partition operations. In specific implementation, the partition ID can be determined by taking the modulo of the business key hash value of the master data with the number of partitions. For example, order-type master data can use the hash value of the order number, customer-type master data can use the hash value of the customer ID, and product-type master data can use the hash value of the product code.

[0131] During the process of writing master data to the memory buffer of the distributed message queue according to the partition write strategy, each partition maintains a separate memory buffer. The buffer size can be dynamically adjusted according to the node's memory resources, typically ranging from 64MB to 256MB. When the buffer reaches a preset buffer threshold, the write-ahead log mechanism is triggered. The write-ahead log records include fields such as operation type, timestamp, master data content, and partition information, and the format uses a compact binary format to improve I / O efficiency. When performing persistent storage, a batch write method is used, and multiple write-ahead logs are merged into a storage block, with the storage block size typically set to 4MB or 8MB. After persistence is completed, a persistent index containing partition information is generated. The index structure is implemented using a B+ tree, with the key being the master data ID or business key, and the value being the physical location of the data in storage and partition information.

[0132] Quality markers for master data are extracted from the persistent index, and a mapping relationship between quality features and processing weights is constructed. Quality features include scores for dimensions such as completeness, accuracy, consistency, and timeliness. Processing weights represent the priority of the distribution thread group in processing this type of master data. The mapping relationship is implemented using a piecewise linear function. For example, a processing weight of 5 is assigned when the quality score is above 0.9; 4 when the quality score is between 0.8 and 0.9; 3 when the quality score is between 0.7 and 0.8; 2 when the quality score is between 0.6 and 0.7; and 1 when the quality score is below 0.6. Processing quotas are allocated to distribution thread groups based on the processing weights. The processing quota determines the number of messages a thread group retrieves from the message queue each time. For example, a thread group with a processing weight of 5 can retrieve 50 messages at a time, a thread group with a processing weight of 3 can retrieve 30 messages at a time, and a thread group with a processing weight of 1 can retrieve 10 messages at a time.

[0133] The system retrieves control parameters based on the processing quota initialization message, including batch size, fetch interval, and maximum wait time. For example, the batch size directly corresponds to the processing quota, the fetch interval is inversely proportional to the processing weight (higher weight, shorter fetch interval), and the maximum wait time is set to a fixed value, such as 500 milliseconds. After generating the processing weights for the distribution thread groups, each thread group allocates computing resources according to its weight ratio. For example, in a system with a total of 5 processing threads, a thread group with a weight of 5 can be allocated 2 threads, a thread group with a weight of 3 can be allocated 2 threads, and a thread group with a weight of 1 can be allocated 1 thread.

[0134] Push conditions are set based on data distribution rules, including target system status conditions, data batch conditions, and time window conditions. For example, pushes are only executed when the target system's CPU utilization is below 80%; batch pushes are performed when more than 100 similar master data entries accumulate; and forced pushes are performed when more than 10 seconds have passed since the last push. Distribution thread groups retrieve master data from the message queue according to processing weights; higher-weighted thread groups retrieve data more frequently and in larger quantities at a time. The push conditions are checked; if met, the push process begins; otherwise, accumulation continues or waiting continues. Master data that meets the conditions undergoes format conversion according to the data distribution rules, including field mapping, data type conversion, and encoding conversion.

[0135] For example, if the source system uses the date format "YYYY-MM-DD" while the target system requires the format "DD / MM / YYYY", a corresponding conversion is necessary. The converted master data is then pushed to the target system via an adapter, which encapsulates the protocol details for communication with the target system.

[0136] During data processing and push, information on processing latency and resource consumption is collected. Processing latency includes multiple stages such as queuing latency, processing latency, and transmission latency. Resource consumption includes CPU utilization, memory usage, and network bandwidth utilization. The collected status information is input into the rule engine at a certain frequency (e.g., every 10 seconds). The rule engine triggers corresponding rules to update the data distribution rules based on the status information. For example, when a target system's response slows down, the push batch size for that system is adjusted or the push frequency is reduced; when the processing time for a certain type of master data increases, the number of corresponding processing threads is increased.

[0137] The processing weights of the distribution thread groups are adjusted based on the updated data distribution rules. For example, a thread group with a weight of 3 is adjusted to a weight of 4, and its processing quota and resource allocation are increased accordingly. Node affinity is recalculated and the partition write strategy is updated. For example, if increased network latency between nodes 2 and 3 leads to a decrease in affinity, their partition assignments may need to be adjusted. Master data is written to the message queue according to the updated partition write strategy, completing the adaptive adjustment cycle.

[0138] In this embodiment, efficient and balanced storage and fast access to master data in a distributed message queue are achieved. By dynamically allocating processing quotas for distribution thread groups based on the mapping relationship between quality tags and processing weights, high-quality or high-priority data can be prioritized during parallel consumption, improving distribution efficiency and resource utilization. Data format conversion and precise delivery are performed using push conditions and data distribution rules to ensure that the data received by different target systems meets their business and technical requirements. Real-time collection of processing latency and resource usage information during operation and feedback to the rule engine allows for dynamic optimization of distribution rules, processing weights, and partitioning strategies, forming an adaptive distribution control mechanism that improves system throughput, real-time responsiveness, and overall stability.

[0139] A second aspect of this invention provides a master data management system based on strategy patterns and a rule engine, the system comprising:

[0140] The first unit is used to receive master data from the source system and perform load distribution. It performs adaptive rate limiting based on the timestamp of the master data and the business priority, and distributes the master data to the parallel processing queue.

[0141] The second unit is used to perform field-level and record-level dual verification on the master data in the parallel processing queue, calculate the data processing priority based on the verification results, generate master data with priority identifiers, and trigger automatic data correction based on the verification failure data.

[0142] The third unit is used to select the corresponding processing rule set from the policy configuration library based on the priority identifier, construct a processing workflow according to the processing rule set, and the processing workflow sets the node execution order according to the priority identifier and generates a main data stream with quality tags.

[0143] The fourth unit is used to input the main data stream with quality tags into the rule engine, dynamically generate data distribution rules based on the quality tags, and determine the optimal distribution path;

[0144] The fifth unit is used to write master data into a distributed message queue based on the optimal distribution path and perform persistent storage. It sets the processing weight of the distribution thread group according to the quality tag, starts the distribution thread group to consume the master data in the queue in parallel according to the weight, and pushes the master data to the target system according to the data distribution rules.

[0145] A third aspect of the present invention provides an electronic device, comprising:

[0146] processor;

[0147] Memory used to store processor-executable instructions;

[0148] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0149] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0150] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A master data management method based on a strategy pattern and a rule engine, characterized in that, The method comprises the following steps: receiving source system master data and performing load distribution, performing adaptive flow control according to the time stamp and business priority of the master data, and distributing the master data to a parallel processing queue; performing field-level and record-level double verification on the master data in the parallel processing queue, calculating data processing priority based on the verification result, generating master data with priority identification, and triggering automatic correction of data based on failed verification data; selecting a corresponding processing rule set from a policy configuration library based on the priority identification, constructing a processing workflow according to the processing rule set, setting node execution order according to the priority identification, and generating a master data stream with quality marks; inputting the master data stream with quality marks into a rule engine, dynamically generating data distribution rules according to the quality marks, and determining the optimal distribution path; based on the optimal distribution path, writing the master data into a distributed message queue and performing persistent storage, setting the processing weight of the distribution thread group according to the quality marks, starting the distribution thread group to consume the master data in the queue in parallel according to the weight, and pushing the master data to the target system according to the data distribution rules; performing field-level and record-level double verification on the master data in the parallel processing queue, calculating data processing priority based on the verification result, generating master data with priority identification, and triggering automatic correction of data based on failed verification data comprises: performing field-level verification on the master data in the parallel processing queue, performing multi-dimensional joint verification on the master data according to the field verification rules, and calculating the field quality score based on the weighted combination of the verification results of each dimension; determining the verified fields based on the field quality score and performing record-level verification, calculating the record integrity score according to the dynamic weight of the required fields, calculating the consistency score according to the deviation of the field value from the historical data, and calculating the record-level score by weighting the integrity score and the consistency score; calculating the data processing priority based on the field quality score and the record-level score, adding the priority identification to the master data according to the data processing priority, and generating master data with priority identification; for the master data that fails the verification, determining the correction priority sequence based on the quality score of the field that fails the verification, selecting the corresponding correction strategy according to the type of the failed verification, and triggering automatic correction of the master data; re-performing field-level verification and record-level verification on the master data that has completed automatic correction, generating master data with priority identification if the verification is passed, continuing to trigger automatic correction if the verification fails and the number of preset corrections is not reached, and marking it as pending manual processing after the verification is passed or the number of corrections reaches the upper limit.

2. The method of claim 1, wherein, The method comprises the following steps: receiving source system master data, extracting the time stamp and business type identification from the master data, converting the time stamp into a time decay value, converting the business type identification into a corresponding business priority, and combining the time decay value and the business priority to calculate the access priority of the master data; Acquiring system resource usage and constructing resource usage curve, calculating throttling coefficient when the resource usage curve exceeds preset usage threshold, combining the throttling coefficient with the admission priority to calculate admission probability, and determining whether the main data enters system processing based on the admission probability; For the main data allowed to enter system processing, the position of the main data feature value on the pre-constructed hash ring is calculated through the consistent hash algorithm, and the hash ring is distributed with virtual nodes corresponding to parallel processing queues; the corresponding virtual node is determined according to the position of the feature value on the hash ring, and the main data is distributed to the parallel processing queue corresponding to the virtual node.

3. The method of claim 1, wherein, Selecting the corresponding processing rule set from the policy configuration library based on the priority identifier, constructing the processing workflow according to the processing rule set, and generating the main data stream with quality marks including: Constructing a policy configuration library, dividing processing rules into different levels of processing rule sets according to data quality requirements, configuring business verification rules and data conversion rules in each processing rule set, and establishing a rule level mapping relationship; Acquiring the priority identifier of the main data and the system resource state, determining the rule selection strategy according to the system resource state, and selecting the corresponding processing rule set from the policy configuration library based on the priority identifier; Analyzing the rules in the selected processing rule set, extracting the dependency relationship between the rules, constructing a directed acyclic graph for rule execution based on the dependency relationship, and setting quality checkpoints at the node positions of the directed acyclic graph; Converting the directed acyclic graph into a processing workflow, setting the node execution order according to the priority identifier, and setting data state recording points between the nodes of the processing workflow, which are used to acquire state information in the processing process; Processing the main data according to the node execution order of the processing workflow, monitoring the processing result through the quality checkpoint, recording the quality mark when the rule execution is abnormal, and adjusting the processing parameters of the subsequent nodes according to the quality mark; Summarizing the quality marks in the processing workflow execution process to generate the main data stream with quality marks, and feeding back the quality marks to the policy configuration library for optimizing the corresponding processing rule set.

4. The method of claim 1, wherein, Inputting the main data stream with quality marks into the rule engine, dynamically generating data distribution rules according to the quality marks, and determining the optimal distribution path including: Inputting the main data stream with quality marks into the rule engine, extracting data quality features from the quality marks, performing hierarchical quantization processing on the quality marks to obtain a quality level matrix, acquiring quality mark configurations in the rule engine, constructing a weight adaptive function based on the quality level matrix, and calculating the distribution quality score of the main data stream; Collecting processing indicators of the target system and recording them as target system states, acquiring data distribution rule templates in the rule engine, constructing a target system feature library, establishing a dynamically floating quality mark threshold curve according to the target system states, fitting and matching the distribution quality score with the quality mark threshold curve, and generating the matching degree of the data distribution rule. According to the matching degree of the data distribution rule, a rule template is screened, distribution parameter configuration is extracted, a self-contained data distribution rule is generated, and a rule dependency relationship graph is constructed, and rule optimization is completed in a rule engine; Network state and node load data are acquired, transmission delay and processing capacity between nodes are calculated, the transmission delay and processing capacity are taken as the weighting coefficients of the nodes and edges of the rule dependency relationship graph, path planning is performed to obtain candidate distribution paths, the candidate distribution paths are scored, and the highest scoring path is determined as the optimal distribution path.

5. The method of claim 1, wherein, Based on the optimal distribution path, the main data is written into a distributed message queue and persistent storage is performed, the processing weight of the distribution thread group is set according to the quality mark, the distribution thread group is started to consume the main data in the queue in parallel according to the weight, and the main data is pushed to the target system according to the data distribution rule, including: The processing capacity and network status of each node in the optimal distribution path are analyzed, the affinity between nodes is calculated to obtain an affinity matrix, and a partitioned writing strategy for the main data is constructed based on the affinity matrix, the main data is written into the memory buffer of the distributed message queue according to the partitioned writing strategy, a prewrite log is generated and persistent storage is performed when the buffer reaches a preset buffer threshold, and a persistent index containing partition information is generated; The quality mark of the main data is extracted from the persistent index, a mapping relationship between quality characteristics and processing weight is constructed, the processing quota of the distribution thread group is allocated based on the processing weight, the message acquisition control parameter is initialized according to the processing quota, and the processing weight of the distribution thread group is generated; Based on the data distribution rule, a push condition is set, the main data is acquired from the message queue by the distribution thread group according to the processing weight, it is judged whether the push condition is met, and the main data meeting the condition is pushed to the target system after format conversion according to the data distribution rule; In the data processing and pushing process, processing delay and resource occupation information are collected, the collected state information is input into the rule engine, and the data distribution rule is updated; based on the updated data distribution rule, the processing weight of the distribution thread group is adjusted, the node affinity is recalculated, and the partitioned writing strategy is updated, and the main data is written into the message queue according to the updated partitioned writing strategy.

6. A master data management system based on a strategy pattern with a rules engine for implementing the method of any of the preceding claims 1-5, characterized in that, It includes: The first unit is used for receiving source system main data and performing load distribution, performing adaptive flow control according to the timestamp and business priority of the main data, and distributing the main data to a parallel processing queue; The second unit is used for performing field-level and record-level double verification on the main data in the parallel processing queue, calculating data processing priority based on the verification result, generating main data with priority identification, and triggering automatic data correction based on verification failure data; The third unit is used for selecting a corresponding processing rule set from a strategy configuration library based on the priority identification, constructing a processing workflow according to the processing rule set, setting node execution order according to the priority identification, and generating a main data stream with quality marks; The fourth unit is used for inputting the main data stream with quality marks into a rule engine, dynamically generating a data distribution rule according to the quality marks, and determining an optimal distribution path. The fifth unit is used for writing the main data into the distributed message queue based on the optimal distribution path and performing the persistent storage, setting the processing weight of the distribution thread group according to the quality mark, starting the distribution thread group to consume the main data in the queue in parallel according to the weight, and pushing the main data to the target system according to the data distribution rule.

7. An electronic device, comprising: Comprise: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the method of any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions, when executed by the processor, implement the method of any one of claims 1 to 5.

Images