A processing method and system of multi-source heterogeneous asset data

Abstract

The application relates to the technical field of data processing, and discloses a multi-source heterogeneous asset data processing method and system. The method first constructs a data feature graph containing field nodes and associated edges, and determines the dynamic semantic complexity score of each field node. Based on the score, the corresponding cleaning rule chain is assembled and distributed computing resources are configured. After cleaning, the score is corrected according to the actual processing time consumption. At the same time, the historical feedback coefficient is updated according to the quality evaluation result after cleaning, and the calculation weight configuration of the score is adjusted accordingly. The corrected and optimized score will be used for rule assembly and resource configuration of subsequent cleaning tasks, forming a closed-loop adaptive mechanism to improve cleaning efficiency and quality.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to a method and system for processing multi-source heterogeneous asset data. Background Technology

[0002] In the wave of digitalization, enterprise data is experiencing explosive growth, with diverse sources and types, forming a typical multi-source heterogeneous data environment. This data encompasses multiple channels, including internal business systems, external collaboration platforms, and IoT devices, and includes various forms such as structured data (e.g., database tables), semi-structured data (e.g., JSON, XML), and unstructured data (e.g., text, images). High-quality data cleaning is an essential prerequisite for effectively managing and utilizing these data assets.

[0003] Existing multi-source heterogeneous data cleaning systems typically employ a two-layer independent architecture of "rule engine + resource scheduler". The rule engine focuses solely on "what to clean," determining cleaning operations based on preset rules or simple logic; the resource scheduler focuses solely on "how to allocate computation," uniformly allocating computational resources to all cleaning tasks or based on static strategies. This linear pipeline architecture has a fundamental flaw: there is a natural matching relationship between rule complexity and resource requirements—high-complexity rules (such as CRF-based entity boundary recognition or BERT-based deep semantic analysis) consume significant computational resources, while low-complexity rules (such as format validation and null value checking) require only a small amount of resources. If these two decisions are made independently, it inevitably leads to a mismatch between "heavy rules with light resources" or "light rules with heavy resources," resulting in resource waste or performance bottlenecks.

[0004] Furthermore, existing feedback optimization mechanisms typically only apply to the rule or model parameters themselves, failing to delve into the adjustment of resource allocation strategies. The system lacks a unified, dynamic metric that spans the entire process of rule generation, resource allocation, and performance evaluation, hindering continuous improvement in overall system efficiency. A long-standing technical bias in this field holds that the rule engine and resource scheduler should be decoupled to maintain module independence, which ironically impedes further breakthroughs in system collaborative efficiency.

[0005] In view of this, a method and system for processing multi-source heterogeneous asset data is proposed. Summary of the Invention

[0006] This application proposes a method and system for processing multi-source heterogeneous asset data to solve one of the aforementioned technical problems.

[0007] The technical solution adopted in this application is as follows:

[0008] In a first aspect, embodiments of this application provide a method for processing multi-source heterogeneous asset data, including:

[0009] Acquire multi-source heterogeneous asset data, construct a data feature graph containing field nodes and associated edges, and determine a dynamic semantic complexity score for the field nodes;

[0010] Based on the dynamic semantic complexity score, assemble the corresponding cleaning rule chain for the field node, and determine the distributed computing resource configuration corresponding to the cleaning rule chain;

[0011] Data cleaning is performed based on the distributed computing resource configuration, and the dynamic semantic complexity score is adjusted according to the actual processing time during the execution process.

[0012] The historical feedback coefficients are updated based on the quality assessment results after cleaning, and the calculation weight configuration of the dynamic semantic complexity score is adjusted based on the historical feedback coefficients, so that the adjusted dynamic semantic complexity score can participate in the assembly of cleaning rule chains and the configuration of distributed computing resources in the subsequent data cleaning process.

[0013] Preferably, acquiring multi-source heterogeneous asset data includes:

[0014] Read the basic priority tag of the access data source;

[0015] The system periodically scans the waiting time of data in the access queue and performs priority over-level judgment based on the waiting time.

[0016] If the waiting time of the second priority data source is detected to be greater than the first preset time threshold but not greater than the second preset time threshold, the effective priority of the second priority data source is dynamically updated to a weight value between the first priority and the second priority.

[0017] If the waiting time of the second priority data source is detected to be greater than the second preset time threshold, the effective priority is further updated to a higher weight value than the weight value between the first priority and the second priority, and concurrent thread resources are allocated to each access data source according to the updated weight ratio.

[0018] Once the backlog queue of the corresponding access data source is cleared, a status reset command is triggered to restore the effective priority of the access data source to its basic priority label.

[0019] Wherein, the first preset time threshold is less than the second preset time threshold, and the first priority is higher than the second priority.

[0020] Preferably, a data feature graph containing field nodes and associated edges is constructed, and a dynamic semantic complexity score is determined for the field nodes, including:

[0021] The multi-source heterogeneous asset data is parsed and semantically annotated to determine the semantic labels, value distribution information, and cross-field relationships of the fields.

[0022] Generate the field node and the association edge based on the field semantic label and the cross-field association relationship;

[0023] Calculate the weights of associated edges based on the field matching rate and functional dependency strength between field nodes;

[0024] The semantic variation coefficient is calculated based on the number of different values ​​in the field corresponding to the field node, the total number of records, and the number of value types.

[0025] The initial dynamic semantic complexity score is calculated using the following formula:

[0026] S0 = α × E + β × V;

[0027] Where S0 represents the initial dynamic semantic complexity score, E represents the weight of the associated edge, V represents the semantic variation coefficient, α represents the initial coefficient of the weight of the associated edge, β represents the initial coefficient of the semantic variation coefficient, and α+β=1.

[0028] Preferably, based on the dynamic semantic complexity score, a corresponding cleaning rule chain is assembled for the field node, including:

[0029] Read the dynamic semantic complexity score of field nodes;

[0030] If the dynamic semantic complexity score is determined to be lower than the second determination threshold, the first assembly instruction is triggered to retrieve the basic format verification rules from the rule template library and assemble them into a lightweight cleaning rule chain.

[0031] If the dynamic semantic complexity score is determined to be higher than the first determination threshold, the second assembly instruction is triggered to insert deep semantic cleaning rules into the cleaning rule chain and activate the entity boundary recognition logic and similarity merging logic.

[0032] If the dynamic semantic complexity score is determined to be no lower than the second determination threshold and no higher than the first determination threshold, then the third assembly instruction is triggered to retrieve the basic format verification rules and the candidate semantic verification rules that match the field semantic tags from the rule template library and assemble them into a default cleaning rule chain.

[0033] Wherein, the first determination threshold is greater than the second determination threshold.

[0034] Preferably, when assembling a cleaning rule chain containing deep semantic cleaning rules, the rule priority score is calculated according to the following formula:

[0035] P = 0.5 × C + 0.3 × M + 0.2 × T;

[0036] Where P represents the rule priority score, C represents the entity boundary recognition confidence score, M represents the semantic similarity score, and T represents the rule template default weight score.

[0037] The rule modules are arranged in descending order of their rule priority scores, so that the rule modules with higher priority scores are placed at the front of the cleaning rule chain and executed first.

[0038] Preferably, determining the distributed computing resource configuration corresponding to the cleaning rule chain includes:

[0039] The cleaning rule chain is transformed into a distributed operator that can be executed in parallel;

[0040] Based on the scoring range in which the dynamic semantic complexity score is located, the corresponding processor core count and memory capacity configuration parameters are mapped and allocated.

[0041] If the weight of the associated edge between multiple field nodes is detected to be higher than the preset weight judgment value, a collaborative constraint instruction is triggered to distribute the cleaning tasks corresponding to the multiple field nodes to the same computing node for processing.

[0042] Preferably, the dynamic semantic complexity score is corrected based on the actual processing time during execution, including:

[0043] Before performing data cleaning, the baseline processing time of the cleaning task corresponding to the field node is calculated by benchmark sampling.

[0044] The normalized dynamic semantic complexity score is linearly mapped to a preset score range to obtain the current mapped score, and the expected processing time is calculated according to the following formula:

[0045] Te = T0 × (1 + Sm / M);

[0046] Where Te represents the expected processing time, T0 represents the baseline processing time, Sm represents the current mapping score, and M is the maximum span value of the preset score interval;

[0047] During the data cleaning process, the actual processing time is obtained. If the actual processing time is determined to be greater than Te×(1+δ), a scoring correction instruction is triggered, where δ represents the preset deviation ratio.

[0048] In response to the scoring correction instruction, the dynamic semantic complexity score of the corresponding field node is increased by a preset increment step, and the increased dynamic semantic complexity score is written back to the data feature map.

[0049] Preferably, the quality assessment results after cleaning include the residual rate of duplicate values ​​and the semantic consistency rate;

[0050] After obtaining the residual rate of the duplicate values, a similarity threshold adjustment judgment is performed;

[0051] If the residual rate of the duplicate value is determined to be greater than the preset residual rate threshold, a threshold reduction instruction is triggered to reduce the text similarity threshold used to determine whether two text segments have performed a synonym merging operation, so as to relax the determination conditions for synonym text merging.

[0052] The lowered text similarity threshold is written back to the rule generation engine so that the lowered text similarity threshold can participate in the assembly of deep semantic cleaning rules for subsequent batches of data.

[0053] Preferably, updating the historical feedback coefficients based on the quality assessment results after cleaning, and adjusting the calculation weight configuration of the dynamic semantic complexity score based on the historical feedback coefficients, includes:

[0054] Obtain the residual rate of duplicate values ​​and the semantic consistency rate for the corresponding cleaning round;

[0055] The feedback contribution value for the corresponding cleaning round is calculated using the following formula:

[0056] Fi = 1 - Ri + Ci;

[0057] Where Fi represents the feedback contribution value corresponding to the i-th round of cleaning, Ri represents the normalized duplicate value residual rate after the i-th round of cleaning, and Ci represents the normalized semantic consistency rate after the i-th round of cleaning.

[0058] The feedback contribution value is configured to increase when the duplicate value residual rate decreases and / or the semantic consistency rate increases, and decrease when the duplicate value residual rate increases and / or the semantic consistency rate decreases.

[0059] Calculate the historical feedback coefficient based on the feedback contribution value and attenuation factor of each cleaning round:

[0060] H = Σ[λ^(ni) × Fi];

[0061] Where H represents the historical feedback coefficient, λ represents the decay factor, n represents the current total round, and i represents the historical cleaning round;

[0062] The historical feedback coefficients are normalized to obtain the normalized historical feedback coefficients H'.

[0063] Calculate historical feedback weights based on the total number of cleaning rounds:

[0064] γ=min(γmax,1-e^(-k×n));

[0065] Where γ represents the historical feedback weight, γmax represents the preset weight upper limit, e represents the natural constant, and k represents the growth constant;

[0066] Based on the residual rate of repeated values ​​and the semantic consistency rate, adjust the corresponding coefficients of the associated edge weights and the corresponding coefficients of the semantic variation coefficients, and perform normalization processing on the adjusted corresponding coefficients of the associated edge weights, the corresponding coefficients of the semantic variation coefficients, and the historical feedback weights so that the sum of the three is 1.

[0067] The updated dynamic semantic complexity score is calculated using the following formula:

[0068] S = α' × E + β' × V + γ × H';

[0069] Where S represents the updated dynamic semantic complexity score, α' represents the adjusted coefficient of the associated edge weight, β' represents the adjusted coefficient of the semantic variation coefficient, E represents the associated edge weight, V represents the semantic variation coefficient, and H' represents the normalized historical feedback coefficient.

[0070] Secondly, embodiments of this application provide a system for processing multi-source heterogeneous asset data to implement the method described in any of the above-mentioned embodiments, comprising:

[0071] The graph scoring determination module is used to acquire multi-source heterogeneous asset data, construct a data feature graph containing field nodes and associated edges, and determine a dynamic semantic complexity score for the field nodes.

[0072] The rule resource determination module is used to assemble a corresponding cleaning rule chain for the field node based on the dynamic semantic complexity score, and determine the distributed computing resource configuration corresponding to the cleaning rule chain;

[0073] The cleaning execution and scoring correction module is used to perform data cleaning based on the distributed computing resource configuration, and to correct the dynamic semantic complexity score according to the actual processing time during the execution process.

[0074] The feedback weight adjustment module is used to update the historical feedback coefficients based on the quality assessment results after cleaning, and adjust the calculation weight configuration of the dynamic semantic complexity score based on the historical feedback coefficients, so that the adjusted dynamic semantic complexity score can participate in the assembly of cleaning rule chains and the configuration of distributed computing resources in the subsequent data cleaning process.

[0075] This application discloses a method and system for processing multi-source heterogeneous asset data. The method constructs a data feature map and introduces a dynamic semantic complexity score as the core metric, enabling the system to accurately quantify the cleaning difficulty of each field node. This score directly drives the assembly of the cleaning rule chain (high scores trigger deep semantic cleaning rules, low scores use lightweight rules) and the allocation of distributed computing resources (high scores allocate more CPU / memory). This dual decision-making mechanism driven by the same score fundamentally solves the mismatch problem between rules and resources, significantly improving resource utilization efficiency.

[0076] This solution not only utilizes scoring to guide initial rules and resource allocation but also incorporates a dynamic correction and optimization mechanism. On one hand, the system adjusts the dynamic semantic complexity score based on the deviation between actual and expected processing time, ensuring the score reflects the true cleaning difficulty in real time. On the other hand, the system updates historical feedback coefficients based on post-cleaning quality assessment results (such as duplicate value retention rate and semantic consistency rate) and adjusts the scoring calculation weights accordingly. This closed-loop mechanism ensures that the scoring is no longer static but continuously evolves and optimizes as the system runs, leading to increasingly accurate rule chain assembly and resource allocation.

[0077] Thanks to the aforementioned collaborative matching and closed-loop optimization mechanisms, this invention can complete higher-quality data cleaning tasks with fewer computing resources. Specifically, end-to-end processing time is significantly reduced, the rate of duplicate values ​​is greatly decreased, the semantic consistency rate is significantly improved, and the idle rate of system resources is effectively controlled. This enables the system to maintain efficient, stable, and high-quality processing capabilities even when facing massive, complex, multi-source heterogeneous asset data. Attached Figure Description

[0078] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0079] Figure 1 A flowchart illustrating a method for processing multi-source heterogeneous asset data provided in an embodiment of the present invention;

[0080] Figure 2 This is a schematic diagram of the structure of a multi-source heterogeneous asset data processing system provided in an embodiment of the present invention. Detailed Implementation

[0081] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0082] like Figure 1 As shown in some embodiments of this application, this embodiment provides a method for processing multi-source heterogeneous asset data. Specifically, the method includes the following steps:

[0083] Step S101: Obtain multi-source heterogeneous asset data, construct a data feature graph containing field nodes and associated edges, and determine a dynamic semantic complexity score for the field nodes.

[0084] The core of this step is to transform the raw, messy, multi-source heterogeneous data into a structured, quantifiable knowledge representation—a data feature graph—and assign each element (field node) in the graph a dynamic semantic complexity score that can guide all subsequent decisions.

[0085] First, the system accesses data from different sources (such as relational databases, JSON / XML files, scanned images, etc.) and different formats (structured, semi-structured, and unstructured) through a flexible adaptation layer. During this process, the system performs necessary automatic format conflict conversions, such as value type conversion, structural nesting flattening, and cross-form semantic alignment based on a lightweight semantic model, ensuring that all data has a consistent and processable form before entering the graph construction stage.

[0086] The process of constructing a data feature graph is not a simple metadata extraction, but an in-depth semantic understanding and relationship mining process. The system uses a pre-trained lightweight BERT model to perform in-depth semantic analysis on field names and their sample data, identify the true meaning of the fields (for example, uniformly labeling "name" and "user name" as "full name"), and generate field semantic labels. Through statistical analysis, the value rules of the field are obtained, such as the number of different values, the total number of records, the range of abnormal values, etc., and these information is the basis for measuring the internal variability of the field. The system further analyzes the logical dependencies between fields. This includes both explicit database constraints (such as primary and foreign key relationships) and implicit semantic associations (such as the hierarchical relationship of "province-city"). These association relationships are modeled as association edges in the graph, and are quantified as association edge weights by calculating the matching rate, functional dependency strength or mutual information. Finally, with fields as nodes and association relationships as edges, a complete graph structure, that is, the data feature graph, is constructed.

[0087] The dynamic semantic complexity score is not static, but a comprehensive quantification of the difficulty of field cleaning. Its initial value is calculated by weighting two core dimensions:

[0088] Association edge weight (E): It reflects the degree to which the field is constrained by other fields. The higher the weight, the more the value of this field depends on other fields, the more context needs to be considered during cleaning, and the higher the complexity.

[0089] Semantic variation coefficient (V): It reflects the degree of chaos of the field's own values. This coefficient comprehensively considers the richness of different values in the field (distinct_count / total_count) and the discreteness of data types (value_types). The larger the value, the more problems such as a large number of synonyms, aliases, and inconsistent formats exist in the field, and the greater the difficulty of cleaning.

[0090] The initial dynamic semantic complexity score S0=α×E+β×V, where α and β are initial weight coefficients. This score will be used as the sole basis for subsequent rule chain assembly and resource allocation.

[0091] It should be noted that the specific values ​​of the initial coefficient α of the associated edge weights and the initial coefficient β of the semantic variation coefficients can be configured according to the source characteristics and association strength characteristics of the data to be processed. In practical applications, the preferred ranges for α and β are 0.2 ≤ α ≤ 0.8 and β = 1 - α. For example, when the multi-source heterogeneous asset data to be processed is mainly internal business system data (such as structured data in enterprise ERP and CRM systems), with a large number of primary and foreign key relationships between fields and relatively standardized values ​​within fields, the preferred value of α is 0.6 ≤ α ≤ 0.8, and the corresponding value of β is 0.4 ≥ β ≥ 0.2. When the multi-source heterogeneous asset data to be processed is mainly external heterogeneous data (such as JSON / XML data from multiple partners, IoT sensor data, OCR-recognized text, etc.), with a large number of synonyms, inconsistent formats, and other variations within fields of each data source, and with weak relationships between cross-source fields, the preferred value of α is 0.2 ≤ α ≤ 0.4, and the corresponding value of β is 0.8 ≥ β ≥ 0.6. When the multi-source heterogeneous asset data to be processed includes both internal business data and external heterogeneous data, with both certain relationships between fields and varying degrees of internal variation, the preferred value of α is 0.4 < α < 0.6, and the corresponding value of β is 0.6 > β > 0.4.

[0092] For example, suppose the system is connected to the following three data sources simultaneously:

[0093] Data source A: MySQL order table from the core business system, containing fields "order_id" (order number), "cust_name" (customer name), and "ship_addr" (shipping address); Data source B: JSON-formatted user profiles from partners, containing fields "user.name" and "address.city"; Data source C: Scanned text of customer information after OCR recognition.

[0094] The process of constructing the map and scoring is as follows:

[0095] Field parsing and labeling: The system identified the fields "cust_name", "user.name", and "Name" in the scanned document, and all were uniformly labeled as "Name". Similarly, "ship_addr" and "address.city" were labeled as "Address".

[0096] Construct association edges: The system finds that there is a strong one-to-one correspondence between "order_id" and "payment_id" in the payment system, and calculates the association edge weight between them as E=0.95. By analyzing a large amount of address data, the system finds that the value of the "province" field can highly predict the value of the "city" field (for example, "Beijing Municipality" almost only corresponds to districts under "Beijing Municipality"), and calculates the association edge weight E=0.85 between "province and city". The "name" field basically has no strong logical dependency with other fields, and its association edge weight E=0.3.

[0097] Calculate the semantic variation coefficient: There are a large number of variants in the "name" field, such as "Zhang San", "Mr. Zhang", "ZhangSan", etc., and a small number of numbers and symbols are included, and its semantic variation coefficient V=0.75 is calculated. The "order number" field has a very regular format, all are unique numeric strings, with almost no variation, and its semantic variation coefficient V=0.1. The "address" field has extremely large variations, with various writing forms such as "Jing", "Beijing", "Beijing Municipality", "Beijing", etc., and the nesting level is chaotic, and its semantic variation coefficient V=0.8.

[0098] Determine the initial dynamic semantic complexity score S0 (set α=0.6, β=0.4, and set the logical judgment criteria: when the association edge weight E≥0.5, it is judged as high association, and when E<0.5, it is judged as low association; when the semantic variation coefficient V≥0.5, it is judged as high variation, and when V<0.5, it is judged as low variation): "Order number" node: S0=α×E+β×V=0.6×0.95+0.4×0.1=0.61; since E=0.95≥0.5 and V=0.1<0.5, it is judged as high association and low variation. "Name" node: S0=0.6×0.3+0.4×0.75=0.48; since E=0.3<0.5 and V=0.75≥0.5, it is judged as low association and high variation. "Address" node: S0=0.6×0.85+0.4×0.8=0.83; since E=0.85≥0.5 and V=0.8≥0.5, it is judged as high association and high variation.

[0099] This score clearly reveals the cleaning difficulties of each field: "address" is the most complex and requires in-depth semantic processing; "name" comes next, which mainly solves internal variations; although "order number" is simple, due to its strong correlation, it also needs to be handled carefully to ensure data consistency.

[0100] It should be noted that, in specific implementation scenarios, regarding the storage of data feature graphs, although graph databases (such as Neo4j) are preferred in the embodiments for storing the graphs, those skilled in the art can also choose other data structures or storage systems that can effectively express the relationships between nodes and edges, such as adjacency list models based on relational databases, or dedicated in-memory graph computing frameworks, as long as they can support efficient graph traversal and reading and writing of node attributes (i.e., dynamic semantic complexity scoring).

[0101] Regarding the semantic annotation model, the lightweight BERT model was used in the embodiment for semantic annotation, but this is not the only option. Other pre-trained language models (such as RoBERTa, ALBERT) or more traditional dictionary- and rule-based named entity recognition (NER) methods can be used instead, as long as the goal of accurately semantically normalizing the field can be achieved, they all fall within the scope of protection of this solution.

[0102] Regarding the calculation logic of the associated edge weights, the embodiments provide two calculation methods: one based on matching rate / functional dependency and the other based on mutual information. In fact, any algorithm capable of quantifying the strength of logical dependencies or semantic associations between fields can be used for this purpose. For example, a graph neural network (GNN) based method can be introduced to learn the implicit association weights between fields, or weights can be directly assigned using rules defined by business experts. These different quantification methods are all within the scope of this invention. All the above optional solutions fall within the protection scope of this application.

[0103] Step S102: Based on the dynamic semantic complexity score, assemble the corresponding cleaning rule chain for the field node, and determine the distributed computing resource configuration corresponding to the cleaning rule chain.

[0104] This step uses the dynamic semantic complexity score generated in step S101 as the sole decision-making basis, simultaneously driving two key actions: adaptive assembly of the cleaning rule chain and precise configuration of distributed computing resources, thereby ensuring that the cleaning task of each field can obtain a processing strategy and computing power that matches its difficulty.

[0105] Assemble the corresponding cleaning rule chain: The system maintains a rule template library containing various cleaning rule modules ranging from lightweight to heavyweight, such as basic format validation, regular expression matching, entity boundary recognition, and deep semantic similarity calculation. The system compares the dynamic semantic complexity score with preset thresholds (such as the first judgment threshold and the second judgment threshold) to determine which rule modules to invoke.

[0106] For low-scoring fields, only lightweight cleaning rule chains are assembled, such as null value imputation and basic format validation, to avoid over-computation. For high-scoring fields, deep semantic cleaning rules are triggered, activating complex logic such as accurate entity boundary identification and record merging based on semantic similarity. For medium-scoring fields, a default cleaning rule chain is assembled, combined with basic validation and specific validation rules that match the semantic labels of that field. This on-demand assembly approach ensures that the complexity of the rule chain is highly consistent with the actual difficulty of data cleaning.

[0107] Determine the corresponding distributed computing resource configuration: The complexity of the rule chain directly determines its computing resource requirements. The system uses a score-resource configuration mapping table to directly map the range of dynamic semantic complexity scores to specific configuration parameters such as the number of processor cores and memory capacity.

[0108] Lightweight rule chains are allocated fewer resources to conserve cluster overhead, while heavyweight rule chains are allocated more CPU cores and more memory to ensure they can complete complex computations within a reasonable timeframe. Furthermore, the system considers the weights of related edges in the graph. If multiple highly correlated fields need to be cleaned simultaneously, the system triggers a collaborative constraint instruction to schedule these related cleaning tasks onto the same compute node. This minimizes cross-node data transfer (shuffle) overhead because data from related fields can interact directly in local memory, significantly improving processing efficiency.

[0109] For example, following the example in step S101, we know the initial dynamic semantic complexity scores of the three fields: "order number" (0.61), "name" (0.48), and "address" (0.83). Assume that the system sets the first judgment threshold to 0.7 and the second judgment threshold to 0.5.

[0110] For the "Address" field (score 0.83 > 0.7):

[0111] Rule chain assembly: The system triggers the second assembly instruction to assemble a rule chain containing deep semantic cleaning rules. This chain may include: (1) basic format verification; (2) address entity boundary recognition (distinguishing between province, city, district, and street); (3) semantic vector-based similarity calculation to merge variants such as "Beijing" and "Beijing Municipality".

[0112] Resource allocation: Due to its high score, the system allocated 4 CPU cores and 8GB of memory to it. Simultaneously, the system detected a high weight (>0.8) in the association edges between the "address" and "province" and "city" fields, thus triggering a collaborative constraint instruction to distribute the cleaning tasks for these three fields to compute node A for execution.

[0113] For the "Order Number" field (0.5 < rating 0.61 < 0.7):

[0114] Rule chain assembly: The system triggers the third assembly instruction to assemble the default cleansing rule chain. This chain includes: (1) basic format verification (ensuring it is pure numbers); (2) association verification with the "Payment ID" field (ensuring a one-to-one correspondence).

[0115] Resource allocation: The system has allocated 2 CPU cores and 2GB of memory to it. Due to its strong correlation but relatively simple computation, the system may schedule it together with the associated "Payment ID" field to a certain computing node.

[0116] For the "Name" field (score 0.48 < 0.5):

[0117] Rule chain assembly: The system triggers the first assembly instruction to assemble only a lightweight rule chain, which mainly includes: (1) null value checking and filling; (2) removing leading and trailing spaces; (3) basic character validity verification.

[0118] Resource allocation: The system has allocated it one CPU core and 1GB of memory. Due to its weak correlation, it can run independently on any computing node with available resources.

[0119] In this way, the system achieves differentiated and refined processing of tasks of varying difficulty.

[0120] It should be noted that, regarding the setting of thresholds in specific implementation scenarios, the embodiments mention fixed first and second judgment thresholds. In practical applications, these thresholds can also be dynamically adjusted. For example, the thresholds can be automatically adjusted based on the current overall load of the cluster; when resources are strained, the thresholds can be increased to reduce the number of deep cleaning tasks; or the thresholds can be fine-tuned based on the success rate of historical cleaning tasks to better suit the actual business scenario. As long as the thresholds serve the purpose of "making decisions based on scoring and grading," they fall within the protection scope of this solution.

[0121] Regarding the triggering condition for collaborative constraints, the embodiment uses "the weight of the associated edge is higher than a preset weight judgment value" as the triggering condition for collaborative constraints. This condition can be extended to more complex logic. For example, the sum of the weight of the associated edge and the dynamic semantic complexity scores of the two fields can be considered, and collaborative constraints are only triggered when both are high, so as to avoid unnecessarily occupying a large amount of resources of the same node for two simple but strongly related fields. This composite judgment logic is also a reasonable extension of the present invention.

[0122] Regarding the dimensions of resource allocation, in addition to the number of CPU cores and memory capacity, distributed computing resources can also include other dimensions such as network bandwidth and disk I / O. The system can allocate corresponding resource quotas based on the specific characteristics of the rule chain (such as whether it involves a large amount of disk read / write). This multi-dimensional resource allocation strategy is also covered by this solution. All the above optional solutions are within the scope of protection of this application.

[0123] Step S103: Perform data cleaning based on the distributed computing resource configuration, and correct the dynamic semantic complexity score according to the actual processing time during the execution process.

[0124] This step comprises two tightly coupled sub-processes: execution and correction. The system first executes the cleaning task strictly according to the resource configuration determined in step S102, and then compares the actual feedback generated during the execution process (i.e., the actual processing time) with the theoretical expectation, thereby calibrating the core parameter driving the entire process—the dynamic semantic complexity score—in real time.

[0125] Data cleaning is performed based on resource configuration: In distributed computing frameworks (such as Spark and Flink), the cleaning task for each field node is submitted as an independent job. This job is assigned computing resources (such as the number of CPU cores and memory size) tailored to it in step S102. Simultaneously, if coordination constraints exist, multiple related tasks are scheduled to the same physical node to leverage the advantages of localized computing and reduce network transmission overhead.

[0126] The system adjusts the score based on actual processing time: Before formal execution, the system performs a small-scale benchmark sampling for each cleaning task to calculate a basic baseline processing time (T0). Then, the system linearly maps the current dynamic semantic complexity score (Sm) to a preset score range (e.g., 0-10) and calculates the expected processing time (Te) using the formula Te=T0×(1+Sm / 10). This formula reflects the reasonable assumption that a higher score corresponds to a longer expected processing time.

[0127] During task execution, the system accurately records the actual processing time. After the task is completed, the system compares the actual time with the expected time Te. If the actual time exceeds an acceptable range (i.e., greater than Te×(1+δ), where δ is a preset deviation percentage, such as 10% or 20%), the system determines that the initial dynamic semantic complexity score failed to accurately reflect the true computational difficulty.

[0128] Once a significant deviation is detected, the system immediately triggers a score correction instruction. In response, the system increases the dynamic semantic complexity score of that field node by a preset increment (e.g., +0.1 or +0.05) and writes this corrected score back into the data feature map, overwriting the old value. This corrected score will immediately be used to guide the assembly of the cleaning rule chain and resource allocation for the next batch of data for that field, enabling the system to quickly adapt to changes in data patterns or deficiencies in the initial assessment.

[0129] For example, continuing with the previous example and focusing on the "address" field:

[0130] Execution phase: The system allocated 4 cores and 8GB of memory to the "address" field and scheduled it, along with the "province" and "city" fields, to compute node A to perform deep semantic cleaning.

[0131] Benchmark and Expectations: The system first uses 1% of the data sample for benchmark testing, and the benchmark processing time T0 is measured to be 10 seconds; the current dynamic semantic complexity score is 0.83, which maps to 8.3 points in the 0-10 range; according to the formula Te=10╳(1+8.3 / 10)=18.3 seconds, the system expects the processing time of the full data to be about 18.3 seconds.

[0132] Actual Execution and Correction: After the full data cleaning task was completed, the system recorded the actual processing time as 25 seconds. The set deviation ratio δ was 20%, so the maximum allowed processing time was 18.3 × 1.2 ≈ 22 seconds. Since 25 seconds > 22 seconds, the system determined that there was a significant deviation and triggered a score correction instruction. The system increased the dynamic semantic complexity score of the "address" field from 0.83 by one incremental step (e.g., 0.05) to 0.88 and updated it in the data feature map.

[0133] Subsequent impact: When the next batch of data containing the "address" field arrives, the system will use a higher score of 0.88 to reassess its difficulty. This may lead to the allocation of more computing resources (such as 6 cores and 12GB) or the addition of stricter optimization strategies to the rule chain, thereby ensuring that the processing time for the next iteration is closer to the expected result and avoiding a recurrence of performance bottlenecks.

[0134] It should be noted that, regarding the bidirectional nature of the correction direction in specific implementation scenarios, the examples mainly describe the situation where the score is increased due to excessive processing time. In practical applications, the system can also be designed with a downward adjustment mechanism. For example, if the actual processing time is much lower than expected (e.g., less than Te×(1-δ')), the initial score can be considered too high, resulting in resource waste. In this case, the score can be appropriately reduced to free up excess resources in subsequent tasks. This bidirectional correction mechanism can more comprehensively optimize system efficiency.

[0135] Regarding the benchmark sampling strategy, benchmark sampling can adopt a fixed proportion (such as 1%), or a more intelligent strategy. For example, the sampling proportion can be dynamically adjusted according to the historical data volume or coefficient of variation of the field; or stratified sampling can be used to ensure that the samples can cover various typical value patterns of the field. Any sampling method that can effectively and quickly estimate the benchmark processing time is within the scope of protection of this scheme.

[0136] Regarding the dynamic adjustment of the correction step size, the preset incremental step size can be fixed or dynamic. For example, the step size can be determined based on the deviation between the actual and expected time; the larger the deviation, the larger the step size, to achieve faster convergence. Alternatively, the concept of a learning rate can be introduced, gradually decreasing the step size as the number of corrections increases, so that the score eventually stabilizes near an optimal value. All of the above options are within the scope of protection of this application.

[0137] Step S104: Update the historical feedback coefficients based on the quality assessment results after cleaning, and adjust the calculation weight configuration of the dynamic semantic complexity score based on the historical feedback coefficients, so that the adjusted dynamic semantic complexity score can participate in the assembly of cleaning rule chains and the configuration of distributed computing resources in the subsequent data cleaning process.

[0138] Compared to the single-execution efficiency correction in step S103, this step focuses more on the long-term, cumulative data cleaning quality, and accordingly performs in-depth optimization of the system's core decision-making logic—that is, the calculation formula for the dynamic semantic complexity score itself.

[0139] Quality assessment and historical feedback coefficient update:

[0140] After each cleaning task, the system performs a rigorous quality assessment. The duplicate value retention rate (measuring deduplication effectiveness) and semantic consistency rate (measuring semantic normalization effectiveness) directly reflect the effectiveness of the cleaning rules. The system uses these two indicators to calculate a feedback contribution value (Fi). This value is cleverly designed to reflect the principle of "the better the quality, the greater the contribution": Fi increases when the duplicate value retention rate decreases or the semantic consistency rate increases. To reflect the idea that recent feedback is more important than long-term feedback, the system introduces a decay factor (λ), which weights and sums the Fi values ​​from each iteration to obtain the historical feedback coefficient (H). This coefficient is a dynamically accumulated, comprehensive indicator that reflects the long-term cleaning effect.

[0141] Adjust the weighting configuration for score calculation:

[0142] The initial formula for calculating the dynamic semantic complexity score is S0 = α × E + β × V, where α and β are fixed initial weights. The system calculates a historical feedback weight (γ) based on the current total number of cleaning rounds (n). As the number of cleaning rounds increases, the system's trust in historical experience gradually increases, and γ also increases, but there is an upper limit γmax. The system readjusts the coefficients corresponding to the associated edge weights (E) and the semantic variation coefficient (V) (becoming α' and β'), and normalizes them together with the historical feedback weight γ to ensure that the sum of the three is 1. Finally, the dynamic semantic complexity score is updated to a new formula that incorporates historical experience: S = α' × E + β' × V + γ × H'. This means that future scores not only depend on the current structure and content features of the field, but also incorporate the experience accumulated by the system throughout its entire operating cycle.

[0143] This new score S, after weight adjustment, will immediately serve as input in step S102 of the next round (or the next batch) of data, guiding the assembly of new cleaning rule chains and resource allocation, thereby forming a continuously evolving intelligent closed loop.

[0144] For example, continuing with the previous example, and assuming the system has already completed multiple rounds of cleaning:

[0145] After the fifth round of cleaning: The system evaluated the cleaning results of the "address" field and found that the duplicate value residual rate was 5% (low), and the semantic consistency rate was 92% (high). According to the formula Fi=1-Ri+Ci, the feedback contribution value F5 of this round was calculated to be very high (e.g., 0.97).

[0146] Update the historical feedback coefficient: The system combines F5 with the contribution values ​​of the previous 4 rounds (assumed to be 0.85, 0.88, 0.90, and 0.95 respectively), and uses a decay factor λ=0.9 to perform a weighted summation to obtain a new, higher historical feedback coefficient H (e.g., 0.93).

[0147] Adjusting the weight configuration: Assuming the current total number of rounds n=5, according to the formula γ=min(γmax,1-e^(-k×n)) (let γmax=0.4, k=0.3), the historical feedback weight γ≈0.35 is calculated. System analysis reveals that for highly variable fields such as "address", the semantic consistency rate has a particularly significant impact on the final quality. Therefore, during normalization, the system may slightly decrease α' (the coefficient of the associated edge weight) while significantly increasing β' (the coefficient of the semantic variation coefficient) to further emphasize the weight of internal variability within the field. Ultimately, the new scoring formula may become S=0.25×E+0.40×V+0.35×H'.

[0148] When the next batch of "address" data arrives, even if its initial E and V values ​​are the same as before, its final dynamic semantic complexity score S will be higher than before because positive historical feedback H' has been added to the formula and V has a higher weight. This will prompt the system to allocate stronger cleaning rules (such as a more refined entity recognition model) and more computing resources to it, which is expected to further improve the semantic consistency rate to over 95%, forming a virtuous cycle.

[0149] It should be noted that, regarding the expansion of quality assessment indicators in specific implementation scenarios, although the examples emphasize "residual rate of duplicate values" and "semantic consistency rate," this does not exhaust all possibilities. Those skilled in the art can introduce other relevant quality indicators based on specific business scenarios, such as completeness (proportion of missing values) and accuracy (correctness of comparison with authoritative data sources). As long as these indicators can effectively quantify the quality of the cleaning results and be integrated into the calculation logic of the feedback contribution value Fi, they fall within the protection scope of this solution.

[0150] Regarding the calculation model for the historical feedback coefficient, the embodiment employs a weighted summation model with a decay factor. In practice, other time series models can also be used to calculate the historical feedback coefficient, such as moving averages, exponential smoothing, or even simple sliding window averages. The core idea is to ensure that the historical feedback coefficient smoothly reflects recent cleaning quality trends; the specific mathematical model can be selected and adjusted according to actual needs.

[0151] Regarding the triggering conditions for weight adjustments, weight adjustments can be performed in every round or specific triggering conditions can be set. For example, weight recalculation and adjustment can be triggered only when the change in the historical feedback coefficient H exceeds a certain threshold, or when the system detects a significant data drift. This on-demand adjustment strategy can reduce unnecessary computational overhead. All of the above options are within the scope of protection of this application.

[0152] In some embodiments of this application, in order to provide a dynamic and intelligent data source access scheduling mechanism, the effective priority of each data source can be dynamically adjusted according to its actual waiting time in the access queue, thereby achieving elastic allocation of concurrent access resources. Acquiring multi-source heterogeneous asset data includes:

[0153] Read the basic priority tag of the access data source;

[0154] The system periodically scans the waiting time of data in the access queue and performs priority over-level judgment based on the waiting time.

[0155] If the waiting time of the second priority data source is detected to be greater than the first preset time threshold but not greater than the second preset time threshold, the effective priority of the second priority data source is dynamically updated to a weight value between the first priority and the second priority.

[0156] If the waiting time of the second priority data source is detected to be greater than the second preset time threshold, the effective priority is further updated to a higher weight value than the weight value between the first priority and the second priority, and concurrent thread resources are allocated to each access data source according to the updated weight ratio.

[0157] Once the backlog queue of the corresponding access data source is cleared, a status reset command is triggered to restore the effective priority of the access data source to its basic priority label.

[0158] Wherein, the first preset time threshold is less than the second preset time threshold, and the first priority is higher than the second priority.

[0159] As described above, firstly, the system pre-configures a basic priority label for each connected data source. For example, data sources from core business systems are marked as "First Priority," and data sources from auxiliary or non-critical systems are marked as "Second Priority." Based on this, the system initiates a monitoring process that periodically scans all access queues to obtain the real-time waiting time of the data awaiting processing for each data source. Next, the system executes priority escalation logic. For a data source marked as Second Priority, the system compares its current waiting time with two preset time thresholds. The first preset time threshold (T1) represents a preliminary warning threshold, and the second preset time threshold (T2) represents a more severe backlog threshold, satisfying T1... <T2。

[0160] Level 1 Priority Boost: When the system detects that the waiting time of a second-priority data source exceeds the first preset time threshold (T1) but has not yet exceeded the second preset time threshold (T2), the system will trigger a gentle priority boost. At this time, the effective priority of the data source no longer uses its original second priority, but is dynamically updated to an intermediate weight value between the first and second priorities. For example, if the first priority weight is 10 and the second priority is 5, it can be updated to 7.

[0161] Level 2 escalation: If the backlog of the data source worsens further, and its waiting time exceeds the second preset time threshold (T2), the system will execute a stronger intervention. At this time, the effective priority of the data source will be further updated to a weight value higher than the aforementioned intermediate weight value, i.e., reaching a "higher level". This new weight value, while higher than the intermediate value, will still be lower than or equal to the first priority weight to maintain the basic order of the priority system. Further examples illustrate how this "higher level weight value" can be determined, such as: it can be a preset fixed value (e.g., first priority = 10, second priority = 5, intermediate value = 7, higher level = 9); it can be a fixed increment added to the intermediate weight value (e.g., +2); or it can be a proportional increase (e.g., intermediate value × 1.2, but the result does not exceed the first priority weight).

[0162] Subsequently, the system recalculates and allocates concurrent thread resources for data access based on the current (and possibly dynamically updated) weight ratios of all data sources. Data sources with higher weights will be allocated more concurrent threads, thereby accelerating the reading of their data and its entry into the subsequent cleaning process. Finally, to ensure the system's dynamic balance and fairness, once the backlog queue of a data source is completely cleared, the system immediately triggers a state reset command. In response to this command, the effective priority of that data source will be restored to its initially configured basic priority label, allowing it to return to a normal state in subsequent data access scheduling, awaiting the next possible overriding judgment.

[0163] In some embodiments of this application, to provide a structured and quantitative method for constructing data feature graphs, and based on this, to comprehensively and objectively calculate an initial dynamic semantic complexity score by fusing the semantic variability of the fields themselves and their correlation strength in the data network. Constructing a data feature graph containing field nodes and associated edges, and determining a dynamic semantic complexity score for the field nodes, includes:

[0164] The multi-source heterogeneous asset data is parsed and semantically annotated to determine the semantic labels, value distribution information, and cross-field relationships of the fields.

[0165] Generate the field node and the association edge based on the field semantic label and the cross-field association relationship;

[0166] Calculate the weights of associated edges based on the field matching rate and functional dependency strength between field nodes;

[0167] The semantic variation coefficient is calculated based on the number of different values ​​in the field corresponding to the field node, the total number of records, and the number of value types.

[0168] The initial dynamic semantic complexity score is calculated using the following formula:

[0169] S0 = α × E + β × V;

[0170] Where S0 represents the initial dynamic semantic complexity score, E represents the weight of the associated edge, V represents the semantic variation coefficient, α represents the initial coefficient of the weight of the associated edge, β represents the initial coefficient of the semantic variation coefficient, and α+β=1.

[0171] As mentioned above, firstly, the system performs deep analysis on the raw data obtained from multi-source heterogeneous assets. This process includes semantically tagging each field to determine its semantic labels (e.g., "Customer ID", "Order Date", "Shipping Address", etc.), analyzing its value distribution information (e.g., number of unique values, null value rate, data type, etc.), and identifying cross-field relationships between different data tables or data sources (e.g., a foreign key relationship exists between "Customer ID" in the order table and "ID" in the customer table).

[0172] Next, the system constructs a data feature map based on the above analysis results. In this map, each identified field is abstracted as a field node, and the relationships between fields are represented as edges connecting two nodes.

[0173] Then, the system quantizes and assigns values ​​to the elements in the graph:

[0174] For related edges, the system calculates their weight (E). This weight calculation considers two factors: first, the field matching rate between the two fields (e.g., the proportion of non-null foreign key fields that successfully match primary key records); and second, the strength of their functional dependency (e.g., whether the value of one field uniquely determines the value of another field). The higher the matching rate and the stronger the dependency, the greater the weight of the related edge.

[0175] For each field node, the system calculates its semantic variation coefficient (V). This coefficient is based on three core statistics for the field: the number of different values ​​(reflecting the dispersion of the data), the total number of records, and the number of value types (e.g., including text, numbers, special symbols, etc.). These three quantities together reflect the disorder and uncertainty of the data in the field. The higher the value, the more unstandardized the semantics of the field, and the greater the difficulty in cleaning it.

[0176] Finally, the system substitutes the calculated edge weight E and semantic variation coefficient V into the preset linear weighting formula S0 = α × E + β × V. Here, α and β are pre-set initial coefficients used to adjust the weight of relevance and variability in the total score, and their sum is 1 to ensure score normalization. The result S0 calculated by this formula is the initial dynamic semantic complexity score of the field node, which will serve as the initial input for subsequent steps (such as rule chain assembly, resource allocation, and feedback correction).

[0177] In some embodiments of this application, a hierarchical adaptive cleaning rule chain assembly mechanism based on dynamic semantic complexity scoring is provided. This mechanism can intelligently select and combine different cleaning rules, from lightweight to deep, according to the field complexity reflected by the scoring, to achieve "on-demand cleaning." Assembling a corresponding cleaning rule chain for the field node based on the dynamic semantic complexity scoring includes:

[0178] Read the dynamic semantic complexity score of field nodes;

[0179] If the dynamic semantic complexity score is determined to be lower than the second determination threshold, the first assembly instruction is triggered to retrieve the basic format verification rules from the rule template library and assemble them into a lightweight cleaning rule chain.

[0180] If the dynamic semantic complexity score is determined to be higher than the first determination threshold, the second assembly instruction is triggered to insert deep semantic cleaning rules into the cleaning rule chain and activate the entity boundary recognition logic and similarity merging logic.

[0181] If the dynamic semantic complexity score is determined to be no lower than the second determination threshold and no higher than the first determination threshold, then the third assembly instruction is triggered to retrieve the basic format verification rules and the candidate semantic verification rules that match the field semantic tags from the rule template library and assemble them into a default cleaning rule chain.

[0182] Wherein, the first determination threshold is greater than the second determination threshold.

[0183] As described above, the system first reads the current dynamic semantic complexity score of each field node in the data feature map. This score is a quantitative indicator calculated in step S102. Subsequently, the system compares this score with two preset judgment thresholds. The first judgment threshold (Th1) represents the boundary of high complexity, and the second judgment threshold (Th2) represents the boundary of low complexity, and Th1>Th2 are satisfied.

[0184] Low-complexity scenario (score < Th2): If the dynamic semantic complexity score of a field is lower than the second determination threshold, the system determines that the field has a clear structure, standardized values, and low cleaning difficulty. At this time, the system triggers a first assembly instruction, selects only the most basic format verification rules (for example, checking whether the date format is YYYY-MM-DD, whether the email address contains the @ symbol, etc.) from the preset rule template library, and assembles these rules into a lightweight cleaning rule chain.

[0185] High-complexity scenario (score > Th1): If the score of a field is higher than the first determination threshold, the system determines that the field has severe semantic confusion or a high degree of unstructuredness and requires in-depth processing. At this time, the system triggers a second assembly instruction. When assembling the cleaning rule chain, in addition to the possibly included basic rules, the system actively inserts deep semantic cleaning rules. Meanwhile, this instruction activates two key advanced processing logics: one is an entity boundary recognition logic (used to accurately segment entity units such as person names, addresses, and institution names from continuous text), and the other is a similarity merging logic (used to identify and merge values with the same semantics but different expressions, such as "Beijing City" and "Beijing", "Co.,Ltd." and "Limited Company").

[0186] Medium-complexity scenario (Th2 ≤ score ≤ Th1): If the score of a field is not lower than the second determination threshold and not higher than the first determination threshold, the system determines that its complexity is at a medium level. At this time, the system triggers a third assembly instruction. This instruction retrieves the basic format verification rules from the rule template library, and further combines the field semantic tag of the field (for example, "address", "product name") to match and load a set of candidate semantic verification rules (for example, the province-city-district hierarchy verification rule for the "address" field). These rules are jointly assembled into a default cleaning rule chain, whose complexity is between the lightweight level and the in-depth level.

[0187] Through the above three-level determination and assembly logic, the system can dynamically generate an optimal cleaning rule chain matching the actual complexity for each field node.

[0188] In some embodiments of the present application, to provide a dynamic rule priority sorting mechanism based on multi-dimensional evaluation indicators inside the deep semantic cleaning rule chain, so as to optimize the execution order of the rules. When assembling a cleaning rule chain containing deep semantic cleaning rules, the rule priority score is calculated according to the following formula:

[0189] P=0.5×C+0.3×M+0.2×T;

[0190] wherein P represents the rule priority score, C represents the entity boundary recognition confidence, M represents the semantic similarity score, and T represents the rule template default weight score;

[0191] The rule modules are arranged in descending order of their rule priority scores, so that the rule modules with higher priority scores are placed at the front of the cleaning rule chain and executed first.

[0192] As mentioned above, when the system assembles the cleaning rule chain, after determining that deep semantic cleaning rules need to be introduced, it calculates a rule priority score (P) for each rule module included in the rule chain (e.g., "address entity recognition module", "company name standardization module", "product model normalization module", etc.). This score (P) is obtained by weighted linear combination of three core evaluation indicators, and the specific formula is: P=0.5×C+0.3×M+0.2×T.

[0193] Where: C represents the entity boundary recognition confidence score, reflecting the reliability of the rule module in identifying accurate entity boundaries in the current field data. This value can be output by the underlying NLP model during the preprocessing stage; M represents the semantic similarity score, which measures the average semantic similarity between the field values ​​processed by the rule and the standard dictionary, and is used to evaluate the necessity and effectiveness of merging similar items in the field; T represents the default weight score of the rule template, which is a basic priority weight preset by the system for this type of rule in the rule template library, used to reflect the importance of the rule in general scenarios.

[0194] The entity boundary identification confidence score (C) is assigned the highest weight (0.5) because accurate entity boundary identification is a prerequisite for all subsequent operations (such as standardization, normalization, and association matching) in the deep semantic processing flow. If the boundary identification is incorrect, subsequent rules will be processed based on incorrect input, which can easily lead to error propagation and result distortion. Therefore, prioritizing the execution of boundary identification rules with high confidence scores plays a decisive role in the stability and accuracy of the entire cleaning chain, hence the highest weight is assigned to it.

[0195] The semantic similarity score (M) is second (0.3) because this metric reflects the degree of semantic consistency within the field data or between the field and the standard library, directly affecting the effectiveness of rules such as similar item merging and fuzzy matching. A higher similarity score implies the existence of a large number of regularizable redundancies or variations, and pre-processing can significantly improve data purity. However, its effectiveness depends on the front end correctly segmenting semantic units, so its importance is slightly lower than the confidence score for entity boundary recognition, but higher than the default weight.

[0196] The default weight score (T) of the rule template is the lowest (0.2) because T, as static prior knowledge, is used to provide a basic ranking basis when sufficient context evaluation is lacking. However, in dynamic scenarios involving specific fields, the actual data features (C and M) better reflect the applicability and urgency of the current rule. Therefore, T is only used as an auxiliary reference to avoid unreasonable execution order due to template preset deviations, hence the minimum weight is assigned.

[0197] After calculating the priority score P of all deep semantic cleaning rule modules to be executed, the system arranges these rule modules in descending order of P value. In the final generated cleaning rule chain, modules with higher priority scores are placed at the front of the chain and executed first; while modules with lower scores are placed at the back, processed after the front-end rules have performed preliminary data structuring and cleaning. This sorting strategy ensures the logical coherence and execution efficiency of the cleaning process.

[0198] In some embodiments of this application, to provide a distributed computing resource allocation method deeply coupled with the complexity and inter-field correlation of the cleaning rule chain, this method can dynamically allocate basic computing resources according to the scoring interval and can identify strongly correlated fields for collaborative scheduling. Determining the distributed computing resource configuration corresponding to the cleaning rule chain includes:

[0199] The cleaning rule chain is transformed into a distributed operator that can be executed in parallel;

[0200] Based on the scoring range in which the dynamic semantic complexity score is located, the corresponding processor core count and memory capacity configuration parameters are mapped and allocated.

[0201] If the weight of the associated edge between multiple field nodes is detected to be higher than the preset weight judgment value, a collaborative constraint instruction is triggered to distribute the cleaning tasks corresponding to the multiple field nodes to the same computing node for processing.

[0202] As mentioned above, firstly, the system logically parses and splits the assembled cleaning rule chain, transforming it into a series of distributed operators that can be executed in parallel (e.g., Map, Filter, Join operations in computing engines such as Spark or Flink), so as to facilitate scheduling and execution in a distributed cluster.

[0203] Next, the system determines which preset scoring interval a rule falls into based on the dynamic semantic complexity score of the field node corresponding to the cleaning rule chain. Each scoring interval is pre-mapped with a set of specific resource configuration parameters, including the number of processor cores and memory capacity to be allocated. The score S ∈ [0, 1] is divided into three levels of intervals (lightweight / default / deep), and the system allocates corresponding computing resources to the operator accordingly. An example scoring-resource configuration mapping table is as follows:

[0204] Table 1

[0205]

[0206] Finally, the system checks whether the field node has strong associations with other field nodes. Specifically, if the weight of the association edges between multiple field nodes is detected to be higher than a preset weight judgment value (this value is used to distinguish between ordinary associations and strong logical coupling), the system will trigger a cooperative constraint instruction. In response to this instruction, the resource scheduler will force all cleaning tasks (i.e., their respective distributed operators) corresponding to these strongly associated fields to be distributed to the same physical or virtual computing node for processing when distributing tasks. This cooperative scheduling strategy effectively avoids the latency and bandwidth consumption of cross-node data retrieval, ensuring the efficiency and atomicity of associated data processing.

[0207] In some embodiments of this application, to introduce a closed-loop feedback mechanism, the dynamic semantic complexity score is dynamically corrected and calibrated by monitoring the actual execution time of the cleaning task and comparing it with the expected execution time predicted based on the initial score. Correcting the dynamic semantic complexity score based on the actual processing time during execution includes:

[0208] Before performing data cleaning, the baseline processing time of the cleaning task corresponding to the field node is calculated by benchmark sampling.

[0209] The normalized dynamic semantic complexity score is linearly mapped to a preset score range to obtain the current mapped score, and the expected processing time is calculated according to the following formula:

[0210] Te = T0 × (1 + Sm / M);

[0211] Where Te represents the expected processing time, T0 represents the baseline processing time, Sm represents the current mapping score, and M is the maximum span value of the preset score interval;

[0212] During the data cleaning process, the actual processing time is obtained. If the actual processing time is determined to be greater than Te×(1+δ), a scoring correction instruction is triggered, where δ represents the preset deviation ratio.

[0213] In response to the scoring correction instruction, the dynamic semantic complexity score of the corresponding field node is increased by a preset increment step, and the increased dynamic semantic complexity score is written back to the data feature map.

[0214] As mentioned above, before formally executing the data cleaning task, the system will first perform a small-scale baseline sampling of the cleaning task corresponding to the target field node (for example, processing 1% of the sample data) and accurately measure the baseline processing time (T0) required to complete the sampling task. This T0 serves as a reference benchmark for subsequent calculations of expected processing time.

[0215] Subsequently, the system normalizes the current dynamic semantic complexity score (e.g., scales it to the 0-100 range) and linearly maps it to a preset score range (e.g., 0-10). The maximum span of this preset score range is 10, resulting in the current mapped score (Sm). Next, the system uses the formula Te = T0 × (1 + Sm / 10) to calculate the expected processing time (Te) for this full-scale cleaning task. This formula shows that the expected processing time is based on the baseline time T0 and increases linearly with the mapping score Sm.

[0216] During the cleaning task, the system monitors and records the actual processing time in real time. After the task is completed, the system compares the actual processing time with the expected processing time Te. If the actual processing time is determined to be greater than Te×(1+δ) (where δ is a preset deviation percentage to tolerate normal fluctuations, for example, δ=0.2 means that a 20% positive deviation is allowed), it indicates that the actual processing difficulty is significantly higher than expected, and the system will trigger a score correction instruction.

[0217] In response to this instruction, the system will increase the dynamic semantic complexity score of the corresponding field node by a preset increment (e.g., by 5 or 10 points). This increase compensates for the initial underestimation of the score. Finally, the system writes the increased score back to the data feature map, updating the metadata of the field node for use in subsequent data processing.

[0218] In some embodiments of this application, a dynamic tuning mechanism for core cleaning parameters is established, driven by the post-cleaning quality assessment results (especially the residual rate of duplicate values). The post-cleaning quality assessment results include the residual rate of duplicate values ​​and the semantic consistency rate.

[0219] After obtaining the residual rate of the duplicate values, a similarity threshold adjustment judgment is performed;

[0220] If the residual rate of the duplicate value is determined to be greater than the preset residual rate threshold, a threshold reduction instruction is triggered to reduce the text similarity threshold used to determine whether two text segments have performed a synonym merging operation, so as to relax the determination conditions for synonym text merging.

[0221] The lowered text similarity threshold is written back to the rule generation engine so that the lowered text similarity threshold can participate in the assembly of deep semantic cleaning rules for subsequent batches of data.

[0222] As mentioned above, after completing one round of data cleaning, the system performs a quality assessment, with two core metrics being the duplicate value residual rate (the proportion of duplicate or nearly duplicate records that still exist in the cleaned dataset and should be merged) and the semantic consistency rate. Upon obtaining the duplicate value residual rate for this cleaning, the system immediately performs a similarity threshold adjustment judgment. Specifically, the system compares this residual rate with a preset residual rate threshold.

[0223] If the residual rate of duplicate values ​​exceeds the preset threshold, it indicates that the current cleaning strategy has failed to effectively eliminate duplicate data. The root cause is likely that the text similarity threshold used to determine whether two text segments are synonyms is set too high, causing many semantically identical but slightly different texts to fail to be merged. In this case, the system will trigger a threshold reduction instruction.

[0224] In response to this instruction, the system will lower the text similarity threshold by a preset adjustment step (for example, from 0.85 to 0.80) to relax the criteria for merging synonymous texts, so that more similar text fragments can be included in the merging scope.

[0225] Finally, the system writes this adjusted text similarity threshold back to the rule generation engine. Subsequently, when assembling deep semantic cleaning rules for subsequent batches of data, the rule generation engine will use this updated threshold to configure the "similarity merging logic" module, thus enabling the optimized parameters to participate in future cleaning tasks.

[0226] In some embodiments of this application, in order to construct an adaptive scoring weight adjustment system that integrates historical cleaning quality feedback, quantifies the effectiveness of each cleaning iteration, and uses this historical experience as a new core factor to dynamically reconstruct the calculation formula for the dynamic semantic complexity score, the following steps are taken: The historical feedback coefficients are updated based on the quality assessment results after cleaning, and the calculation weight configuration of the dynamic semantic complexity score is adjusted based on these historical feedback coefficients.

[0227] Obtain the residual rate of duplicate values ​​and the semantic consistency rate for the corresponding cleaning round;

[0228] The feedback contribution value for the corresponding cleaning round is calculated using the following formula:

[0229] Fi = 1 - Ri + Ci;

[0230] Where Fi represents the feedback contribution value corresponding to the i-th round of cleaning, Ri represents the normalized duplicate value residual rate after the i-th round of cleaning, and Ci represents the normalized semantic consistency rate after the i-th round of cleaning.

[0231] The feedback contribution value is configured to increase when the duplicate value residual rate decreases and / or the semantic consistency rate increases, and decrease when the duplicate value residual rate increases and / or the semantic consistency rate decreases.

[0232] Calculate the historical feedback coefficient based on the feedback contribution value and attenuation factor of each cleaning round:

[0233] H = Σ[λ^(ni) × Fi];

[0234] Where H represents the historical feedback coefficient, λ represents the decay factor, n represents the current total round, and i represents the historical cleaning round;

[0235] The historical feedback coefficients are normalized to obtain the normalized historical feedback coefficients H'.

[0236] Calculate historical feedback weights based on the total number of cleaning rounds:

[0237] γ=min(γmax,1-e^(-k×n));

[0238] Where γ represents the historical feedback weight, γmax represents the preset weight upper limit, e represents the natural constant, and k represents the growth constant;

[0239] Based on the residual rate of repeated values ​​and the semantic consistency rate, adjust the corresponding coefficients of the associated edge weights and the corresponding coefficients of the semantic variation coefficients, and perform normalization processing on the adjusted corresponding coefficients of the associated edge weights, the corresponding coefficients of the semantic variation coefficients, and the historical feedback weights so that the sum of the three is 1.

[0240] The updated dynamic semantic complexity score is calculated using the following formula:

[0241] S = α' × E + β' × V + γ × H';

[0242] Where S represents the updated dynamic semantic complexity score, α' represents the adjusted coefficient of the associated edge weight, β' represents the adjusted coefficient of the semantic variation coefficient, E represents the associated edge weight, V represents the semantic variation coefficient, and H' represents the normalized historical feedback coefficient.

[0243] As mentioned above, after each round of data cleaning is completed, the system first obtains two core quality assessment metrics for that round: the residual rate of duplicate values ​​and the semantic consistency rate. These two metrics are then normalized to obtain normalized Ri and Ci.

[0244] Next, the system calculates the feedback contribution value (Fi) of the i-th round of cleaning according to the formula Fi=1-Ri+Ci. This design ensures that when the cleaning effect is good (i.e., Ri is small and Ci is large), the Fi value increases; conversely, when the cleaning effect is poor (i.e., Ri is large and Ci is small), the Fi value decreases, thus accurately reflecting the quality of this round of cleaning.

[0245] The system then integrates the feedback contributions from all historical rounds. To reflect the principle that "recent feedback is more important than long-term feedback," the system introduces a decay factor (λ, 0 < λ < 1) and calculates the historical feedback coefficient (H) according to the formula H = Σ[λ^(ni) × Fi]. Here, n is the current total number of rounds, and i is the historical round number. This formula assigns higher weight to more recent rounds.

[0246] After calculating H, the system normalizes it to obtain the normalized historical feedback coefficient (H'), ensuring that its numerical range is consistent with other features.

[0247] Simultaneously, the system needs to determine the influence of H' in the final score, namely the historical feedback weight (γ). This weight is not fixed but increases with the accumulation of cleaning experience. The system calculates it according to the formula γ=min(γmax,1-e^(-k×n)), where γmax is the preset upper limit of the weight, and k is a constant controlling the growth rate. This ensures that γ is small in the initial stage, and gradually increases and approaches γmax as the number of cleaning rounds n increases.

[0248] Next, the system will dynamically adjust the coefficients corresponding to the associated edge weights (α) and the semantic variation coefficients (β) in the original scoring formula based on the specific performance of the duplicate value residual rate (Ri) and semantic consistency rate (Ci) in this round, respectively, to obtain the adjusted α' and β'. For example, if it is found that high associated edge weights do not bring good cleaning results (high Ri), α' may be lowered.

[0249] Finally, the system normalizes the adjusted α', β', and the newly calculated γ to ensure that their sum is 1. Based on this, the system calculates the updated dynamic semantic complexity score (S) according to the updated formula S=α'×E+β'×V+γ×H'. This new score S integrates the inherent characteristics of the data (E,V) and the system's historical cleaning experience (H'), providing a more comprehensive and intelligent decision-making basis for rule generation and resource scheduling in the next cleaning task.

[0250] Compared with existing technologies, this application discloses a method for processing multi-source heterogeneous asset data. This method constructs a data feature map and introduces a dynamic semantic complexity score as the core metric, enabling the system to accurately quantify the cleaning difficulty of each field node. This score directly drives the assembly of the cleaning rule chain (high scores trigger deep semantic cleaning rules, low scores use lightweight rules) and the allocation of distributed computing resources (high scores allocate more CPU / memory). This dual decision-making mechanism driven by the same score fundamentally solves the problem of rule and resource mismatch, significantly improving resource utilization efficiency.

[0251] This solution not only utilizes scoring to guide initial rules and resource allocation but also incorporates a dynamic correction and optimization mechanism. On one hand, the system adjusts the dynamic semantic complexity score based on the deviation between actual and expected processing time, ensuring the score reflects the true cleaning difficulty in real time. On the other hand, the system updates historical feedback coefficients based on post-cleaning quality assessment results (such as duplicate value retention rate and semantic consistency rate) and adjusts the scoring calculation weights accordingly. This closed-loop mechanism ensures that the scoring is no longer static but continuously evolves and optimizes as the system runs, leading to increasingly accurate rule chain assembly and resource allocation.

[0252] Thanks to the aforementioned collaborative matching and closed-loop optimization mechanisms, this invention can complete higher-quality data cleaning tasks with fewer computing resources. Specifically, end-to-end processing time is significantly reduced, the rate of duplicate values ​​is greatly decreased, the semantic consistency rate is significantly improved, and the idle rate of system resources is effectively controlled. This enables the system to maintain efficient, stable, and high-quality processing capabilities even when facing massive, complex, multi-source heterogeneous asset data.

[0253] Based on the same inventive concept as the methods described above, this application also proposes a processing system for multi-source heterogeneous asset data, such as... Figure 2 The diagram shown is a structural schematic of a multi-source heterogeneous asset data processing system, which includes:

[0254] The graph scoring determination module is used to acquire multi-source heterogeneous asset data, construct a data feature graph containing field nodes and associated edges, and determine a dynamic semantic complexity score for the field nodes.

[0255] The rule resource determination module is used to assemble a corresponding cleaning rule chain for the field node based on the dynamic semantic complexity score, and determine the distributed computing resource configuration corresponding to the cleaning rule chain;

[0256] The cleaning execution and scoring correction module is used to perform data cleaning based on the distributed computing resource configuration, and to correct the dynamic semantic complexity score according to the actual processing time during the execution process.

[0257] The feedback weight adjustment module is used to update the historical feedback coefficients based on the quality assessment results after cleaning, and adjust the calculation weight configuration of the dynamic semantic complexity score based on the historical feedback coefficients, so that the adjusted dynamic semantic complexity score can participate in the assembly of cleaning rule chains and the configuration of distributed computing resources in the subsequent data cleaning process.

[0258] As described above, this multi-source heterogeneous asset data processing system aims to achieve an intelligent and adaptive data cleaning closed loop. Its core consists of four functional modules, and the specific workflow of each module is as follows:

[0259] The graph scoring module is responsible for initializing the entire process. This module first extracts metadata from asset data from multiple sources and formats (i.e., multi-source heterogeneous data). Based on this information, it constructs a data feature graph, where each data field is abstracted as a field node, and the logical dependencies, primary / foreign key relationships, or co-occurrence statistics between fields are modeled as edges connecting the nodes. Building upon this graph, the module comprehensively considers factors such as the semantic variation of the field itself and its topological structure in the graph (e.g., the weight and number of edges) to calculate and assign an initial dynamic semantic complexity score to each field node. This score serves as the quantitative basis for all subsequent decisions.

[0260] The rule resource determination module takes over the scoring results mentioned above. Its core task is to transform the abstract complexity score into a concrete execution strategy. On one hand, based on the dynamic semantic complexity score of each field node, it intelligently assembles a dedicated cleaning rule chain from a pre-set rule base. This rule chain consists of a series of sequentially executed cleaning operations (such as format standardization, null value filling, synonym merging, etc.). On the other hand, this module also needs to plan the computing resources required to execute these rules. It parses the cleaning rule chain into parallelizable distributed operators and maps and allocates the corresponding number of processor cores and memory capacity according to the interval in which the score is located. Simultaneously, if a strong correlation is detected between multiple fields (the weight of the related edge exceeds a threshold), a collaborative constraint is triggered to ensure that the cleaning tasks for these fields are scheduled to the same computing node to optimize execution efficiency.

[0261] The data cleaning execution and scoring correction module is responsible for the actual data cleaning tasks and real-time feedback. This module utilizes the distributed computing resource configuration planned by the rule resource determination module to execute data cleaning tasks in parallel on the cluster. During execution, it precisely monitors the actual processing time for each field cleaning task. After the task is completed, the module compares the actual processing time with the expected processing time predicted based on the initial score. If the actual processing time significantly exceeds the expectation (exceeding a preset deviation ratio), it is determined that the initial score underestimated the processing difficulty. In this case, a scoring correction instruction is triggered, increasing the dynamic semantic complexity score of the corresponding field node by a preset step size, and updating the corrected score back to the data feature map to guide subsequent processing.

[0262] The feedback weight adjustment module focuses on long-term system optimization and model evolution. After each cleaning task, this module obtains the quality assessment results of this cleaning, mainly including the residual rate of duplicate values ​​and the semantic consistency rate. It uses these indicators to calculate the feedback contribution value of this round and integrates the contributions of previous cleanings with a decay factor to obtain a historical feedback coefficient. Subsequently, this module determines the weight of historical feedback in the scoring model based on the total number of cleaning rounds and dynamically adjusts the calculation weight configuration of each factor (such as the weight of related edges and the semantic variation coefficient) in the original scoring formula. Finally, it generates an updated dynamic semantic complexity score that incorporates historical experience. This globally optimized new score is written back to the data feature graph and provided as input to the graph scoring determination module and the rule resource determination module, thereby participating in the assembly of the cleaning rule chain and the configuration of distributed computing resources in the next round of data cleaning, forming a complete closed loop of continuous learning and optimization.

[0263] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program goods. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program goods embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0264] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program goods according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0265] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0266] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0267] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for processing multi-source heterogeneous asset data, characterized in that, include: Acquire multi-source heterogeneous asset data, construct a data feature graph containing field nodes and associated edges, and determine a dynamic semantic complexity score for each field node; wherein, the dynamic semantic complexity score is used to characterize the cleaning difficulty of the field corresponding to the field node, and the initial dynamic semantic complexity score is calculated according to the following formula: S0 = α × E + β × V; Where S0 represents the initial dynamic semantic complexity score, E represents the associated edge weight determined based on the field matching rate and functional dependency strength between field nodes, V represents the semantic variation coefficient determined based on the number of different values ​​in the corresponding field of the field node, the total number of records, and the number of value types, α represents the initial coefficient of the associated edge weight, β represents the initial coefficient of the semantic variation coefficient, and α+β=1. Based on the dynamic semantic complexity score, assemble the corresponding cleaning rule chain for the field node, and determine the distributed computing resource configuration corresponding to the cleaning rule chain; Data cleaning is performed based on the distributed computing resource configuration, and the dynamic semantic complexity score is adjusted according to the actual processing time during the execution process. The historical feedback coefficients are updated based on the quality assessment results after cleaning, and the calculation weight configuration of the dynamic semantic complexity score is adjusted based on the historical feedback coefficients, so that the adjusted dynamic semantic complexity score can participate in the assembly of cleaning rule chains and the configuration of distributed computing resources in the subsequent data cleaning process. The quality assessment results after cleaning include the residual rate of duplicate values ​​and the semantic consistency rate. The normalized residual rate of duplicate values ​​after the i-th round of cleaning is denoted as Ri, and the normalized semantic consistency rate after the i-th round of cleaning is denoted as Ci. The feedback contribution value corresponding to the i-th round of cleaning is calculated according to the following formula: Fi = 1 - Ri + Ci; Based on the feedback contribution value and attenuation factor of each cleaning round, the historical feedback coefficient is calculated according to the following formula: H = Σ[λ^(ni) × Fi]; Where H represents the historical feedback coefficient, λ represents the decay factor, n represents the current total round, and i represents the historical cleaning round; The weight configuration for calculating the dynamic semantic complexity score includes the coefficients corresponding to the weights of associated edges, the coefficients corresponding to the semantic variation coefficients, and the weights of historical feedback.

2. The method as described in claim 1, characterized in that, The acquisition of multi-source heterogeneous asset data includes: Read the basic priority tag of the access data source; The system periodically scans the waiting time of data in the access queue and performs priority over-level judgment based on the waiting time. If the waiting time of the second priority data source is detected to be greater than the first preset time threshold but not greater than the second preset time threshold, the effective priority of the second priority data source is dynamically updated to a weight value between the first priority and the second priority. If the waiting time of the second priority data source is detected to be greater than the second preset time threshold, the effective priority is further updated to a higher weight value than the weight value between the first priority and the second priority, and concurrent thread resources are allocated to each access data source according to the updated weight ratio. Once the backlog queue of the corresponding access data source is cleared, a status reset command is triggered to restore the effective priority of the access data source to its basic priority label. Wherein, the first preset time threshold is less than the second preset time threshold, and the first priority is higher than the second priority.

3. The method as described in claim 1, characterized in that, The construction of a data feature graph containing field nodes and associated edges, and the determination of a dynamic semantic complexity score for the field nodes, includes: The multi-source heterogeneous asset data is parsed and semantically annotated to determine the semantic labels, value distribution information, and cross-field relationships of the fields. The field node and the associated edge are generated based on the field semantic label and the cross-field association relationship.

4. The method as described in claim 1, characterized in that, The step of assembling a corresponding cleaning rule chain for the field node based on the dynamic semantic complexity score includes: Read the dynamic semantic complexity score of field nodes; If the dynamic semantic complexity score is determined to be lower than the second determination threshold, the first assembly instruction is triggered to retrieve basic format verification rules from the rule template library and assemble them into a lightweight cleaning rule chain. If the dynamic semantic complexity score is determined to be higher than the first determination threshold, the second assembly instruction is triggered to insert deep semantic cleaning rules into the cleaning rule chain and activate the entity boundary recognition logic and similarity merging logic. If the dynamic semantic complexity score is determined to be no lower than the second determination threshold and no higher than the first determination threshold, then the third assembly instruction is triggered to retrieve the basic format verification rules and the candidate semantic verification rules that match the field semantic tags from the rule template library and assemble them into a default cleaning rule chain. Wherein, the first determination threshold is greater than the second determination threshold.

5. The method as described in claim 4, characterized in that, When assembling a cleaning rule chain containing deep semantic cleaning rules, the rule priority score is calculated according to the following formula: P = 0.5 × C + 0.3 × M + 0.2 × T; Where P represents the rule priority score, C represents the entity boundary recognition confidence score, M represents the semantic similarity score, and T represents the rule template default weight score. The rule modules are arranged in descending order of their rule priority scores, so that the rule modules with higher priority scores are placed at the front of the cleaning rule chain and executed first.

6. The method as described in claim 1, characterized in that, The determination of the distributed computing resource configuration corresponding to the cleaning rule chain includes: The cleaning rule chain is transformed into a distributed operator that can be executed in parallel; Based on the scoring range in which the dynamic semantic complexity score is located, the corresponding processor core count and memory capacity configuration parameters are mapped and allocated. If the weight of the associated edge between multiple field nodes is detected to be higher than the preset weight judgment value, a collaborative constraint instruction is triggered to distribute the cleaning tasks corresponding to the multiple field nodes to the same computing node for processing.

7. The method as described in claim 1, characterized in that, The step of correcting the dynamic semantic complexity score based on the actual processing time during execution includes: Before performing data cleaning, the baseline processing time of the cleaning task corresponding to the field node is calculated by benchmark sampling. The normalized dynamic semantic complexity score is linearly mapped to a preset score range to obtain the current mapped score, and the expected processing time is calculated according to the following formula: Te = T0 × (1 + Sm / M); Where Te represents the expected processing time, T0 represents the baseline processing time, Sm represents the current mapping score, and M is the maximum span value of the preset score interval; During the data cleaning process, the actual processing time is obtained. If the actual processing time is determined to be greater than Te×(1+δ), a scoring correction instruction is triggered, where δ represents the preset deviation ratio. In response to the scoring correction instruction, the dynamic semantic complexity score of the corresponding field node is increased by a preset increment step, and the increased dynamic semantic complexity score is written back to the data feature map.

8. The method as described in claim 4 or 5, characterized in that, The quality assessment results after cleaning include the rate of residual duplicate values ​​and the semantic consistency rate. After obtaining the residual rate of the duplicate values, a similarity threshold adjustment judgment is performed; If the residual rate of the duplicate value is determined to be greater than the preset residual rate threshold, a threshold reduction instruction is triggered to reduce the text similarity threshold used to determine whether two text segments have performed a synonym merging operation, so as to relax the determination conditions for synonym text merging. The lowered text similarity threshold is written back to the rule generation engine so that the lowered text similarity threshold can participate in the assembly of deep semantic cleaning rules for subsequent batches of data.

9. The method as described in claim 1, characterized in that, The step of updating the historical feedback coefficients based on the quality assessment results after cleaning, and adjusting the calculation weight configuration of the dynamic semantic complexity score based on the historical feedback coefficients, includes: The historical feedback coefficients are normalized to obtain the normalized historical feedback coefficients H'. The historical feedback weight is calculated based on the total number of cleaning cycles using the following formula: γ=min(γmax,1-e^(-k×n)); Where γ represents the historical feedback weight, γmax represents the preset weight upper limit, e represents the natural constant, and k represents the growth constant; Based on the residual rate of repeated values ​​and the semantic consistency rate, adjust the corresponding coefficients of the associated edge weights and the corresponding coefficients of the semantic variation coefficients, and perform normalization processing on the adjusted corresponding coefficients of the associated edge weights, the corresponding coefficients of the semantic variation coefficients, and the historical feedback weights so that the sum of the three is 1. The updated dynamic semantic complexity score is calculated using the following formula: S = α' × E + β' × V + γ × H'; Where S represents the updated dynamic semantic complexity score, α' represents the adjusted coefficient of the associated edge weight, β' represents the adjusted coefficient of the semantic variation coefficient, E represents the associated edge weight, V represents the semantic variation coefficient, and H' represents the normalized historical feedback coefficient.

10. A system for processing multi-source heterogeneous asset data, used to perform the method as described in any one of claims 1 to 9, characterized in that, The system includes: The graph scoring module is used to acquire multi-source heterogeneous asset data, construct a data feature graph containing field nodes and related edges, and determine a dynamic semantic complexity score for the field nodes. The dynamic semantic complexity score is used to characterize the cleaning difficulty of the field corresponding to the field node. The initial dynamic semantic complexity score S0 = α × E + β × V, where E represents the weight of the related edge determined based on the field matching rate and functional dependency strength between field nodes, V represents the semantic variation coefficient determined based on the number of different values ​​in the field corresponding to the field node, the total number of records, and the number of value types, α represents the initial coefficient of the related edge weight, β represents the initial coefficient of the semantic variation coefficient, and α + β = 1. The rule resource determination module is used to assemble a corresponding cleaning rule chain for the field node based on the dynamic semantic complexity score, and determine the distributed computing resource configuration corresponding to the cleaning rule chain; The cleaning execution and scoring correction module is used to perform data cleaning based on the distributed computing resource configuration, and to correct the dynamic semantic complexity score according to the actual processing time during the execution process. The feedback weight adjustment module is used to update the historical feedback coefficients based on the quality assessment results after cleaning, and adjust the calculation weight configuration of the dynamic semantic complexity score based on the historical feedback coefficients, so that the adjusted dynamic semantic complexity score can participate in the assembly of cleaning rule chains and the configuration of distributed computing resources in the subsequent data cleaning process. The quality assessment results after cleaning include the duplicate value residual rate and semantic consistency rate. The historical feedback coefficient is determined based on the feedback contribution value and attenuation factor determined in each cleaning round based on the duplicate value residual rate and semantic consistency rate. The calculation weight configuration of the dynamic semantic complexity score includes the coefficient corresponding to the associated edge weight, the coefficient corresponding to the semantic variation coefficient, and the historical feedback weight.

Images