Distributed data unified management and intelligent scheduling method based on data weaving

Abstract

The present application relates to a distributed data unified management and intelligent scheduling method based on data weaving, which collects real-time state data of nodes and generates node capability portraits; constructs a dynamic association matrix of metadata and node capability portraits; establishes a multi-dimensional scheduling evaluation model, inputs business data dependency relationship, performance data in node capability portraits and node future load predicted by the LSTM model, and outputs an initial scheduling scheme; presets static threshold and dynamic prediction threshold as scheduling optimization trigger conditions, optimizes the initial scheme through reinforcement learning algorithm to obtain optimized scheduling decision; sends scheduling instructions containing priority identifiers to corresponding nodes, collects feedback data such as response delay and error rate in real time, updates the association strength of the dynamic association matrix and optimizes the multi-dimensional scheduling evaluation model parameters. The present application realizes unified management of distributed data, improves the intelligence and accuracy of scheduling, dynamically adapts to node state changes, and effectively guarantees the efficiency and reliability of data processing.

Description

Technical Field

[0001] This invention belongs to the field of data processing and analysis technology, and in particular relates to a distributed unified data management and intelligent scheduling method based on data weaving. Background Technology

[0002] With the rapid development of big data, cloud computing, and IoT technologies, enterprise data is exhibiting significant characteristics such as multi-source heterogeneity, distributed storage, and dynamic growth. To meet the storage and efficient processing needs of massive amounts of data, distributed data architecture has become the mainstream choice, with data being stored in different physical locations and on different types of distributed nodes, forming a complex distributed data ecosystem.

[0003] In existing distributed data management solutions, node access often relies on fixed protocol adaptation, failing to dynamically identify the communication protocols of various node types. This results in low efficiency and poor compatibility for heterogeneous node access. Furthermore, the characterization of node capabilities is limited to static information such as hardware configuration, neglecting to consider historical data processing performance and real-time status, thus affecting the matching of data requirements with node capabilities in subsequent scheduling decisions.

[0004] Metadata is the core of distributed data management, but current technologies mostly only collect basic data attributes, failing to integrate key information such as data lineage, business tags, and access permissions. This limits the value of metadata in supporting business decisions. More importantly, existing solutions do not establish a correlation between metadata and node capabilities, preventing metadata from providing real-time and accurate matching criteria for scheduling decisions, further exacerbating resource waste and processing delays.

[0005] Existing scheduling schemes are mostly based on static threshold triggers, lacking the ability to predict the future load status of nodes. They cannot avoid node overload caused by sudden load surges in the future, resulting in delayed scheduling decisions and a high risk of service interruption. At the same time, the scheduling evaluation dimension is singular, focusing only on node load balancing, without comprehensively considering the multi-dimensional objectives of business data dependencies, scheduling costs, and business reliability. This can easily lead to situations where business response speed is sacrificed in pursuit of load balancing, or data consistency is ignored in order to reduce transmission costs, resulting in unintended consequences.

[0006] Therefore, this invention aims to provide a distributed data unified management and intelligent scheduling method based on data weaving to solve the above-mentioned technical problems. Summary of the Invention

[0007] The purpose of this invention is to provide a distributed data unified management and intelligent scheduling method based on data weaving, so as to solve the above-mentioned technical problems existing in the prior art.

[0008] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0009] The distributed data unified management and intelligent scheduling method based on data weaving includes the following steps:

[0010] S1: Identify the communication protocol type of distributed data nodes in real time through a preset protocol feature library, collect real-time status data of nodes based on the communication protocol type, and generate a node capability profile based on node hardware configuration, storage capacity, and historical data processing performance.

[0011] S2: Based on node capability profiles and real-time node status data, obtain metadata including data lineage, business tags, and access permissions; construct a dynamic correlation matrix between metadata and node capability profiles, where the matrix element values ​​represent the correlation strength between metadata.

[0012] S3: Establish a multi-dimensional scheduling evaluation model, inputting business data dependencies, performance in node capability profiles, and load prediction values ​​of nodes within a specified future time period obtained through the LSTM model, and outputting an initial scheduling scheme;

[0013] S4: Preset scheduling optimization trigger conditions, including static thresholds and dynamic prediction thresholds. When any trigger condition is met, the initial scheduling scheme is optimized through reinforcement learning algorithm to obtain the optimized scheduling decision.

[0014] S5: Based on the optimized scheduling decision, send scheduling instructions containing priority identifiers to the corresponding nodes, collect feedback data including response latency and error rate in real time, update the correlation strength in the dynamic correlation matrix based on the feedback data, and optimize the parameters of the multi-dimensional scheduling evaluation model.

[0015] Preferably, the specific process of generating a node capability profile based on node hardware configuration, storage capacity, and historical data processing performance in step S1 is as follows:

[0016] S11: Collect multi-source data including hardware configuration data, storage capacity data, and historical data processing performance, and perform preprocessing;

[0017] S12: Construct a three-level indicator system based on multi-source data as the core dimension of capability profiling:

[0018] Primary indicators: hardware infrastructure capabilities, storage service capabilities, and data processing capabilities;

[0019] Secondary indicators:

[0020] Basic hardware capabilities include: computing power, memory support capacity, and network transmission capacity;

[0021] Storage service capabilities include: effective storage capacity and storage I / O performance;

[0022] Data processing capabilities include: task throughput, response sensitivity, and fault tolerance.

[0023] S13: Normalize the secondary indicators and map them to 0-10 points; and assign weights to the primary and secondary indicators.

[0024] S14: Capability portrait generation:

[0025] Calculate the overall capability score: Overall capability score = Σ(secondary indicator score × corresponding weight), forming a node overall capability score of 0-10;

[0026] Generate a multi-dimensional profile: with comprehensive capability value as the core, and additional feature labels, including high-throughput nodes, low-latency nodes, and large-capacity storage nodes.

[0027] Preferably, the specific process of step S2 is as follows:

[0028] S21: Collect metadata related to data lineage, business tags, and access permissions:

[0029] Data lineage metadata: records the entire chain of data flow from generation to destruction, including the mapping relationship of "data source → processing process → target node";

[0030] Business tag metadata: Tags that characterize the business attributes of data, including business domain, data theme, data level, and timeliness requirements;

[0031] Access permission metadata: The permission rules that control data access, including the access subject, permission type, permission effective time, and data scope;

[0032] S22: Construction of a dynamic association matrix between metadata and node capability profiles:

[0033] Based on the above structured metadata and node capability profiles, a two-dimensional dynamic association matrix is ​​constructed, with matrix element values ​​ranging from 0 to 100, to quantify the association strength between metadata and node capabilities.

[0034] Preferably, the specific process of step S22 is as follows:

[0035] S221: Definition of matrix dimension:

[0036] Matrix row dimension: metadata entries;

[0037] Matrix column dimension: Core indicators for profiling node capabilities;

[0038] Matrix element M[i,j]: the correlation strength between the i-th metadata item and the j-th node capability index;

[0039] S222: For each piece of metadata i and node capability index j, perform an initial calculation of the association strength based on the matching degree between metadata features and node capabilities;

[0040] S223: Dynamically adjust the association strength based on the real-time status of nodes and the frequency of business interactions. The matrix element values ​​are dynamically updated every 5 minutes.

[0041] Preferably, step S2 also includes an incremental update mechanism, as follows:

[0042] S224: Set the trigger condition for incremental updates of matrix elements:

[0043] Metadata has changed;

[0044] The node's real-time status data exceeds the preset threshold;

[0045] The frequency of business interactions has reached the adjustment threshold;

[0046] S225: When the incremental update trigger condition is met, only the association strength between the affected metadata entries and the node capability indicators is recalculated, while other elements remain unchanged.

[0047] Preferably, the specific process in step S3 is as follows:

[0048] S31: Parse the business data dependencies from the dynamic association matrix and transform them into two types of quantitative features: dependency strength matrix and dependency type label;

[0049] S32: Extract core performance indicators from the node capability profile and standardize them into feature values ​​of 0-100:

[0050] Static performance characteristics: computing power, storage I / O performance, network transmission capability, fault tolerance;

[0051] Dynamic performance characteristics: current remaining resources, current task concurrency;

[0052] S33: Trend Feature Transformation of LSTM Node Load Forecast Values: Based on the load forecast data for the next 10-30 minutes output by the LSTM model, trend features are extracted:

[0053] Peak load and time point: The highest load value and the time point of occurrence within the prediction period;

[0054] Load growth slope: Calculate the rate of load change over 10-30 minutes;

[0055] S34: Construct a three-dimensional evaluation model that includes dependency adaptability, performance matching degree, and load risk controllability, and calculate the comprehensive adaptability score of each node:

[0056] Dimension 1: Assess dependency adaptability, weight 30%, with core metrics including dependency localization rate and real-time dependency response capability;

[0057] Dimension 1 score = (localization rate × 0.6 + real-time dependency response capability × 0.4) × 30%;

[0058] Dimension 2: Assess performance matching degree, weight 40%, core indicators include capability requirement matching degree and resource margin suitability;

[0059] Dimension 2 score = (Capability requirement matching degree × 0.7 + Resource margin suitability degree × 0.3) × 40%;

[0060] Dimension 3: Assess the controllability of load risk, with a weight of 30%. Key indicators include predicted load compliance rate and load growth buffer.

[0061] Dimension 3 score = (Predicted load compliance rate × 0.6 + Load growth buffer × 0.4) × 30%.

[0062] Node overall score calculation: Overall score = Dimension 1 score + Dimension 2 score + Dimension 3 score, score range 0-100;

[0063] S35: Select candidate nodes that meet the criteria based on the overall score, and sort them by scheduling adaptability:

[0064] Screening criteria: Overall score ≥ 70 points;

[0065] Priority sorting rule: Sort by overall score in descending order;

[0066] S36: Based on the sorted candidate nodes, generate an initial scheduling scheme that includes data sharding, routing paths, and resource reservation.

[0067] Preferably, in step S4, the initial scheduling scheme is optimized using a reinforcement learning algorithm. The specific process for obtaining the optimized scheduling decision is as follows:

[0068] S41: Monitor the static threshold and the dynamic prediction threshold in real time, and determine whether the optimization triggering conditions are met;

[0069] S42: Define the core elements of a reinforcement learning model:

[0070] State space: includes node state, initial plan execution state, business request characteristics, data dependency state, fault risk indicators, and constraint boundaries;

[0071] Action space: quantifiable adjustment operations on the initial scheduling scheme, including data fragmentation adjustment, cross-node path adjustment, and resource reservation adjustment;

[0072] Reward function: quantifies the quality of an action, the formula is as follows:

[0073] Reward value = α × load balancing reward + β × response time reward + γ × resource utilization reward + δ × reliability reward - λ × adjustment cost;

[0074] S43: Using the initial scheduling scheme as the initial strategy, the system optimizes through multiple rounds of "sampling-evaluation-update". When the average reward value of three consecutive iterations increases by ≤1%, or the number of iterations reaches the upper limit, or the core indicators of the current scheme are better than the preset optimal threshold, the iteration stops.

[0075] Preferably, the specific process of updating the association strength in the dynamic association matrix based on feedback data and optimizing the parameters of the multi-dimensional scheduling evaluation model in step S5 is as follows:

[0076] S51: Adjust the correlation strength based on the preset mapping table between feedback data and node capability indicators;

[0077] S52: Optimize the parameters of the multi-dimensional scheduling evaluation model based on feedback data, including the weights of each evaluation dimension and the quantitative benchmark of the indicators.

[0078] The beneficial effects of this invention include:

[0079] 1. By dynamically identifying heterogeneous node protocols, compatible access for multiple types of distributed nodes is achieved, solving the problem of poor compatibility with traditional fixed protocols. Simultaneously, a three-level indicator system for node capability profiling is constructed, covering not only static hardware characteristics but also integrating dynamic performance dimensions such as task throughput, response sensitivity, and fault tolerance. Capability quantification is achieved through weighted calculation and feature labeling. This upgrades the description of node capabilities from a single static parameter to a multi-dimensional dynamic profile, ensuring accurate matching between data requirements and node capabilities in subsequent scheduling decisions, avoiding resource mismatch, and improving node resource utilization.

[0080] 2. By collecting comprehensive metadata encompassing data lineage, business tags, and access permissions, the system overcomes the information gaps inherent in traditional metadata, which only contains basic attributes. This allows metadata to accurately reflect business needs and data flow relationships. Furthermore, the constructed dynamic association matrix, through initial matching calculations and dynamic adjustments, coupled with an incremental update mechanism, upgrades association relationships from static binding to real-time adaptive adaptation. This design transforms metadata from isolated data attribute records into a scheduling decision-making basis that can correlate node capabilities in real time.

[0081] 3. The constructed three-dimensional evaluation model of dependency adaptability, performance matching degree, and load risk controllability breaks through the limitation of traditional scheduling focusing only on the single dimension of load balancing: it improves business adaptability by analyzing business data dependencies, enhances decision-making foresight by combining LSTM model to predict node load 10-30 minutes in the future, and ensures a balance between matching degree, risk, and performance by screening candidate nodes through comprehensive scores. This design upgrades scheduling decision-making from passively responding to thresholds to proactively predicting and optimizing.

[0082] 4. By employing a dual-layer triggering condition of preset static thresholds and dynamic prediction thresholds, the scheduling optimization is ensured to respond promptly to real-time overload and future risks. Furthermore, a reinforcement learning algorithm is used, with node status, scheme execution status, and business request characteristics as the state space, and sharding adjustment, path switching, and resource reservation as executable actions. The initial scheme is iteratively optimized through sampling, evaluation, and updating, and the merits of these actions are quantified using load balancing, response time, resource utilization, and reliability as reward functions. This upgrades the scheduling scheme from one-time generation to dynamic iterative optimization, improving scheduling adaptation efficiency under sudden business scenarios.

[0083] 5. By collecting real-time feedback data such as interface response latency and error rate, two major optimizations are driven: first, updating the correlation strength of the dynamic correlation matrix; and second, optimizing the parameters of the multi-dimensional scheduling evaluation model, forming a closed loop of scheduling execution → service feedback → model optimization → scheduling iteration. This enables the system to have self-learning and self-optimization capabilities, avoiding the problem of scheduling logic becoming disconnected from business logic after long-term operation of traditional technologies. The model parameters can be adaptively adjusted to the optimal state through feedback data. Attached Figure Description

[0084] Figure 1 This is a flowchart illustrating the distributed unified data management and intelligent scheduling method based on data weaving of the present invention.

[0085] Figure 2 This is a schematic diagram of the two-dimensional dynamic correlation matrix of the present invention.

[0086] Figure 3 This is a schematic diagram illustrating the principle of the multi-dimensional scheduling evaluation model of the present invention. Detailed Implementation

[0087] The following is in conjunction with the appendix Figures 1-3 The present invention will be further described in detail below:

[0088] See appendix Figure 1 As shown, the distributed data unified management and intelligent scheduling method based on data weaving includes the following steps:

[0089] S1: Real-time identification of the communication protocol types of distributed data nodes using a pre-defined protocol feature library, including structured database protocols, unstructured storage protocols, and streaming data transmission protocols. A node capability profile is generated based on node hardware configuration, storage capacity, and historical data processing performance. Real-time node status data is collected, including the current number of connections, remaining storage resources, and data update frequency, enabling the synchronous establishment of node access and capability baselines. The capability baseline refers to a quantifiable performance benchmark threshold or optimal operating range set for distributed data nodes in core capability dimensions (such as computing, storage, network, and fault tolerance) based on the node capability profile. It serves as a core reference for subsequent load prediction, scheduling triggering, risk assessment, and parameter optimization.

[0090] S2: Based on node capability profiles and real-time node status data, acquire metadata including data lineage, business tags, and access permissions; construct a dynamic association matrix between metadata and node capability profiles, where the matrix element values ​​represent the association strength between metadata, and the association strength is dynamically adjusted according to the frequency of interaction between node real-time status data and business data; when a node experiences data changes or status anomalies, trigger an incremental update mechanism for the association matrix, updating only the affected matrix elements to ensure the timeliness of metadata association.

[0091] S3: Establish a multi-dimensional scheduling evaluation model. Input the business data dependency relationship, the performance in the node capability profile, and the node load prediction value for the next 10-30 minutes obtained through the LSTM model. Output the initial scheduling scheme, which includes data sharding strategy, cross-node computing task routing path, and resource reservation scheme.

[0092] S4: Preset scheduling optimization trigger conditions, including static thresholds and dynamic prediction thresholds (predicted load exceeds 80% of the capacity baseline). Static thresholds include business request volume and node load, while dynamic prediction thresholds are predicted load exceeding 80% of the capacity baseline. When any trigger condition is met, the initial scheduling scheme is optimized through reinforcement learning algorithm to obtain optimized scheduling decisions.

[0093] S5: Based on the optimized scheduling decision, send scheduling instructions containing priority identifiers to the corresponding nodes, collect feedback data including response latency and error rate in real time, update the correlation strength in the dynamic correlation matrix based on the feedback data, and optimize the parameters of the multi-dimensional scheduling evaluation model to form a closed-loop iteration of management-scheduling-service.

[0094] The specific process of generating a node capability profile based on node hardware configuration, storage capacity, and historical data processing performance in step S1 is as follows:

[0095] S11: Multi-source data acquisition and preprocessing:

[0096] Hardware configuration data acquisition: Collect static hardware parameters, including CPU core count / frequency / cache size, total memory capacity / type (DDR4 / DDR5), network card bandwidth (10G / 40G), disk type (SSD / HDD), and IOPS baseline value.

[0097] Storage capacity data acquisition: Real-time acquisition of the total storage capacity, used capacity, remaining available capacity of nodes, as well as the sharding strategy (RAID level) and data redundancy (number of replicas) of the storage pool.

[0098] Historical data processing performance collection: Extract historical operation data for the past 30 days from the node log system or monitoring platform Prometheus, including: average processing time per task, maximum number of concurrent tasks per unit time, peak data transmission throughput, and task failure rate (number of abnormal terminations / total number of tasks).

[0099] The preprocessing stage involves cleaning the collected data, removing outliers such as zero values ​​caused by transient failures, standardizing units (e.g., unifying the storage capacity of different nodes to GB), and standardizing the data.

[0100] S12: Construction of Competency Indicator System

[0101] A three-level indicator system is constructed based on the collected data, serving as the core dimension of capability profiling:

[0102] Primary indicators: hardware infrastructure capabilities, storage service capabilities, and data processing capabilities;

[0103] Secondary indicators:

[0104] Basic hardware capabilities include: computing power (number of CPU cores × clock speed weighted value), memory support capacity (memory capacity × read and write speed), and network transmission capacity (bandwidth × stability coefficient, stability coefficient = 1 - historical packet loss rate).

[0105] Storage service capabilities include: effective storage capacity (remaining capacity × redundancy factor, redundancy factor = 1 / number of replicas) and storage IO performance (measured IOPS / theoretical IOPS).

[0106] Data processing capabilities include: task throughput (average number of tasks processed per day), response sensitivity (reciprocal of the p95 quantile response time), and fault tolerance (1 - historical task failure rate).

[0107] S13: Indicator Quantification and Weight Allocation:

[0108] Quantitative scoring: The secondary indicators are normalized and mapped to a score of 0-10. For example: computing power score = (number of CPU cores in the node × clock speed) / the highest number of CPU cores in the entire network × clock speed × 10; response sensitivity score = (the lowest p95 response time in the entire network / the p95 response time of the node) × 10.

[0109] Weighting: Determine the weights of the primary indicators, with hardware infrastructure capabilities at 30%, storage service capabilities at 25%, and data processing capabilities at 45%.

[0110] The weights of secondary indicators are allocated based on the contribution of the primary indicators. For example, in data processing capability, task throughput accounts for 50%, response sensitivity accounts for 30%, and fault tolerance accounts for 20%.

[0111] S14: Capability Profile Generation and Dynamic Tagging:

[0112] Calculate the overall capability score: Overall capability score = Σ(secondary indicator score × corresponding weight), forming a node overall capability score of 0-10;

[0113] Generate a multi-dimensional profile: with comprehensive capability value as the core, and add feature labels, including high throughput nodes (throughput score ≥ 8 points), low latency nodes (response sensitivity score ≥ 8 points), and large capacity storage nodes (effective storage capacity score ≥ 8 points).

[0114] Dynamic calibration mechanism: The profile is fine-tuned every 24 hours based on real-time status data. For example, when the remaining storage capacity of a node drops by 20%, the storage service capability score is downgraded in real time and the status of the large-capacity storage node is updated, such as downgrading it to a medium-capacity storage node.

[0115] Step S2 involves obtaining metadata including data lineage, business tags, and access permissions; and constructing a dynamic association matrix between metadata and node capability profiles, with matrix element values ​​representing the strength of association between metadata. The specific process is as follows:

[0116] S21: Data lineage metadata collection:

[0117] Data collection targets: Record the entire data flow from generation to destruction, including the mapping relationship between data source, processing process and target node, such as the link from sensor node to data cleaning task and storage in distributed database.

[0118] Data collection method:

[0119] For structured data, table-level / field-level lineage is extracted by parsing the SQL execution logs. For example, "Order Table.UserID" originates from "User Table.UserID".

[0120] For unstructured data, the distribution of data replicas is recorded by tracking the data transmission protocol, such as "the replica of file A on node 1 is generated synchronously by node 2".

[0121] For computational tasks, data processing links can be extracted using task execution plan parsing tools, such as SparkUI's DAG visualization interface, for example, "Task T1 converts data D1 into D2, which depends on the computing resources of node 3".

[0122] Structured storage: A lineage table is formed with source data ID, processing operation, target data ID, and timestamp as the core fields.

[0123] Business tag metadata collection:

[0124] Data collection targets: Tags that characterize the attributes of data business, including business domain (e-commerce, finance, etc.), data theme (user behavior, transaction flow, etc.), data level (core data, general data, etc.), and timeliness requirements.

[0125] Data collection method:

[0126] Automatic extraction: Parse data name and description fields, such as extracting the "order" and "payment" tags from "202310_order_payment.csv", and combining data content characteristics, such as marking data containing "amount" and "transaction number" fields as "transaction data".

[0127] Structured storage: A tag association table is formed with "data ID-tag set-tag weight (representing the matching degree between tags and data)" as the core fields.

[0128] Access permission metadata collection:

[0129] Data collection targets: permission rules that control data access, including access subjects (users / roles / applications), permission types (read / write / modify / delete), permission effective time, and data range (full data / partial fields / data under specific conditions).

[0130] Data collection method:

[0131] Synchronize basic permission rules from the data node's permission management system (HDFS ACL list, relational database GRANT statement log).

[0132] Extract the actual access relationships by accessing audit logs (API gateway call records, database login logs), such as "Application A has accessed read permissions for data D3 in the past 24 hours".

[0133] Pre-set permission templates based on business needs, such as "the finance role has read-only permission for transaction data by default", and associate them with specific data.

[0134] Structured storage: The access control table is formed with "data ID-access subject-permission type-effective status-access frequency" as the core fields.

[0135] S22: Construction of a dynamic association matrix between metadata and node capability profiles:

[0136] Based on the above structured metadata and node capability profiles, a two-dimensional dynamic association matrix is ​​constructed. The matrix element values ​​(0-100) quantify the association strength between metadata and node capabilities. See [link to documentation]. Figure 2 As shown.

[0137] The specific process of step S22 is as follows:

[0138] S221: Definition of matrix dimension:

[0139] Matrix row dimension: Metadata entries, uniquely identified by data ID, covering all collected lineage, tag, and permission metadata.

[0140] Matrix column dimension: The core indicator of node capability profile, namely the secondary indicator of the capability indicator system.

[0141] Matrix element M[i,j]: The correlation strength between the i-th metadata and the j-th node capability index. The higher the value, the higher the matching degree between the processing / storage requirements of the metadata and the j-th index of the node.

[0142] S222: Initial Calculation of Association Strength (Based on the Matching Degree of Metadata Features and Node Capabilities) For each metadata element i and node capability index j, the initial association strength is calculated through the rule engine:

[0143] Correlation with computing power indicators: If the business tag corresponding to the metadata contains "high computing intensity" (such as video transcoding data), or the blood relationship contains complex computing tasks (such as machine learning model training data), then M[i, computing power] = base score (60) + computing task complexity coefficient (0-40, the higher the complexity, the larger the coefficient).

[0144] Correlation with storage IO performance metrics: If the write operation frequency in the metadata access permission record is ≥100 times / hour (high frequency write), or the business tag is real-time streaming data (such as IoT sensor data), then M[i, storage IO performance] = base score (50) + write frequency coefficient (0-50, the higher the frequency, the larger the coefficient).

[0145] Correlation with response sensitivity index: If the business tag of the metadata contains low latency requirements (such as payment result data), or the access permission "read operation p95 response time requirement" ≤ 100ms, then M[i, response sensitivity] = base score (50) + latency requirement coefficient (0-50, the more stringent the latency requirement, the larger the coefficient).

[0146] The correlation strength calculation logic for other indicators (such as network transmission capacity and fault tolerance capacity) is similar, and matching rules are set based on the lineage of metadata (such as cross-node transmission distance), business tags (such as cross-regional data synchronization), and access frequency (such as cross-node access frequency).

[0147] S223: Dynamic adjustment of association strength (based on real-time node status and frequency of business interactions). The matrix element values ​​are dynamically updated every 5 minutes, and the adjustment logic is as follows:

[0148] Based on real-time node status data:

[0149] If the node's current remaining storage resources are ≤20% (threshold is configurable), then the M[i, storage service capability] of all metadata i associated with the node's storage service capability metrics will be reduced by 20%.

[0150] If the current number of connections of a node is greater than or equal to 80% of the maximum number of connections, then the metadata i associated with "network transmission capacity" will have its M[i, network transmission capacity] reduced by 15%.

[0151] Based on the frequency of business data interaction:

[0152] Count the number of interactions between metadata i and metadata k in the past hour. If i is the data source of k, or i and k are called by the same application at the same time, for every 100 interactions, the correlation strength of the corresponding node capability indicators of the two will be increased by 5% synchronously, with a maximum of 100.

[0153] If the access frequency of metadata i is ≤ 1 time for 3 consecutive hours, then the strength of all its associations will be reduced by 30% to lower the scheduling priority of inactive data.

[0154] Example 2

[0155] Based on Example 1, the incremental update mechanism in step S2 is as follows:

[0156] S224: To avoid the resource consumption of full computation, set the trigger condition for incremental updates of matrix elements:

[0157] Metadata changes have occurred, including the addition of new data lineage links and modifications to business tags;

[0158] The node's real-time status data exceeded the preset threshold, including a sudden 30% increase in CPU utilization;

[0159] The frequency of business interactions has reached the adjustment threshold, including exceeding 1,000 interactions within one hour;

[0160] S225: When the incremental update trigger condition is met, only the association strength between the affected metadata entries and the node capability indicators is recalculated, while other elements remain unchanged;

[0161] Recalculation rules for metadata change scenarios:

[0162] When a new "cross-node transmission" link is added for blood relations, the element values ​​in the relevant metadata row and the "network transmission capability" column will be increased by 5%-20% based on the transmission distance coefficient (the closer the distance, the higher the coefficient).

[0163] If the business tag "Reduced real-time requirements" is selected, the element values ​​in the relevant metadata rows and the "Response Sensitivity" column will be reduced by 30%-50%.

[0164] The access permission now includes "Batch Write Operations": the element values ​​in the relevant metadata rows and the "Storage IO Performance" column will be increased by 10%-30% based on the batch size, with larger batches seeing a greater increase.

[0165] Recalculation rules for node state anomalies:

[0166] If the remaining storage capacity of a node is ≤10%, the element values ​​of the relevant columns (storage service capacity) and metadata rows will be uniformly reduced by 20%-40%, with the reduction being greater when the capacity is tighter.

[0167] If the node network packet loss rate is ≥5%, the element values ​​of the relevant column (network transmission capacity) and metadata row will be reduced by 10%-30% based on the packet loss rate, and reduced by 2% for every 1% increase in the packet loss rate.

[0168] Recalculation rules for business interaction frequency scenarios:

[0169] For every 100 times / hour increase in metadata interaction frequency, the element values ​​of the relevant rows and the capability indicator columns of common dependencies will be increased by 5%-10%, with a maximum of 100.

[0170] If the frequency of metadata access increases sharply (increase ≥ 50%), the element values ​​of the relevant rows and the "Computing Power" and "Concurrent Processing Power" columns will be adjusted upwards proportionally, with a 50% increase corresponding to an increase of 10%; the element values ​​of the "Storage Capacity" column will be adjusted upwards according to the amount of data accessed, with a larger amount resulting in a greater increase.

[0171] The specific process in step S3 is as follows:

[0172] S31: Extraction of Business Data Dependency Features:

[0173] The business data dependencies are parsed from the dynamic association matrix and transformed into two types of quantitative features:

[0174] Dependency Strength Matrix: Let the dataset be {D1,D2,...,Dn}. The element E[i,j] in the matrix represents the dependency strength (0-100) of data Di on data Dj. It is calculated as follows: E[i,j] = (Frequency of interaction between Di and Dj / Maximum interaction frequency across the entire network) × 100 + (Weight percentage of Dj in Di's lineage × 50). For example, if Di calls Dj 80 times per hour (maximum 100 times per hour), and Dj accounts for 30% of Di's lineage, then E[i,j] = 80% × 100 + 30% × 50 = 95.

[0175] Dependency type tags: Based on the timeliness of the dependency, it is divided into "strong real-time dependency" (such as payment results depending on user balance, with a latency requirement of ≤100ms), "weak time-series dependency" (such as report data depending on the transactions of the day, with a latency requirement of ≤1 hour), and "no direct dependency", which are marked as T1, T2, and T0 respectively.

[0176] S32: Performance Feature Quantification of Node Capability Profile: Extracting core performance indicators from the node capability profile and standardizing them to feature values ​​of 0-100:

[0177] Static performance characteristics: computing power (number of CPU cores × clock speed weighted value), storage IO performance (measured IOPS / theoretical value), network transmission capacity (bandwidth × (1 - historical packet loss rate)), fault tolerance (1 - historical task failure rate).

[0178] Dynamic performance characteristics: Current remaining resources (remaining memory / total, remaining storage / total), current task concurrency (current running tasks / maximum concurrency capacity).

[0179] S33: Trend Feature Transformation of LSTM Node Load Prediction Values. Based on the load prediction data (CPU utilization, memory usage, network bandwidth usage) output by the LSTM model for the next 10-30 minutes, trend features are extracted:

[0180] Peak load and time point: The highest load value and the time of occurrence within the prediction period.

[0181] Load growth slope: Calculate the rate of change of load over 10-30 minutes. For example, if the CPU increases from 60% to 80%, the slope = 20% / 20 minutes = 1% / minute.

[0182] S34: See also Figure 3 As shown, a three-dimensional scheduling evaluation model is constructed, including "dependency adaptability, performance matching degree, and load risk controllability," and the comprehensive adaptability score of each node is calculated:

[0183] Dimension 1: Dependency Adaptability Assessment (Weight 30%): Assessing the node's ability to support business data dependencies. Key metrics:

[0184] Dependency link localization rate: If Di and Dj are both stored / processed on the same node in the dependency links (E[i,j]≥50), then it is counted as "localized". The localization rate = number of localized links / total number of dependency links × 100.

[0185] Real-time dependency response capability: For T1 type dependencies, the p95 response time of the node must be ≤100ms to get 100 points, and 10 points will be deducted for every 10ms exceeding the standard; for T2 type dependencies, the response time must be ≤1 hour to get 100 points, and 5 points will be deducted for every 10 minutes exceeding the standard.

[0186] Dimension 1 score = (localization rate × 0.6 + real-time dependency response capability × 0.4) × 30%.

[0187] Dimension 2: Performance Matching Assessment (Weight 40%): Assessing the degree to which node performance matches data processing requirements. Key metrics:

[0188] Capability requirement matching degree: Based on the correlation strength (M[i,j]) between data and node capabilities in the dynamic correlation matrix, the weighted average is calculated (the weight corresponds to the proportion of data processing requirements for each capability, such as the "computing capability" weight of 0.6 for computationally intensive data).

[0189] Resource availability fit: The current remaining resources (memory / storage) are greater than or equal to the resources required for data processing. For example, if the data size is 10GB, the remaining storage of the node must be greater than or equal to 12GB (including 20% ​​redundancy). If the standard is met, 100 points will be awarded. For every 10% shortfall, 20 points will be deducted.

[0190] Dimension 2 score = (capability requirement matching degree × 0.7 + resource margin suitability degree × 0.3) × 40%.

[0191] Dimension 3: Load Risk Controllability Assessment (Weight 30%): Assessing the load risk of nodes after scheduling based on LSTM predictions. Key metrics:

[0192] Predicted load compliance rate: If the predicted load of the node after scheduling (original predicted value + new task load) is less than or equal to 80% of the capacity baseline, 100 points are awarded for compliance, and 30 points are deducted for every 5% exceeding the baseline.

[0193] Load growth buffer: The difference between the node's current load and the predicted peak load. For example, if the current CPU load is 50% and the predicted peak load is 70%, the buffer is 20%. A buffer of ≥20% earns 100 points, and 20 points are deducted for every 5% less.

[0194] Dimension 3 score = (Predicted load compliance rate × 0.6 + Load growth buffer × 0.4) × 30%.

[0195] Node comprehensive score calculation: Comprehensive score = Dimension 1 score + Dimension 2 score + Dimension 3 score, with a score range of 0-100.

[0196] S35: Select candidate nodes that meet the criteria based on the overall score, and sort them by scheduling adaptability:

[0197] Screening criteria: Overall score ≥ 70 points (basic threshold, can be adjusted according to business scenarios);

[0198] Priority sorting rule: Sort by overall score in descending order.

[0199] S36: Based on the sorted candidate nodes, generate an initial scheduling scheme that includes data sharding, routing paths, and resource reservations.

[0200] The data sharding strategy generates sharding rules for target data D based on the processing capabilities and storage capacity of candidate nodes:

[0201] Number of shards N: N = min(total data size / maximum processing capacity per shard on a node / maximum number of parallel tasks on a node). For example: if the data size is 100GB, the maximum processing capacity per shard on a node is 20GB, and the maximum number of parallel tasks is 4, then N = 4, take the smaller value.

[0202] Segmentation allocation principle:

[0203] Data with strong dependencies (E[i,j]≥80) should be sharded and assigned to the same or adjacent nodes.

[0204] Shard sizes are allocated based on the differences in node storage IO performance. High-performance nodes (IOPS score ≥ 80) are allocated larger shards, while low-performance nodes are allocated smaller shards.

[0205] Cross-node computation task routing planning: For tasks that require processing across multiple nodes, such as those whose dependent data is distributed across multiple nodes, paths are planned based on network topology and predicted bandwidth.

[0206] Path candidate set: All reachable paths from the source node to the target node, based on a preset network topology, such as node A → switch S1 → node B as a path.

[0207] Path scoring metrics: Path latency (current measured latency + transmission latency under predicted bandwidth), Path load (predicted load rate of switches / routers in the path), Path redundancy (whether there is a backup path).

[0208] Optimal path selection: Select the path with the highest score (latency ≤20ms, load rate ≤70%, and a backup path) as the primary path, and designate one secondary path as a failover backup.

[0209] Resource reservation scheme determined: Based on the LSTM load forecast and task resource requirements, resources are reserved for the nodes:

[0210] Reserved resource types: CPU (number of cores), memory (GB), network bandwidth (MB / s), which must cover peak task demand plus predicted load fluctuations.

[0211] Calculation of reserve ratio:

[0212] Basic reserve = peak resource requirement of the task × 1.2, i.e. 20% redundancy;

[0213] Fluctuation reserve = (predicted maximum load of node - current load) × 0.3, that is, a 30% fluctuation margin is reserved;

[0214] Total reservation amount = basic reservation amount + fluctuating reservation amount, which shall not exceed 50% of the remaining resources of the node to avoid over-reservation.

[0215] Reserved time limit: Matches the task execution cycle, such as 10 minutes for short-term tasks and 30 minutes for long-term tasks, and automatically releases after the time limit expires.

[0216] Example 3

[0217] Based on Example 1 or Example 2, in step S4, the initial scheduling scheme is optimized using a reinforcement learning algorithm. The specific process for obtaining the optimized scheduling decision is as follows:

[0218] S41: Monitor the static threshold and the dynamic predicted threshold in real time, and determine whether the optimization triggering conditions are met.

[0219] Static threshold verification:

[0220] Business request volume threshold: Count the total number of business data requests within 5 minutes, classified by data type, such as query requests and write requests. If the volume of a certain type of request exceeds the preset threshold, such as query requests ≥ 1000 times / minute, the threshold is set based on 120% of the historical peak and does not decrease for 2 minutes, then optimization is triggered.

[0221] Node load threshold: Real-time collection of core load metrics of nodes, CPU utilization ≥85%, memory utilization ≥90%, disk I / O utilization ≥90%. If any metric exceeds the threshold for 3 consecutive sampling periods (10 seconds / period), or if 2 or more metrics in a single node exceed the threshold at the same time, optimization will be triggered.

[0222] Dynamic prediction threshold verification:

[0223] Based on the LSTM load prediction value in step S3, if the predicted load (CPU / memory / network) of a node in the next 10-30 minutes exceeds 80% of its capacity baseline, and the prediction error rate is ≤10% (calculated by the deviation between historical prediction values ​​and actual values), then optimization is triggered.

[0224] Trigger priority rules:

[0225] If multiple conditions are met simultaneously, the following priority will be applied: dynamic prediction threshold (high risk, to be avoided in advance) > node load threshold (real-time overload) > business request volume threshold (potential pressure), to ensure that emergency scenarios are handled first.

[0226] S42: Defining the core elements of a reinforcement learning model:

[0227] The state space consists of six types of real-time data, reflecting the overall picture of the current scheduling environment:

[0228] Node status: Current load (CPU / memory / network utilization), remaining resources (GB / cores), and LSTM-predicted load trend for the next 10 minutes for each node.

[0229] Initial scheme execution status: data fragmentation transmission progress (completed percentage), real-time latency of cross-node paths (ms), and actual utilization rate of reserved resources (%).

[0230] Business request characteristics: current request volume (times / minute), request type ratio (query / write), SLA requirements (such as response time limit).

[0231] Data dependency status: real-time interaction frequency (times / minute) of business data dependency links, and distribution nodes of strongly dependent data.

[0232] Fault risk indicators: number of error logs in the node in the last 5 minutes (such as transmission timeout, task failure), and network packet loss rate (%).

[0233] Constraint boundary: Maximum resource limit of a node.

[0234] The state space is represented in vector form, with the dimension being the sum of the above indicators, and is normalized to the interval [-1,1] to eliminate the influence of dimensions.

[0235] Action space:

[0236] Actions are defined as quantifiable adjustments to the initial scheduling scheme, ensuring the granularity and feasibility of the adjustments:

[0237] Data sharding adjustment:

[0238] The number of fragments ±1 must satisfy the requirement that the total size remains unchanged and the size of a single fragment is greater than or equal to the smallest processing unit, such as 1GB.

[0239] Sharding target node switching only occurs within the candidate node list, and the dependency localization rate after switching is ≥80% of the initial value.

[0240] Cross-node path adjustment:

[0241] The primary path is switched to the backup path, which is triggered when the delay of the primary path is greater than or equal to 150% of the preset value.

[0242] The path bandwidth allocation has been adjusted, increasing the bandwidth quota for high-priority tasks by 20%.

[0243] Resource reservation adjustments:

[0244] CPU / memory reservation is ±10%, and the remaining resources of the node after reservation must be ≥5%.

[0245] Allow 5 minutes for extension / shortening of the timeframe, with a maximum extension not exceeding twice the task execution cycle.

[0246] Reward function:

[0247] The reward function integrates four optimization objectives to quantify the merits of actions, as shown in the following formula:

[0248] Reward value = α × Load balancing reward + β × Response time reward + γ × Resource utilization reward + δ × Reliability reward - λ × Adjustment cost. Where α is the weight of the load balancing reward, β is the weight of the response time reward, γ is the weight of the resource utilization reward, δ is the weight of the reliability reward, and λ is the weight of the adjustment cost.

[0249] Calculation rules for each item:

[0250] Load balancing reward: inversely proportional to the standard deviation of node load (the smaller the better), formula = 100 / (1+load standard deviation), α=0.3;

[0251] Response time bonus: The ratio of actual response time to SLA requirement (≤1 earns 100 points, deducting 20 points for every 10% exceeding), β=0.3;

[0252] Resource utilization bonus: Nodes with average resource utilization (target 60%-80%) receive 100 points if they fall within this range, and points are deducted linearly if they deviate from this range, γ=0.2;

[0253] Reliability bonus: Task success rate (1 - failure rate) × 100, 50 points are deducted for each failure, δ=0.15;

[0254] Adjustment cost: Additional overhead of action execution, such as session reconstruction time for path switching and data transfer volume for fragment migration, deduct 10-50 points according to the size of the overhead, λ=0.05.

[0255] The reward value ranges from [-50, 100] to ensure that positive incentives dominate while suppressing over-adjustment;

[0256] S43: Using the initial scheduling scheme as the initial strategy, the system is optimized through multiple rounds of iterative "sampling-evaluation-update" processes. The specific process is as follows:

[0257] The initial policy initialization transforms the initial scheduling scheme into the initial policy π0: that is, the initial parameters of data sharding, traffic splitting paths, and resource reservation are used as the initial action selection probability of the policy, such as prioritizing the sharding allocation in the initial scheme, with a probability set to 0.8.

[0258] Exploration and Sampling (Collection Track):

[0259] In the current state S, action A is selected based on strategy π. In the initial stage, the exploration rate ε is increased by 0.3, and non-initial action schemes are randomly selected. As the iteration increases, ε decreases linearly to 0.1.

[0260] After executing action A, obtain the new state S', such as the node load change after sharding switch, the latency change after path adjustment, and the reward value R.

[0261] Repeat the above process, stopping the iteration when any of the following conditions are met:

[0262] The average reward value increases by ≤1% over three consecutive iterations (convergence);

[0263] The number of iterations has reached its limit (15 rounds, to avoid overcomputation affecting real-time performance).

[0264] The core metrics of the current solution (load balancing and response time) are already better than the preset optimal thresholds (such as load standard deviation ≤ 5% and response time compliance rate ≥ 99%).

[0265] The specific process of updating the association strength in the dynamic association matrix based on feedback data and optimizing the parameters of the multi-dimensional scheduling evaluation model in step S5 is as follows:

[0266] S51: Adjust the correlation strength based on the preset mapping table between feedback data and node capability indicators:

[0267] Table 1. Mapping Relationship between Feedback Data and Node Capability Indicators

[0268]

[0269] For each piece of metadata i and its corresponding node capability index j, the adjusted association strength M'[i,j] is calculated using the following formula: M'[i,j]=M[i,j]×(1+K×ΔS)

[0270] in:

[0271] ΔS = (Current standardized score - Historical average score) / Historical average score, reflecting the rate of change of current performance relative to history;

[0272] K is an adjustment coefficient, set according to the importance of the indicator:

[0273] Response sensitivity and fault tolerance: K=0.2 (adjustment range ±20%);

[0274] Network transmission capacity and concurrent processing capacity: K=0.15 (±15%);

[0275] Other indicators K=0.1 (±10%);

[0276] After adjustment, M'[i,j] should be limited to the range of 0-100. If it is lower than 0, take 0; if it is higher than 100, take 100.

[0277] for example:

[0278] The correlation strength between metadata D1 and the "response sensitivity" metric of node NodeA is originally M=80;

[0279] Historical average latency score = 90 points, current latency score = 72 points (ΔS = (72-90) / 90 = -0.2).

[0280] After adjustment, M' = 80 × (1 + 0.2 × (-0.2)) = 80 × 0.96 = 76.8 (reduced by 4%).

[0281] S52: Optimize the parameters of the multi-dimensional scheduling evaluation model based on feedback data:

[0282] The parameters of the multi-dimensional scheduling evaluation model (including the weights of each evaluation dimension and the quantitative benchmarks of the indicators) are dynamically optimized based on feedback data to ensure that the model output is consistent with the actual service effect. The specific process is as follows:

[0283] The weights of the four evaluation dimensions (task-node matching degree, resource utilization efficiency, scheduling cost, and business reliability) in the dynamic adjustment model need to be adjusted based on the shortcomings exposed in the feedback data, according to the following rules:

[0284] If the response latency compliance rate (the percentage of calls with a latency score ≥ 80) is < 90%, the weight of the "business reliability" dimension will be increased by 5% (up to a maximum of 40%), while the weight of the "scheduling cost" dimension will be decreased by 5% (down to a minimum of 5%).

[0285] If the node-level error rate is greater than 1%, the weight of the "fault tolerance matching" indicator in the "task-node matching degree" dimension will be increased by 3%, while the total weight of indicators in this dimension will remain at 100%.

[0286] If the concurrent processing success rate of high-frequency calls (≥500 times / minute) is <95%, the weight of the "task concurrency" indicator in the "resource utilization efficiency" dimension will be increased by 4%.

[0287] Weight adjustment is achieved through "sliding window weighting": the weight after each adjustment = 0.8 × original weight + 0.2 × target adjustment weight (smoothing fluctuations and avoiding drastic changes).

[0288] In the calibration model of the indicator quantification benchmark, the indicator quantification benchmark (such as the target range of node load rate, response time achievement threshold) needs to be updated according to the statistical characteristics of the feedback data to ensure that the benchmark fits the actual operating status:

[0289] Target range for node load rate: If the feedback data states that "the response latency compliance rate is highest (≥95%) when the node load rate is between 70% and 80%", then the original target range (60%-80%) will be adjusted to (70%-80%).

[0290] Response time threshold: If the actual average latency of a certain type of service remains stable at 80ms for a long period of time, which is lower than the original SLA threshold of 100ms, the response time threshold for that service will be lowered to 80ms to improve the model's accuracy in identifying high-quality nodes.

[0291] The quantitative benchmark is updated daily at midnight based on the statistical characteristics (mean, median, 95th percentile) of the feedback data from the previous 24 hours to ensure the timeliness of the benchmark.

[0292] Parameter optimization of the reinforcement learning reward function involves adjusting the weights (α, β, γ, δ, λ) of each component based on feedback data to make the reward more closely reflect the actual service quality.

[0293] If the response delay has the greatest impact on user experience in the feedback data (e.g., when the delay is >100ms, the caller complaint rate increases by 30%), then β (response time reward weight) is increased by 0.05 (maximum 0.4).

[0294] If "the error rate decreases most significantly after the node load balancing is improved" (the error rate decreases by 5% when the load standard deviation decreases by 10%), then α (load balancing reward weight) is increased by 0.03 (maximum 0.35).

[0295] The optimization of the reward function parameters is achieved through gradient descent: the deviation between the reward value and the actual service satisfaction (such as the caller's rating) is used as the loss function, and the parameters are iteratively adjusted to minimize the deviation.

Claims

1. A distributed data unified management and intelligent scheduling method based on data weaving, characterized in that, Includes the following steps: S1: Identify the communication protocol type of distributed data nodes in real time through a preset protocol feature library, and collect real-time status data of the nodes based on the communication protocol type; A node capability profile is generated based on node hardware configuration, storage capacity, and historical data processing performance. S2: Based on node capability profiles and real-time node status data, obtain metadata including data lineage, business tags, and access permissions; construct a dynamic correlation matrix between metadata and node capability profiles, where the matrix element values ​​represent the correlation strength between metadata. S3: Establish a multi-dimensional scheduling evaluation model, inputting business data dependencies, performance in node capability profiles, and load prediction values ​​of nodes within a specified future time period obtained through the LSTM model, and outputting an initial scheduling scheme; S4: Preset scheduling optimization trigger conditions, including static thresholds and dynamic prediction thresholds. When any trigger condition is met, the initial scheduling scheme is optimized through reinforcement learning algorithm to obtain the optimized scheduling decision. S5: Based on the optimized scheduling decision, send scheduling instructions containing priority identifiers to the corresponding nodes, collect feedback data including response latency and error rate in real time, update the correlation strength in the dynamic correlation matrix based on the feedback data, and optimize the parameters of the multi-dimensional scheduling evaluation model. In step S4, the initial scheduling scheme is optimized using a reinforcement learning algorithm. The specific process for obtaining the optimized scheduling decision is as follows: S41: Monitor the static threshold and the dynamic prediction threshold in real time, and determine whether the optimization triggering conditions are met; S42: Define the core elements of a reinforcement learning model: State space: includes node state, initial plan execution state, business request characteristics, data dependency state, fault risk indicators, and constraint boundaries; Action space: quantifiable adjustment operations on the initial scheduling scheme, including data fragmentation adjustment, cross-node path adjustment, and resource reservation adjustment; Reward function: quantifies the quality of an action, the formula is as follows: Reward value = α × load balancing reward + β × response time reward + γ × resource utilization reward + δ × reliability reward - λ × adjustment cost; Where α is the load balancing reward weight, β is the response time reward weight, γ is the resource utilization reward weight, δ is the reliability reward weight, and λ is the adjustment cost weight. S43: Using the initial scheduling scheme as the initial strategy, the system optimizes through multiple rounds of "sampling-evaluation-update". When the average reward value of three consecutive iterations increases by ≤1%, or the number of iterations reaches the upper limit, or the core indicators of the current scheme are better than the preset optimal threshold, the iteration stops.

2. The distributed data unified management and intelligent scheduling method based on data weaving according to claim 1, characterized in that, The specific process of generating a node capability profile based on node hardware configuration, storage capacity, and historical data processing performance in step S1 is as follows: S11: Collect multi-source data including hardware configuration data, storage capacity data, and historical data processing performance, and perform preprocessing; S12: Construct a three-level indicator system based on multi-source data as the core dimension of capability profiling: Primary indicators: hardware infrastructure capabilities, storage service capabilities, and data processing capabilities; Secondary indicators: Basic hardware capabilities include: computing power, memory support capacity, and network transmission capacity; Storage service capabilities include: effective storage capacity and storage I / O performance; Data processing capabilities include: task throughput, response sensitivity, and fault tolerance. S13: Normalize the secondary indicators and map them to 0-10 points; And assign weights to primary and secondary indicators; S14: Capability portrait generation: Calculate the overall capability score: Overall capability score = Σ(secondary indicator score × corresponding weight), forming a node overall capability score of 0-10; Generate a multi-dimensional profile: with comprehensive capability value as the core, and additional feature labels, including high-throughput nodes, low-latency nodes, and large-capacity storage nodes.

3. The distributed data unified management and intelligent scheduling method based on data weaving according to claim 1, characterized in that, The specific process of step S2 is as follows: S21: Collect metadata related to data lineage, business tags, and access permissions: Data lineage metadata: records the entire chain of data flow from generation to destruction, including the mapping relationship of "data source → processing process → target node"; Business tag metadata: Tags that characterize the business attributes of data, including business domain, data theme, data level, and timeliness requirements; Access permission metadata: The permission rules that control data access, including the access subject, permission type, permission effective time, and data scope; S22: Construction of a dynamic association matrix between metadata and node capability profiles: Based on metadata and node capability profiles, a two-dimensional dynamic association matrix is ​​constructed, with matrix element values ​​ranging from 0 to 100, to quantify the association strength between metadata and node capabilities.

4. The distributed data unified management and intelligent scheduling method based on data weaving according to claim 3, characterized in that, The specific process of step S22 is as follows: S221: Definition of matrix dimension: Matrix row dimension: metadata entries; Matrix column dimension: Core indicators for profiling node capabilities; Matrix element M[i,j]: the correlation strength between the i-th metadata item and the j-th node capability index; S222: For each piece of metadata i and node capability index j, perform an initial calculation of the association strength based on the matching degree between metadata features and node capabilities; S223: Dynamically adjust the association strength based on the real-time status of nodes and the frequency of business interactions. The matrix element values ​​are dynamically updated every 5 minutes.

5. The distributed data unified management and intelligent scheduling method based on data weaving according to claim 4, characterized in that, Step S2 also includes an incremental update mechanism, as detailed below: S224: Set the trigger condition for incremental updates of matrix elements: Metadata has changed; The node's real-time status data exceeds the preset threshold; The frequency of business interactions has reached the adjustment threshold; S225: When the incremental update trigger condition is met, only the association strength between the affected metadata entries and the node capability indicators is recalculated, while other elements remain unchanged.

6. The distributed data unified management and intelligent scheduling method based on data weaving according to claim 1, characterized in that, The specific process in step S3 is as follows: S31: Parse the business data dependencies from the dynamic association matrix and transform them into two types of quantitative features: dependency strength matrix and dependency type label; S32: Extract core performance indicators from the node capability profile, including static performance characteristics: computing power, storage I / O performance, network transmission capacity, and fault tolerance; dynamic performance characteristics: current remaining resources and current task concurrency; and standardize them to feature values ​​from 0 to 100. S33: Trend Feature Transformation of LSTM Node Load Forecast Values: Based on the load forecast data for the next 10-30 minutes output by the LSTM model, trend features are extracted: Peak load and time point: The highest load value and the time point of occurrence within the prediction period; Load growth slope: Calculate the rate of load change over 10-30 minutes; S34: Construct a three-dimensional evaluation model that includes dependency adaptability, performance matching degree, and load risk controllability, and calculate the comprehensive adaptability score of each node: Dimension 1: Assess dependency adaptability, weight 30%, with core metrics including dependency localization rate and real-time dependency response capability; Dimension 1 score = (localization rate × 0.6 + real-time dependency response capability × 0.4) × 30%; Dimension 2: Assess performance matching degree, weight 40%, core indicators include capability requirement matching degree and resource margin suitability; Dimension 2 score = (Capability requirement matching degree × 0.7 + Resource margin suitability degree × 0.3) × 40%; Dimension 3: Assess the controllability of load risk, with a weight of 30%. Key indicators include predicted load compliance rate and load growth buffer. Dimension 3 score = (Predicted load compliance rate × 0.6 + Load growth buffer level × 0.4) × 30%; Node overall score calculation: Overall score = Dimension 1 score + Dimension 2 score + Dimension 3 score, score range 0-100; S35: Select candidate nodes that meet the criteria based on the overall score, and sort them by scheduling adaptability: Screening criteria: Overall score ≥ 70 points; Priority sorting rule: Sort by overall score in descending order; S36: Based on the sorted candidate nodes, generate an initial scheduling scheme that includes data sharding, routing paths, and resource reservation.

7. The distributed data unified management and intelligent scheduling method based on data weaving according to claim 6, characterized in that, The specific process of updating the association strength in the dynamic association matrix based on feedback data and optimizing the parameters of the multi-dimensional scheduling evaluation model in step S5 is as follows: S51: Adjust the correlation strength based on the preset mapping table between feedback data and node capability indicators; S52: Optimize the parameters of the multi-dimensional scheduling evaluation model based on feedback data, including the weights of each evaluation dimension and the quantitative benchmark of the indicators.

Images