Distributed data storage management system and method

Abstract

The invention discloses a distributed data storage management system and method, and relates to the technical field of distributed storage, and the method comprises the steps: obtaining an access log, metadata and content features of each node; data object attributes are judged, cold data are marked, and cold data distribution nodes are positioned; the method comprises the following steps: dividing cold data into permanent, periodic and sudden basic access modes through first-layer classification; generating a composite classification label through second-layer classification; matching a predefined strategy matrix according to a classification result, dynamically selecting a compression algorithm and a storage medium, and allocating bandwidth; operation auditing, compliance verification and exception handling are realized through the smart contract; and collecting strategy execution indexes in real time, and adjusting classification model parameters and resource scheduling weights. Through an intelligent cold data judgment and classification mechanism, cold data is accurately identified and managed, occupation of storage resources is reduced, and the overall utilization rate of the storage resources is improved.

Description

Technical Field

[0001] The present invention relates to distributed storage technology, in particular to a distributed data storage management system and method. Background Art

[0002] With the advent of the big data era, the amount of data generated by various applications has exploded. Distributed storage systems have become the primary means of processing and storing massive amounts of data. However, data access frequency varies significantly within these distributed systems. In particular, cold data occupies a large amount of storage resources, reducing storage system performance and resource management efficiency.

[0003] Traditional cold data management methods are mostly based on fixed threshold rules for judgment, which makes it difficult to adapt to dynamically changing business loads and data access patterns, and is prone to problems such as high misjudgment rates and rigid policies. In existing technologies, cold data classification usually adopts a single dimension and lacks multi-level analysis of time series characteristics and data value, resulting in a low match between storage strategies and business needs. In addition, resource scheduling mostly relies on static rules and cannot dynamically adjust compression algorithms, storage media, and bandwidth allocation according to the real-time load of nodes, which can easily cause fluctuations in storage performance. In terms of data positioning, traditional hash algorithms have difficulty balancing the distribution of virtual nodes and the load of physical nodes, resulting in high data migration costs; in terms of compliance assurance, existing systems mostly rely on manual audits, which have defects such as difficulty in tracing operations and delayed response to exceptions. Summary of the Invention

[0004] The object of the present invention is to provide a distributed data storage management system and method to solve the problems raised in the prior art.

[0005] To achieve the above object, the present invention provides the following technical solution: a distributed data storage management method, the method comprising: The distributed collector obtains access logs, metadata, and content features of each node, and generates standardized feature vectors after preprocessing. Based on the standardized feature vector, the data object attributes are judged by the preset cold data determination rules, the cold data is marked, and the cold data distribution nodes are located in combination with the consistent hashing algorithm; Through the first-level classification, the time series analysis model is used to classify cold data into permanent, periodic, and sudden basic access patterns. Based on the first-level classification results, the second-level classification is carried out, integrating the data value scoring model and the business rule library to generate composite classification labels. Based on the classification results, the predefined policy matrix is matched, compression algorithms and storage media are dynamically selected according to node load, and bandwidth is dynamically allocated according to the predefined policy matrix; operation auditing, compliance verification, and exception handling are implemented through smart contracts. Collect strategy execution indicators in real time and automatically adjust classification model parameters and resource scheduling weights through control loops.

[0006] According to the above scheme, the pretreatment includes: The access logs are timestamp-normalized to eliminate time zone differences and unify the time base; the metadata is structurally parsed and semantically verified to eliminate field ambiguity and complete missing mandatory fields; the content features are dimensionality-reduced to extract key feature dimensions through principal component analysis, and the values of each dimension are mapped to the interval [0, 1] using a normalization algorithm; Establish a mapping table between feature vectors and node physical locations, and verify the uniqueness of the vectors.

[0007] According to the above solution, the cold data distribution nodes are located by using the preset cold data determination rules in combination with the consistent hashing algorithm, including: Based on the real-time status of the storage cluster, an adaptive threshold model is constructed. The adaptive threshold model comprehensively calculates capacity utilization, data redundancy level, and access frequency decay trend to generate a dynamic cold data determination threshold. The capacity utilization indicator weight is dynamically adjusted according to the cluster load level. A multi-dimensional joint determination strategy is adopted to implement a hierarchical determination process. When a data object meets the cold data characteristics of each dimension simultaneously, the final marking is triggered and the data object is marked as cold data. The multiple dimensions include access frequency, storage duration, and association relationship. In the access frequency dimension, the access frequency variation characteristics within a preset sliding time window are analyzed. In the storage duration dimension, the life cycle stage characteristics of the data object are evaluated. In the association relationship dimension, the topological characteristics in the data association graph are detected. Build a virtual node topology, assign multiple virtual nodes to each physical node, evenly map the virtual nodes to a ring key space using a consistent hashing algorithm, and establish a mapping relationship table between virtual nodes and physical locations; The cold data distribution node is located, the globally unique identifier of the data object marked as cold data is extracted, the mapping position of the globally unique identifier in the ring key space is calculated, the original positioning point is determined, and the K nearest virtual nodes are located clockwise along the ring topology with the original positioning point as the starting point to form a basic candidate node set, where K is a positive integer and is greater than or equal to a preset redundancy; the basic candidate node set is filtered based on the mapping relationship table between the virtual node and the physical location, and the filtering includes rack-level filtering, regional filtering and load balancing filtering; the rack-level filtering ensures that the replicas are distributed in at least N different physical racks; the regional filtering selects nodes located in different data centers when cross-regional redundancy is enabled; the load balancing filtering excludes candidates whose associated physical node loads exceed a dynamic threshold; From the filtered candidate nodes, the first N nodes are selected in clockwise order, where N represents the preset redundancy, as the final cold data distribution.

[0008] According to the above solution, the time series analysis model is used to classify cold data into permanent, periodic, and sudden basic access patterns, including: A sliding time window analysis mechanism is constructed to extract multi-granularity time series features from cold data access records. These features include long-term trend features, periodic features, and burst features. For long-term trend features, high-order differences of the time series are calculated to analyze the decay rate of data access volume. For periodic features, significant periodic components and harmonic characteristics of access frequency are analyzed through spectral analysis. For burst features, the coefficient of variation and burst intensity index of access intervals are monitored. The long short-term memory network is used to process the original time series data, output a preliminary pattern probability distribution, and weightedly fuse the long-term trend characteristics, periodic characteristics, and sudden characteristics to generate the probability distribution of the permanent, periodic, and sudden basic access patterns. When the probability of a certain type of basic access pattern exceeds a preset confidence threshold, the corresponding basic access pattern classification result is output; The basic access pattern classification result is encoded into a structured label; the structured label includes a pattern type identifier, a pattern feature parameter and a classification confidence; and the structured label is written into a data object marked as cold data.

[0009] According to the above solution, the second-level classification integrates the data value scoring model and the business rule library to generate a composite classification label, including: Construct a multi-dimensional value assessment system; the multi-dimensional value assessment system includes business value, compliance value, and association value dimensions. Through weighted decision-making, the assessment of each dimension is integrated to generate a standardized value score. The business value dimension calculates a basic score based on the importance and access priority of the business line to which the data belongs. The compliance value dimension calculates the compliance weight based on the data retention period requirements and privacy protection level. The association value dimension analyzes the centrality and connectivity characteristics of the data in the association graph. Load a predefined business rule library; the business rule library includes storage policy rules, lifecycle rules, and access control rules. The rule engine matches data attributes with business rules and outputs rule matching tags. The storage policy rules define the storage medium preferences for different business types of data; the lifecycle rules specify the minimum retention period for each type of data; and the access control rules specify special handling requirements for sensitive data. The basic access mode label output by the first-level classification is received, and combined with the data value score and the business rule label to generate a composite classification label; the composite classification label includes the basic access mode type, the data value level and the business rule label.

[0010] According to the above scheme, a predefined policy matrix is constructed, which includes data classification, storage resources and node status; the data classification is based on the basic access mode type and data value level in the composite classification label; the storage resources include storage medium type and compression algorithm level; the node status includes real-time load indicators of physical nodes.

[0011] According to the above scheme, an initial matching unit of the strategy matrix is positioned according to the composite classification label; When the node CPU utilization exceeds the first threshold, the compression algorithm level is automatically downgraded; when the memory usage exceeds the second threshold, the compression algorithm with low memory usage is preferentially selected; based on the real-time delay data of the network topology, the bandwidth allocation weight coefficient is dynamically adjusted; and the final strategy is output; The compression algorithm selects an algorithm that matches the node CPU utilization according to the compression algorithm priority list; the storage medium stores data to the corresponding medium according to the storage medium selection strategy; the bandwidth allocation allocates guaranteed bandwidth for the data transmission task according to the bandwidth allocation weight.

[0012] According to the above scheme, the smart contract records the key decision parameters of the strategy matching process, the execution results of resource scheduling, and abnormal operation events and handling measures; the key decision parameters include the execution efficiency index of the compression algorithm, the performance index of the storage medium and the bandwidth utilization index; the execution results of the resource scheduling include the storage medium allocation certificate, bandwidth weight calculation data and the compression algorithm execution fingerprint; the abnormal operation events and handling measures include the event type code, the compensation result hash and the node confirmation signature.

[0013] According to the above solution, the real-time acquisition strategy execution indicators are automatically adjusted through the control loop to adjust the classification model parameters and resource scheduling weights, including: The policy execution indicators include compression execution indicators, storage performance indicators and network transmission indicators; Adjusting the classification model parameters includes adjusting the dynamic judgment threshold in the cold data judgment rule when the cold data misjudgment rate exceeds the threshold, and optimizing the sliding time window size of the timing analysis model when the pattern classification error rate increases; adjusting the resource scheduling weight includes dynamically updating the weight distribution of the storage resource dimension according to the storage performance index, and correcting the bandwidth allocation weight coefficient in real time in combination with the network transmission index.

[0014] A distributed data storage management system, characterized in that: the system includes a data acquisition module, a cold data analysis module, an intelligent classification module, a strategy execution module and a dynamic optimization module; The data acquisition module includes an acquisition module, a preprocessing module, a feature extraction module, and a mapping module; the acquisition module is used to collect node access logs, metadata, and content features; the preprocessing module is used to preprocess the collected data; the feature extraction module is used to generate standardized feature vectors; and the mapping module is used to establish a mapping table between feature vectors and physical nodes. The cold data analysis module includes a threshold module, a multi-dimensional determination module, and a node positioning module; the threshold module is used to generate a dynamic cold data determination threshold based on the cluster status; the multi-dimensional determination module analyzes data object attributes based on the access frequency dimension, storage duration dimension, and association relationship dimension to mark cold data; the node positioning module locates cold data distribution nodes using a consistent hashing algorithm; The intelligent classification module includes a time series analysis module, a value assessment module, and a rule matching module; the time series analysis module uses a time series analysis model to classify cold data into permanent, periodic, and sudden basic access patterns; the value assessment module is used to perform standardized value scoring; and the rule matching module loads a business rule library to generate matching tags. The policy execution module includes a policy matrix module and a dynamic decision module; the policy matrix module is used to construct a policy matrix including data classification, storage resources and node status; the dynamic decision module dynamically selects compression algorithm, storage medium and allocates bandwidth; The dynamic optimization module includes a smart contract module and an indicator monitoring module; the smart contract module records the key decision parameters of the strategy matching process, the execution results of resource scheduling, and abnormal operation events and processing measures; the indicator monitoring module collects compression execution indicators, storage performance indicators and network transmission indicators in real time.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention uses an intelligent cold data determination and classification mechanism to accurately identify and manage cold data, reduce cold data storage resource usage, and improve the overall utilization of storage resources; 2. The present invention dynamically adjusts storage strategies based on data access patterns and value levels, selects appropriate compression algorithms and storage media, and dynamically allocates bandwidth, thereby optimizing storage and access performance. 3. The present invention generates composite classification labels that meet business needs and compliance requirements by integrating the data value scoring model and the business rule library, thereby ensuring the compliance and security of data storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flowchart of the steps of a distributed data storage management method of the present invention; Figure 2 This is a structural diagram of a distributed data storage management system of the present invention. DETAILED DESCRIPTION

[0017] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0018] Example: Figure 1-Figure 2 As shown, the present invention provides a technical solution, a distributed data storage management method, the method comprising the steps of: S1. Obtain access logs, metadata, and content features of each node through distributed collectors, and generate standardized feature vectors after preprocessing. Specifically, for example: access logs contain information such as request time, IP address, and request path; metadata may include file size, creation time, and modification time; content features are extracted by extracting key features of file content; Furthermore, the access logs are timestamp-standardized to eliminate time zone differences and unify the time base; metadata is structurally parsed and semantically verified to eliminate field ambiguity and complete missing mandatory fields; content features are dimensionality reduced, key feature dimensions are extracted through principal component analysis, and the normalization algorithm is used to map the values of each dimension to the [0,1] interval; a mapping relationship table between feature vectors and the physical location of nodes is established, and the vector uniqueness is verified to ensure that each feature vector can uniquely identify a data object and its storage location.

[0019] S2. Based on the standardized feature vector, the data object attributes are determined using the preset cold data determination rules, cold data is marked, and the cold data distribution nodes are located using the consistent hashing algorithm. Specifically, an adaptive threshold model is constructed based on the real-time status of the storage cluster. The adaptive threshold model comprehensively calculates capacity utilization, data redundancy level, and access frequency attenuation trend to generate a dynamic cold data judgment threshold. The capacity utilization indicator weight is dynamically adjusted with the cluster load level. A multi-dimensional joint judgment strategy is adopted to implement a hierarchical judgment process. When a data object meets the cold data characteristics of each dimension at the same time, the final marking is triggered and the data object is marked as cold data. The multiple dimensions include access frequency dimension, storage duration dimension, and association relationship dimension. In the access frequency dimension, the access frequency change characteristics within the preset sliding time window are analyzed. In the storage duration dimension, the life cycle stage characteristics of the data object are evaluated. In the association relationship dimension, the topological characteristics in the data association graph are detected. For example, data object A has only been accessed once in the past 30 days, which is far below the average access frequency. Data object A was created 6 months ago and has been accessed very rarely during this period. Data object A is in a marginal position in the association graph and has a very low connectivity with other important data objects. Based on the above three dimensions, data object A is marked as cold data.

[0020] Furthermore, a virtual node topology is constructed, multiple virtual nodes are assigned to each physical node, the virtual nodes are evenly mapped to the annular key space through a consistent hashing algorithm, and a mapping relationship table between virtual nodes and physical locations is established; the cold data distribution node is located, the globally unique identifier of the data object marked as cold data is extracted, the mapping position of the globally unique identifier in the annular key space is calculated, the original positioning point is determined, and the original positioning point is used as the starting point. The nearest K virtual nodes are located clockwise along the annular topology to form a basic candidate node set, where K is a positive integer and is greater than or equal to the preset redundancy; the basic candidate node set is filtered based on the mapping relationship table between virtual nodes and physical locations, and the filtering includes rack-level filtering, regional filtering and load balancing filtering; rack-level filtering ensures that replicas are distributed in at least N different physical racks; regional filtering, when cross-regional redundancy is enabled, nodes located in different data centers are selected; load balancing filtering, candidates whose associated physical node loads exceed the dynamic threshold are excluded; from the filtered candidate nodes, the first N nodes are selected in clockwise order, where N represents the preset redundancy, as the final cold data distribution.

[0021] S3. Through the first-level classification, the time series analysis model is used to classify cold data into permanent, periodic, and sudden basic access patterns. Based on the first-level classification results, the second-level classification is used to integrate the data value scoring model and the business rule library to generate composite classification labels. Specifically, a sliding time window analysis mechanism is constructed to extract multi-granularity time series features from the access records of cold data, for example, using sliding time windows of 7 days, 30 days, and 90 days for analysis; the features include long-term trend features, periodic features, and burst features; for long-term trend features, high-order differences of time series are calculated to analyze the decay rate of data access volume, for example, analysis shows that data access volume has continued to decline in the past 90 days; for periodic features, significant periodic components and harmonic features of access frequency are analyzed through spectrum analysis, for example, access frequency has a weekly periodic fluctuation; for burst features, the coefficient of variation and burst intensity index of access intervals are monitored, for example, access frequency within a specific time period Sudden increase; use long short-term memory network to process the original time series data, output preliminary pattern probability distribution, weighted fusion of long-term trend characteristics, periodic characteristics and sudden characteristics, and generate probability distribution of permanent, periodic and sudden basic access patterns; when the probability of a certain type of basic access pattern exceeds the preset confidence threshold, output the corresponding basic access pattern classification result; encode the basic access pattern classification result into a structured label; the structured label includes a pattern type identifier, pattern feature parameters and classification confidence; write the structured label to the data object marked as cold data, for example: the structured label is: pattern type identifier: periodicity, pattern feature parameters: period is one week, classification confidence: 90%.

[0022] Furthermore, a multi-dimensional value assessment system is constructed; the multi-dimensional value assessment system includes business value dimension, compliance value dimension and association value dimension; through weighted decision-making, the assessment of each dimension is integrated to generate a standardized value score; in the business value dimension, the basic score is calculated based on the importance and access priority of the business line to which the data belongs; in the compliance value dimension, the compliance weight is calculated according to the data retention period requirements and privacy protection level; in the association value dimension, the centrality and connectivity characteristics of the data in the association graph are analyzed; a predefined business rule library is loaded; the business rule library includes storage policy rules, lifecycle rules and access control rules; the data attributes are matched with the business rules through the rule engine, and the rule matching tags are output; storage policy Rules define the storage media preferences for data of different business types; lifecycle rules stipulate the minimum retention period for each type of data; access control rules specify special processing requirements for sensitive data; receive the basic access mode labels output by the first-level classification, combine the data value score and business rule labels, and generate composite classification labels; composite classification labels include basic access mode types, data value levels and business rule labels, for example: basic access mode label: periodic, data value score: level c, business rule label: low-cost media storage, retention period of 1 year; (level a is core business data, level b is important auxiliary data, level c is routine operation data, level d is low-frequency reference data, and level e is temporary or discarded data).

[0023] Furthermore, a predefined policy matrix is constructed, which includes data classification, storage resources and node status; data classification is based on the basic access mode type and data value level in the composite classification label; storage resources include storage medium type and compression algorithm level; node status includes real-time load indicators of physical nodes.

[0024] S4. Match the classification results to a predefined policy matrix, dynamically select compression algorithms and storage media based on node load, and dynamically allocate bandwidth according to the predefined policy matrix; implement operational auditing, compliance verification, and exception handling through smart contracts; Specifically, the initial matching unit of the policy matrix is located according to the composite classification label. For example, the policy matrix stipulates that data with periodic access patterns and a data value score below 50 points is stored on low-cost media and uses a low compression algorithm level. When the node CPU utilization exceeds the first threshold, the compression algorithm level is automatically downgraded. When the memory occupancy exceeds the second threshold, a compression algorithm with low memory occupancy is given priority. The bandwidth allocation weight coefficient is dynamically adjusted according to the real-time delay data of the network topology. The final policy is output. For the compression algorithm, the algorithm that matches the node CPU utilization is selected according to the compression algorithm priority list. For the storage medium, the data is stored in the corresponding medium according to the storage medium selection strategy. For bandwidth allocation, the guaranteed bandwidth is allocated for the data transmission task according to the bandwidth allocation weight.

[0025] Furthermore, the smart contract records the key decision parameters of the strategy matching process, the execution results of resource scheduling, and abnormal operation events and handling measures; the key decision parameters include the execution efficiency index of the compression algorithm, the performance index of the storage medium and the bandwidth utilization index; the execution results of resource scheduling include storage medium allocation proof, bandwidth weight calculation data and compression algorithm execution fingerprint; abnormal operation events and handling measures include event type code, compensation result hash and node confirmation signature.

[0026] S5, collects strategy execution indicators in real time and automatically adjusts classification model parameters and resource scheduling weights through control loops; Specifically, the policy execution indicators include compression execution indicators, storage performance indicators, and network transmission indicators; adjusting the classification model parameters includes adjusting the dynamic judgment threshold in the cold data judgment rule when the cold data misjudgment rate exceeds the threshold, and optimizing the sliding time window size of the timing analysis model when the pattern classification error rate increases; adjusting the resource scheduling weight includes dynamically updating the weight distribution of the storage resource dimension according to the storage performance indicators, and correcting the bandwidth allocation weight coefficient in real time in combination with the network transmission indicators.

[0027] The present invention provides another technical solution, a distributed data storage and management method, cold data determination and location; The storage cluster consists of 200 nodes and requires cold storage for data that has not been accessed for three months. An adaptive threshold model was constructed. The model comprehensively calculated that the cluster capacity utilization rate was 78%, the data redundancy level was 3 copies, and the access frequency decay rate decreased by 12% on a daily basis. A dynamic cold data determination threshold of 0.1 was generated. Data object B met the cold data characteristics of all dimensions simultaneously, triggering the final marking, marking the data object as cold data. Allocate 20,000 virtual nodes to 200 physical nodes, with 100 virtual nodes allocated to each physical node; Extract the globally unique identifier z of the data object B marked as cold data and map it to the corresponding virtual node V1; locate the nearest K virtual nodes clockwise along the ring topology to form the basic candidate node set, which is: V1, V2 and V3; The basic candidate node set is filtered based on the mapping relationship table between virtual nodes and physical locations. The filtering includes rack-level filtering, region-level filtering, and load balancing filtering. Through rack-level filtering, V2 in the same rack is filtered; through region-level filtering, V3 in different availability zones is filtered. From the filtered candidate nodes, the first N nodes are selected in clockwise order, where N represents the preset redundancy, and the final distribution nodes are {N1, N3, N5} (N=3, meeting the preset redundancy) as the final cold data distribution.

[0028] The present invention provides another technical solution, a distributed data storage and management method, in which cold data is classified to generate composite classification labels; Analyze the access records of cold data C and cold data D, perform multi-granularity time series feature extraction through sliding time window analysis, and obtain the following characteristics: cold data C shows periodic peaks, triggering access on the 25th of each month, while cold data D has no significant period and has no access in the past 30 days. The output probability is: Cold data C: Permanence: 0.11, Periodicity: 0.86, Burst: 0.03; Cold data D: Permanence: 0.92, Periodicity: 0.05, Burst: 0.03; Output the corresponding basic access pattern classification results and generate structured labels: Cold data C: Pattern type: periodic, characteristic parameters: period length: 30, strength: 0.86, confidence: 0.86; Cold data D: Pattern type: permanent, characteristic parameters: decay rate: 0.95, confidence: 0.92; Build a multi-dimensional value assessment system; the multi-dimensional value assessment system includes business value dimensions, compliance value dimensions, and associated value dimensions. Through weighted decision-making, the assessment of each dimension is integrated to generate a standardized value score. The standardized value score is mapped into five levels: A, B, C, D, and E. Level A is core business data with high priority; Level B is important auxiliary data that supports core business operations; Level C is routine operational data, including daily operation logs and basic compliance requirements; Level D is low-frequency reference data; and Level E is temporary or discarded data and can be deleted. The value of cold data C is level a, and the value of cold data D is level d; Matching based on preset business rules: cold data C business rule tag storage medium type is NVMe, and is retained for 5 years; cold data D business rule tag storage medium type is HDD, low cost, and is archived after 1 year; Output composite classification labels: Cold data C has a periodic access pattern and a value level of A. The policies required are: Storage medium: NVMe (according to storage policy rules), Retention period: 5 years (according to lifecycle rules), and Bandwidth allocation: Priority guarantee (due to value level A). Cold data D is in permanent access mode, value level D, and requires the following policies to be executed: storage medium: HDD (low-cost media rule), compression algorithm: Zstd-3 (dynamically selected according to node load), access control: desensitization processing (according to privacy rules).

[0029] The present invention provides another technical solution, a distributed data storage management method, a strategy matrix; this is only used as an example and is not limiting; The policy matrix includes data classification, storage resources, and node status. Data classification is based on the basic access mode type and data value level in the composite classification label. Storage resources include storage media type and compression algorithm level. Node status includes real-time load indicators of physical nodes. For example: Data classification: Periodic access mode, Value level A; corresponding policy actions: Storage medium: High-performance SSD, Compression algorithm: Zstd level 2, Bandwidth weight: 0.8; Storage resources: Redundancy requires three copies; corresponding policy actions: Distribute across three different racks, with the primary copy placed in a high-performance storage node; Node status: CPU utilization > 70%; corresponding policy action: Because the CPU utilization of the target node is > 70%, exceeding the first threshold, it is automatically downgraded to Zstd level 3 and the replica is migrated to a node with a CPU utilization lower than the second threshold; if the downgrade operation conflicts with the business rule library, the business rule takes precedence.

[0030] The present invention provides another technical solution, a distributed data storage and management system, characterized in that: the system includes a data acquisition module, a cold data analysis module, an intelligent classification module, a strategy execution module and a dynamic optimization module; The data acquisition module includes an acquisition module, a preprocessing module, a feature extraction module, and a mapping module; the acquisition module is used to collect node access logs, metadata, and content features; the preprocessing module is used to preprocess the collected data; the feature extraction module is used to generate standardized feature vectors; and the mapping module is used to establish a mapping table between feature vectors and physical nodes. The cold data analysis module includes a threshold module, a multi-dimensional judgment module, and a node positioning module. The threshold module is used to generate a dynamic cold data judgment threshold based on the cluster status. The multi-dimensional judgment module analyzes data object attributes based on the access frequency, storage duration, and association relationship dimensions to mark cold data. The node positioning module locates cold data distribution nodes using a consistent hashing algorithm. The intelligent classification module includes a time series analysis module, a value assessment module, and a rule matching module. The time series analysis module uses a time series analysis model to classify cold data into permanent, periodic, and sudden basic access patterns. The value assessment module is used to perform standardized value scoring. The rule matching module loads the business rule library to generate matching tags. The policy execution module includes a policy matrix module and a dynamic decision module; the policy matrix module is used to build a policy matrix including data classification, storage resources and node status; the dynamic decision module dynamically selects compression algorithms, storage media and allocates bandwidth; The dynamic optimization module includes a smart contract module and an indicator monitoring module; the smart contract module records the key decision parameters of the strategy matching process, the execution results of resource scheduling, abnormal operation events and handling measures; the indicator monitoring module collects compression execution indicators, storage performance indicators and network transmission indicators in real time.

[0031] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A distributed data storage management method, characterized by: The method includes: The distributed collector obtains access logs, metadata, and content features of each node, and generates standardized feature vectors after preprocessing. Based on the standardized feature vector, the data object attributes are judged by the preset cold data determination rules, the cold data is marked, and the cold data distribution nodes are located in combination with the consistent hashing algorithm; Through the first-level classification, the time series analysis model is used to classify cold data into permanent, periodic, and sudden basic access patterns. Based on the first-level classification results, the second-level classification is carried out, integrating the data value scoring model and the business rule library to generate composite classification labels. Based on the classification results, the predefined policy matrix is matched, compression algorithms and storage media are dynamically selected according to node load, and bandwidth is dynamically allocated according to the predefined policy matrix; operation auditing, compliance verification, and exception handling are implemented through smart contracts. Collect strategy execution indicators in real time and automatically adjust classification model parameters and resource scheduling weights through control loops.

2. A distributed data storage management method according to claim 1, characterized in that: The pretreatment includes: The access log is timestamp-normalized to eliminate time zone differences and unify the time base; the metadata is structurally parsed and semantically verified to eliminate field ambiguity and complete missing required fields; the content features are dimensionality reduced, key feature dimensions are extracted through principal component analysis, and the normalization algorithm is used to map the values of each dimension to the [0,1] interval; a mapping relationship table between feature vectors and node physical locations is established, and the vector uniqueness is verified.

3. A distributed data storage management method according to claim 1, characterized in that: The method of locating cold data distribution nodes by using preset cold data determination rules and combining the consistent hashing algorithm includes: Based on the real-time status of the storage cluster, an adaptive threshold model is constructed. The adaptive threshold model comprehensively calculates capacity utilization, data redundancy level, and access frequency decay trend to generate a dynamic cold data determination threshold. The capacity utilization indicator weight is dynamically adjusted according to the cluster load level. A multi-dimensional joint determination strategy is adopted to implement a hierarchical determination process. When a data object meets the cold data characteristics of each dimension simultaneously, the final marking is triggered and the data object is marked as cold data. The multiple dimensions include access frequency, storage duration, and association relationship. In the access frequency dimension, the access frequency variation characteristics within a preset sliding time window are analyzed. In the storage duration dimension, the life cycle stage characteristics of the data object are evaluated. In the association relationship dimension, the topological characteristics in the data association graph are detected. Build a virtual node topology, assign multiple virtual nodes to each physical node, evenly map the virtual nodes to a ring key space using a consistent hashing algorithm, and establish a mapping relationship table between virtual nodes and physical locations; The cold data distribution node is located, the globally unique identifier of the data object marked as cold data is extracted, the mapping position of the globally unique identifier in the annular key value space is calculated, the original positioning point is determined, and the K nearest virtual nodes are located clockwise along the annular topology with the original positioning point as the starting point to form a basic candidate node set, where K is a positive integer and is greater than or equal to a preset redundancy; the basic candidate node set is filtered based on the mapping relationship table between the virtual nodes and the physical locations, and the first N nodes are selected from the filtered candidate nodes in a clockwise order, where N represents the preset redundancy, as the final cold data distribution.

4. A distributed data storage management method according to claim 1, characterized in that: The time series analysis model is used to classify cold data into permanent, periodic, and sudden basic access patterns, including: A sliding time window analysis mechanism is constructed to extract multi-granularity time series features from cold data access records. These features include long-term trend features, periodic features, and burst features. For long-term trend features, high-order differences of the time series are calculated to analyze the decay rate of data access volume. For periodic features, significant periodic components and harmonic characteristics of access frequency are analyzed through spectral analysis. For burst features, the coefficient of variation and burst intensity index of access intervals are monitored. The long short-term memory network is used to process the original time series data, output a preliminary pattern probability distribution, and weightedly fuse the long-term trend characteristics, periodic characteristics, and sudden characteristics to generate the probability distribution of the permanent, periodic, and sudden basic access patterns. When the probability of a certain type of basic access pattern exceeds a preset confidence threshold, the corresponding basic access pattern classification result is output; The basic access pattern classification result is encoded into a structured label; the structured label includes a pattern type identifier, a pattern feature parameter and a classification confidence; and the structured label is written into a data object marked as cold data.

5. The distributed data storage management method according to claim 1, wherein: The second-level classification integrates the data value scoring model and the business rule library to generate a composite classification label, including: Construct a multi-dimensional value assessment system; the multi-dimensional value assessment system includes business value dimensions, compliance value dimensions, and associated value dimensions; through weighted decision-making, integrate the assessments of each dimension to generate a standardized value score; Loading a predefined business rule library; the business rule library includes storage policy rules, lifecycle rules, and access control rules; matching data attributes with business rules through a rule engine, and outputting rule matching tags; The basic access mode label output by the first-level classification is received, and combined with the data value score and the business rule label to generate a composite classification label; the composite classification label includes the basic access mode type, the data value level and the business rule label.

6. A distributed data storage management method according to claim 1, characterized in that: A predefined policy matrix is constructed, wherein the policy matrix includes data classification, storage resources and node status; the data classification is based on the basic access mode type and data value level in the composite classification label; the storage resources include storage medium type and compression algorithm level; the node status includes real-time load indicators of physical nodes.

7. A distributed data storage management method according to claim 6, characterized in that: Positioning an initial matching unit of a strategy matrix according to the composite classification label; When the CPU utilization of a node exceeds the first threshold, the compression algorithm level is automatically downgraded; when the memory usage exceeds the second threshold, the compression algorithm with low memory usage is preferentially selected; Dynamically adjust bandwidth allocation weight coefficients based on real-time latency data of the network topology and output the final strategy.

8. A distributed data storage management method according to claim 1, characterized in that: The smart contract records the key decision parameters of the strategy matching process, the execution results of resource scheduling, and abnormal operation events and handling measures; the key decision parameters include the execution efficiency index of the compression algorithm, the performance index of the storage medium, and the bandwidth utilization index; the execution results of the resource scheduling include the storage medium allocation certificate, bandwidth weight calculation data, and the compression algorithm execution fingerprint; the abnormal operation events and handling measures include the event type code, the compensation result hash, and the node confirmation signature.

9. A distributed data storage management method according to claim 1, characterized in that: The real-time acquisition strategy execution indicators automatically adjust the classification model parameters and resource scheduling weights through the control loop, including: The policy execution indicators include compression execution indicators, storage performance indicators and network transmission indicators; Adjusting the classification model parameters includes adjusting the dynamic judgment threshold in the cold data judgment rule when the cold data misjudgment rate exceeds the threshold, and optimizing the sliding time window size of the timing analysis model when the pattern classification error rate increases; adjusting the resource scheduling weight includes dynamically updating the weight distribution of the storage resource dimension according to the storage performance index, and correcting the bandwidth allocation weight coefficient in real time in combination with the network transmission index.

10. A distributed data storage management system, applied to a distributed data storage management method according to any one of claims 1 to 9, characterized in that: The system includes data acquisition module, cold data analysis module, intelligent classification module, strategy execution module and dynamic optimization module; The data acquisition module includes an acquisition module, a preprocessing module, a feature extraction module and a mapping module; the acquisition module is used to collect node access logs, metadata and content features; The preprocessing module is used to preprocess the collected data; the feature extraction module is used to generate a standardized feature vector; the mapping module is used to establish a mapping table between feature vectors and physical nodes; The cold data analysis module includes a threshold module, a multi-dimensional determination module, and a node positioning module; the threshold module is used to generate a dynamic cold data determination threshold based on the cluster status; the multi-dimensional determination module analyzes data object attributes based on the access frequency dimension, storage duration dimension, and association relationship dimension to mark cold data; the node positioning module locates cold data distribution nodes using a consistent hashing algorithm; The intelligent classification module includes a time series analysis module, a value assessment module, and a rule matching module; the time series analysis module uses a time series analysis model to classify cold data into permanent, periodic, and sudden basic access patterns; the value assessment module is used to perform standardized value scoring; and the rule matching module loads a business rule library to generate matching tags. The policy execution module includes a policy matrix module and a dynamic decision module; the policy matrix module is used to construct a policy matrix including data classification, storage resources and node status; the dynamic decision module dynamically selects compression algorithm, storage medium and allocates bandwidth; The dynamic optimization module includes a smart contract module and an indicator monitoring module; the smart contract module records the key decision parameters of the strategy matching process, the execution results of resource scheduling, and abnormal operation events and handling measures; The indicator monitoring module collects compression execution indicators, storage performance indicators and network transmission indicators in real time.

Images