Data management method based on distributed storage system

Abstract

The present invention discloses a data management method based on a distributed storage system, relates to the technical field of data management, and is used to solve the problem of storage management behavior strategy mismatch; based on the dynamic perception of data access behavior in the distributed storage system, the present invention constructs a behavior feature vector that integrates access mutation rate, periodicity and access span, completes behavior pattern recognition and policy structure generation, and through node resource status perception, constructs a cost function to achieve optimal deployment and hierarchical storage of replicas, performs compression and consistency configuration, and after the policy is implemented, analyzes the compressibility and offset trend of the behavior path, extracts scheduling management information, and generates stable execution or policy adjustment signals, thereby reducing the policy execution mismatch problem caused by the lack of behavior perception in the distributed storage system, realizing behavior-driven policy closed-loop control and resource scheduling optimization, thereby improving the data management efficiency and policy adaptability of the distributed storage system.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and more particularly, to a data management method based on a distributed storage system. Background Art

[0002] As a storage architecture designed for massive data, high scalability, and high elasticity and fault tolerance, distributed storage systems are widely used in cloud platforms, Internet services, edge computing, and data-intensive application scenarios. Distributed storage systems divide data into multiple fragments and store them in a distributed manner on different storage nodes to achieve data redundancy backup, concurrent read and write access, and fault node replacement. The system usually has basic functions such as replication mechanism, heat awareness, multi-copy scheduling, consistency control, and supports decoupling of storage services from upper-level business systems.

[0003] Deficiencies in existing technologies: In distributed storage systems, strategies such as data copy deployment, compression control, and storage heat tiering generally rely on preset rules or static templates, and lack the ability to dynamically perceive and classify data access behaviors. When data objects experience behavioral changes such as access mode drift, sudden high-frequency reading, phased use, or collaborative access during operation, the storage system is unable to effectively identify and respond to policies, resulting in continuous mismatch of the original strategy, imbalanced distribution of redundant copies, misaligned configuration of hot and cold resources, increased scheduling path delays, and other problems. Especially in a multi-node, heterogeneous resource deployment environment, the lack of dynamic coupling between strategy and resource status further limits the accuracy, adaptability, and resource optimization capabilities of data management strategies. Summary of the Invention

[0004] In order to overcome the above-mentioned defects of the prior art, there is the following solution to solve the problem of storage management behavior policy mismatch in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] The data management method based on the distributed storage system includes the following steps:

[0007] Collect and standardize data access logs in distributed storage systems, construct time-ordered access behavior sequences, extract access mutation rate, access periodicity, and access span indicators to generate a fused behavior feature vector;

[0008] Based on the fused behavioral feature vector, behavioral pattern clustering and labeling are performed to build a mapping relationship between corresponding behavioral categories and strategic intents, and a policy structure is generated to control resource allocation.

[0009] Schedule and calculate the node resource status based on the policy structure, build a cost function to select the optimal node, complete replica creation, compression configuration and layered deployment, establish a consistent path and record the scheduling results, and generate replica operation status information for strategy evolution;

[0010] Analyze the behavioral path response process after strategy deployment, obtain and analyze the management scheduling information in the behavioral path compression and execution displacement trends, and perform scheduling management based on the different signals generated by the analysis results.

[0011] In a preferred embodiment, data access logs in a distributed storage system are collected and standardized to construct a time-ordered access behavior sequence. The specific steps include:

[0012] Collect all access events of data objects in the data access log. Each access event record includes the access time, initiating node, operation type, and data volume.

[0013] Standardize the collected data, including unifying timestamp accuracy to seconds in UTC format, normalizing operation types to a unified identifier, filtering out non-business access, and merging repeated access requests within a set time interval;

[0014] The access logs of the standardized data objects are transformed into a time-ordered event sequence, and the preprocessed event sequence is converted into a structured access behavior sequence.

[0015] In a preferred embodiment, extracting the access mutation rate, access periodicity, and access span indicators to generate a fused behavior feature vector includes the following steps:

[0016] Each valid access event in the event sequence is regarded as a behavior node of the data object, and a continuous behavior trajectory is formed;

[0017] Extract access rate fluctuation characteristics, count the access volume difference and time interval difference between adjacent accesses, and determine whether there is a rate mutation in the access behavior. If there is a rate mutation, obtain it and use it as the access mutation rate indicator;

[0018] Extract periodic features, perform frequency domain analysis on access time series, observe activities over time periods and use them as indicators of access concentration;

[0019] Extract access concentration and span. The time interval between the first and last visits is used as the behavior span. The ratio of the access concentration to the behavior span is used as the access concentration. The access concentration and span are summed up to form the access span indicator.

[0020] Frequency domain analysis is used to detect the periodic trend of access behavior sequences. The amplitude of behavior fluctuation corresponding to each frequency is calculated. The period component with the largest amplitude of behavior fluctuation is taken as the dominant period. A continuous time series composed of the trend of the number of visits per unit time is used as the trajectory of access density changes.

[0021] Analyze the contextual tag information of data object binding. The contextual tag information dimensions include data source module, tenant or project code, data function role label, whether the data is accessed collaboratively, and task priority.

[0022] Context vector representation is performed based on the context embedding function, and the context label is projected to adjacent positions in the Euclidean space according to the semantic structure;

[0023] After completing the access behavior modeling, a fused behavior feature vector is generated.

[0024] In a preferred embodiment, behavioral pattern clustering and label division are performed based on the fused behavioral feature vector, a mapping relationship between corresponding behavioral categories and policy intentions is constructed, and a policy structure is generated to perform resource allocation control. The specific steps include:

[0025] Use logarithmic functions to normalize the values of different feature dimensions, and use a clustering method based on density gradient optimization to divide the feature space;

[0026] Define behavior categories corresponding to different access patterns and business semantics. Specific behavior categories include: periodic access, sudden hotspot, long-term cold storage, collaborative sharing, and unstable.

[0027] Each data object is assigned a unique behavior label;

[0028] Map the behavior label to a set of policy parameters. The policy structure includes the following fields: number of replicas, storage heat level identifier, whether compression is enabled, replica deployment area, and consistency level requirement.

[0029] Different rule mappings are performed based on different behavior labels;

[0030] For periodic access, set up regular compression and delayed replica refresh policies;

[0031] For sudden hotspots, set the primary replica to reside and enable cache-level snapshots;

[0032] For long-term cold storage, compression, copy redundancy, and archival storage are performed;

[0033] For collaborative sharing, set up replica distribution and enable request scheduling;

[0034] For unstable types, periodic behavior identification is triggered.

[0035] In a preferred embodiment, the node resource status is scheduled and calculated according to the policy structure, a cost function is constructed to select the optimal node, replica creation, compression configuration and layered deployment are completed, a consistency path is established and the scheduling results are recorded, and replica operation status information is generated for strategy evolution, including the following steps:

[0036] A set of dynamic state parameters is constructed for each storage node to form a resource state vector. The resource state vector includes CPU usage, disk space usage, network bandwidth usage, real-time packet loss rate, and node error history.

[0037] Based on satisfying policy constraints, replica deployment is performed and a cost function is constructed. The cost function measures the cost of each storage node as a candidate replica deployment point, and storage nodes are selected based on the cost function.

[0038] Sort candidate storage nodes based on the cost function result, and select the node combination with the minimum total cost for replica deployment;

[0039] Construct an ascending sorted candidate set based on the cost function, select nodes based on the ascending sorted candidate set and form a target deployment set;

[0040] Issue deployment instructions to each node in the target deployment set to execute the deployment operation:

[0041] The deployment operation refers to the initialization of physical writes to data replicas. If the compression strategy is 1, the compressor is enabled to perform structural compression on the data replicas. The replicas are then stored in the corresponding storage media based on the storage popularity level. Then, according to the consistency level requirements, a data synchronization path and confirmation mechanism are established between the master and slave replicas.

[0042] After the deployment is complete, a scheduling task execution log is generated, binding the task number and behavior label.

[0043] In a preferred embodiment, analyzing the behavior path response process after policy deployment, obtaining and analyzing management scheduling information in behavior path compression and execution displacement trends, includes the following steps:

[0044] Obtaining management scheduling information generated during the behavior path compression and execution displacement trend analysis process, the management scheduling information including the behavior path compression index and the strategy guidance offset displacement index;

[0045] The behavior path compression index indicates whether the actual access path of the data object tends to be stable and convergent after the policy deployment;

[0046] The policy-guided offset index indicates the degree of topological offset between the target deployment area set by the policy structure and the actual replica scheduling and access behavior;

[0047] The obtained behavior path compression index and strategy guidance offset displacement index are jointly generated into a data management coefficient;

[0048] The behavior path compression index is positively correlated with the data management coefficient, and the strategy guidance offset displacement index is negatively correlated with the data management coefficient.

[0049] In a preferred embodiment, scheduling management is performed based on different signals generated by the analysis results, including the following steps:

[0050] Compare the generated data management coefficient with the set data management threshold;

[0051] If the data management coefficient is greater than or equal to the data management threshold, a stable execution signal is generated, and the current policy structure and corresponding scheduling path are maintained, and the original scheduling plan continues to be executed;

[0052] If the data management coefficient is less than the data management threshold, an evolution adjustment signal is generated, the scheduling plan is stopped, the behavior re-identification mechanism is triggered, the behavior label is reallocated, and a new policy structure is mapped.

[0053] The technical effects and advantages of the data management method based on the distributed storage system of the present invention are as follows:

[0054] The present invention achieves dynamic perception and differentiated control of data access behavior in distributed storage systems by building a data management mechanism based on behavior perception and policy linkage. By collecting and standardizing access logs, extracting access mutation rate, periodicity, and access span indicators, and generating behavioral feature vectors that integrate contextual semantics, the present invention identifies behavioral patterns based on clustering methods and maps them to generate a policy structure with the number of replicas, compression strategy, heat level, and consistency requirements. A scheduling cost function is constructed in combination with node resource status to achieve optimal deployment, hierarchical storage, and consistent configuration of replicas.

[0055] After the policy is executed, the compressibility and deviation trend of the behavior path are analyzed, the management and scheduling features are extracted, the data management coefficient is constructed and a stable execution or policy adjustment signal is generated, thereby realizing behavior-driven closed-loop policy control and dynamic resource scheduling, reducing the policy execution mismatch problem caused by the lack of behavior perception in the distributed storage system, and significantly improving the accuracy of data management of the distributed system and the stability of system operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 The figure is a flow chart of the data management method based on the distributed storage system of the present invention. DETAILED DESCRIPTION

[0057] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0058] In order to achieve the above objectives, Figure 1 A structural diagram of a data management method based on a distributed storage system of the present invention is given, which specifically includes the following steps:

[0059] Collect and standardize data access logs in distributed storage systems, construct time-ordered access behavior sequences, extract access mutation rate, access periodicity, and access span indicators to generate a fused behavior feature vector;

[0060] Based on the fused behavioral feature vector, behavioral pattern clustering and labeling are performed to build a mapping relationship between corresponding behavioral categories and strategic intents, and a policy structure is generated to control resource allocation.

[0061] Schedule and calculate the node resource status based on the policy structure, build a cost function to select the optimal node, complete replica creation, compression configuration and layered deployment, establish a consistent path and record the scheduling results, and generate replica operation status information for strategy evolution;

[0062] Analyze the behavioral path response process after strategy deployment, obtain and analyze the management scheduling information in the behavioral path compression and execution displacement trends, and perform scheduling management based on the different signals generated by the analysis results.

[0063] In distributed storage systems, the access behavior of different data objects shows significant heterogeneity: some data is highly periodic, some data is accessed at high frequency for short periods of time in specific scenarios, and some data may be in a low-frequency, passive reading state for a long time.

[0064] Step 1: Model and extract data access behavior features. This involves modeling the access behavior of each data object and extracting behavior vectors that reflect its timing, operational, and contextual features. The specific steps are as follows:

[0065] Data access logs are collected and standardized. All access events to data objects in the data paths integrated into the distributed storage platform are collected. Each access event record includes the access time, initiating node, operation type (such as read, write, delete, etc.), and data volume.

[0066] Standardize the collected data. This includes standardizing timestamps to second-level UTC format, unifying operation types with unified identifiers, filtering out non-business access requests, including health check and heartbeat background task access, and merging repeated access requests within a set time interval (e.g., 0.1 second).

[0067] Finally, the access log of each data object forms a time-ordered event sequence in the format of (access time, operation type, access volume, source identifier, task context), which serves as the original input for subsequent behavioral analysis.

[0068] Construct access behavior sequences and extract dynamic features to identify access patterns of data objects in the time dimension. Convert the preprocessed event sequence into a structured access behavior sequence. During the construction process, define each valid access event in the event sequence as a behavior node of the data object, forming a continuous behavior trajectory.

[0069] A set of dynamic access features is extracted to determine its access rate, fluctuation degree, and mutation characteristics, and modeled from the following three dimensions:

[0070] Access rate fluctuation characteristics: Statistically calculate the difference in access volume and time interval between adjacent accesses to determine whether there are significant rate mutations in access behavior. If a certain access segment frequently alternates between writes and reads, or if the data volume suddenly increases or decreases, its behavior pattern is more similar to highly volatile data, such as AI training task input files or log aggregation systems. This is used as the designed access mutation rate indicator to represent the proportion of mutation points in the entire sequence.

[0071] Periodicity feature extraction: Periodicity is a core indicator for measuring behavioral regularity. Frequency domain analysis is performed on access time series to observe activity intensity within specific time periods (such as minutes, hours, and days). If behavior recurs within 24 hours, it indicates that the data is related to daily scheduled tasks. Hourly peaks of high-frequency access may indicate database caching or periodic synchronization. This is used as an indicator of access periodicity. By scanning multiple candidate periods (such as 10 minutes, 1 hour, and 1 day), the dominant period is extracted and the strength of the periodicity is determined.

[0072] Access concentration and span extraction, that is, defining the behavior span as the time interval between the first and last two visits, and using the ratio of the access concentration before and after in the behavior span as the access concentration. This item is used to distinguish between stable access data and periodic access data. For example, if a certain data is accessed 100 times in the last hour, but there is no access record in the previous 24 hours, it is considered a periodic access type and the above content is used as the access span indicator.

[0073] Some data objects have obvious periodic behavior characteristics, such as scheduled backup tasks, periodic indicator writing, and daily log archiving;

[0074] Perform periodic feature recognition and behavior density modeling on the data, and use frequency domain analysis to detect periodic trends in access behavior sequences;

[0075] By constructing a frequency scanning grid at multiple time scales (such as ten minutes, one hour, and one day), the behavioral fluctuation amplitude corresponding to each frequency is calculated respectively, and the periodic component with the largest behavioral fluctuation amplitude is regarded as the dominant period. In the identification results, if a specific period shows a continuous and stable amplitude peak, it can be considered that the data has a periodic access characteristic. This feature serves as an important criterion for distinguishing periodic from non-periodic data.

[0076] At the same time, based on the trend of the number of accesses per unit time, the access density change trajectory is calculated. The access density change trajectory is a continuous time series consisting of the trend of the number of accesses per unit time over consecutive time periods. It helps identify short-term burst access or distributed behavior shifts. Combining the mutation rate with access density fluctuations, it can determine whether there is a behavioral phase change, such as access evolution from cold to hot or hot to cold.

[0077] In distributed storage systems, behavioral patterns extracted from access time series (such as mutation rate, periodicity, and access density) can reflect certain usage trends, but they still lack auxiliary information about the usage context. In actual business, the access behavior of data objects is often influenced by non-time series factors such as their business role, tenant scenario, generation source, and task binding relationship. For example, even if two data objects have similar access frequency and periodic structure, one may be a log archive file and the other a model parameter snapshot. The two should have completely different replica deployment, consistency requirements, and scheduling strategies. Therefore, relying solely on behavioral characteristics may lead to policy misjudgment or classification ambiguity.

[0078] The contextual tag information bound to the data object includes the following dimensions: data source module (such as log center, AI task output, archiving system); tenant or project code (used for resource isolation and access privilege control); data function role label (such as input source, intermediate task results, and final output); whether data is accessed collaboratively (such as whether it is accessed by multiple nodes); task priority (Such as whether to bind real-time tasks or asynchronous tasks).

[0079] Introducing context embedding function: ,in, is the j-th context label, j = 1, 2, ..., k, k is the number of label categories; is the context encoding function (which can use distributed embedding table, sparse hash map or low-rank coding); is the context vector representation, Represents the real number R dimensional vector space, generally dense vectors, for example, vectors in three-dimensional space belong to ;

[0080] Project context labels with similar semantic structures to adjacent positions in the Euclidean space to maintain the consistency of their task semantics or resource preferences;

[0081] After completing the access behavior modeling part (such as mutation rate, cycle, access concentration, etc.), let the behavior vector be , Represents the behavior vector of the i-th data object, that is, the feature representation extracted by modeling its access behavior sequence, Represents the real number R dimensional vector space, the final fusion behavior feature vector is constructed as follows: ,in, Represents vector concatenation operation; is the full-dimensional behavior representation that is ultimately used for behavior pattern recognition. Represents the real number R dimensional vector space.

[0082] Step 1 has completed modeling the access behavior of each data object and output structured behavior feature vectors. These vectors carry the behavior information of the data object. However, the feature vectors themselves cannot directly guide the resource allocation and policy control adopted by the system. Therefore, the behavior feature vectors need to be parsed into clear behavior category labels and further bound to specific storage policies to implement differentiated responses.

[0083] Step 2: Behavior pattern recognition and policy intent mapping. The specific steps are as follows:

[0084] Nonlinear mapping functions (such as logarithmic function and Sigmoid function) are introduced to perform numerical regularization on different feature dimensions. Taking logarithmic mapping as an example, its form is: ,in, is an adjustment parameter used to control the degree of compression, usually selected between 0.1 and 1.0; x is the original eigenvalue, that is, the original value of a dimension in the behavioral eigenvector; The normalized eigenvalues after nonlinear mapping, and log represents the logarithmic function. This operation compresses large values into a comparable range while enhancing the discriminative power of low-value features. It is particularly useful for accessing low-amplitude, high-information-density features such as mutation rates. Once completed, all features are normalized to a uniform relative magnitude, facilitating clustering algorithms to reasonably determine boundaries in multidimensional space.

[0085] Perform behavioral pattern clustering and label assignment, use density gradient optimization-based clustering methods (such as spectral clustering or minimum redundancy clustering based on bimodal point convergence) to divide the feature space, avoid excessive assumptions about Euclidean space distribution, and define the center of each class as , For real number R dimensional vector space, is the center of the overall behavior characteristic distribution of the clustered data object J. Assign behavioral category labels: ,in It is a non-Euclidean metric function, such as the projected distance function or the negative entropy kernel function under the Mahalanobis distance structure, which is used to avoid the misjudgment of non-convex distribution by linear distance. Represents the input vector of the i-th data object after feature normalization or mapping;

[0086] Several typical behavior categories are predefined, corresponding to different access modes and business semantics, including: periodic access type , with fixed frequency timing write or query behavior; sudden hotspot type , high-frequency writing or reading in a short period of time; long-term cold storage type , no active access for a long time or only low-frequency reading; collaborative sharing type , frequently accessed by multiple nodes or users; unstable , the behavior pattern changes frequently and has access noise.

[0087] Each data object is ultimately assigned a unique behavior label ,The label is used as the action recognition result for policy reasoning.

[0088] The behavior label itself is only a semantic classification result and does not have the ability to directly control the storage system operation. Therefore, the behavior label is mapped to a set of policy parameters;

[0089] Design strategy structure ,The policy structure includes the following fields: the number of replicas (determines the redundancy level), the storage heat tier identifier (determines the data placement tier, such as hot storage, archive, or cold chain), whether compression is enabled (controls whether compression occurs and the compression depth), the replica deployment area (e.g., deployed to edge nodes or backbone cores), and the consistency level requirement (controls the replica synchronization strength);

[0090] Depending on the behavior label, the mapping rules are as follows: for periodic access, set regular compression and delayed replica refresh strategies; for sudden hotspots, set the primary replica to reside in the core node and enable cache-level fast snapshots; for cold data, enable high compression, low replica redundancy, and archival storage; for collaborative access, set multi-point distribution of replicas and enable request scheduling nearby; for unstable types, trigger short-cycle behavior re-identification and strategy rapid iteration channels.

[0091] The policy structure is written into the policy configuration center, and an index is created for each data object for subsequent policy execution scheduling modules to read and perform operations such as replica creation, compression management, hot and cold tiering, and data migration. If the system has an online policy adjustment mechanism, the historical sequence of behavior labels and behavior characteristics will be continuously monitored to trigger the policy evolution mechanism.

[0092] Each data object is assigned a behavior tag, and a policy structure is generated based on it. This policy structure includes key fields such as the number of replicas, replica deployment area, storage popularity level, compression requirements, and consistency level. The policy structure is still a static configuration template and cannot be directly executed. Due to the dynamic changes in the resource status of each node in the distributed system, frequent fluctuations in network status, and significant differences in disaster recovery costs, the effectiveness of policy implementation varies greatly over time and space.

[0093] In distributed storage systems, nodes exhibit significant heterogeneity and load volatility. Consequently, the execution costs of the same policy on different nodes at different points in time can vary significantly. For example, if a node with nearly saturated disk I / O continues to receive replica write requests, performance will be further degraded, potentially leading to task backlogs or service drift.

[0094] Step 3: Perform policy linkage execution and resource-aware scheduling. This involves selecting the optimal storage node combination, path scheduling, and storage layout while meeting the policy intent. The specific steps are as follows:

[0095] A set of dynamic state parameters is constructed for each storage node to form a resource state vector. This vector should at least include the following: CPU utilization, reflecting computing resource usage; disk space utilization, reflecting storage pressure; current I / O load (such as write / read request queue length); network bandwidth utilization and real-time packet loss rate; node error history (such as the number of abnormal responses within 30 minutes); and task completion reliability assessment (based on historical task execution delays and failures).

[0096] Interpret policy constraints and construct a scheduling objective function, mapping the policy to the optimal node combination in the current resource state. This involves completing replica deployment at the minimum cost while satisfying policy constraints. The cost of each storage node as a candidate replica deployment point is measured, and storage node selection is performed based on this cost function.

[0097] The cost function is constructed as follows: ,in, is the node CPU usage; is the node disk usage; is the current bandwidth throughput of the node; is a constant to prevent division by zero; 、 、 They are the weight of the CPU occupancy rate of the scheduling node, the weight of the node disk usage rate, and the weight of the node's current bandwidth throughput, which can be defined differently according to actual needs; is the execution cost of deploying the replica of the i-th data object to the j-th storage node;

[0098] It should be noted that the task completion reliability assessment is used to measure the execution success rate and execution quality of a storage node for system-assigned tasks (such as replica creation, synchronous writing, compression operations, heat migration, etc.) in the past scheduling cycle. A segmented quantitative score is given according to the completion quality of each task as a task completion reliability assessment; the cost function design idea is that the tighter the resource usage, the weaker the network throughput, and the lower the historical task success rate, the greater the scheduling cost of the node should be, and the less suitable it is as a target replica deployment point.

[0099] Based on the cost function result, the candidate storage nodes are sorted and the node combination with the minimum total cost is selected for replica deployment;

[0100] Construct an ascending candidate set based on the cost function and sort them according to the size of the cost function. The smaller the cost function, the higher the ranking. Select the first N nodes in the ascending candidate set to form the target deployment set.

[0101] Send deployment instructions to each node in the target deployment set and perform the following deployment operations:

[0102] The deployment operation refers to the initialization of physical writes to data replicas. If the compression strategy is 1, the compressor is enabled to perform structural compression on the replicas. Based on the storage heat level identifier, the replicas are stored in the corresponding level storage media (such as SSD cache and HDD archive). Then, according to the consistency level requirements, a data synchronization path and confirmation mechanism are established between the master and slave replicas.

[0103] After the deployment is complete, a scheduling task execution log is generated, binding the task number and behavior label;

[0104] Write the behavior label, deployment replica list, and final policy structure into the policy state table, and record the actual deployment success timestamp, path, and resource usage. Set the current state pointer of the data object to point to the policy replica mapping. Initialize the monitoring task for each replica, record the synchronization status, access delay, and error count, and use them to generate the policy execution feedback vector.

[0105] Step 4: Conduct behavioral feedback closed-loop strategy self-evolution analysis. The specific steps are as follows:

[0106] Analyze the behavioral path response process after policy deployment and obtain management scheduling information from the behavioral path compression and execution displacement trends. The management scheduling information includes the behavioral path compression index and the policy-guided offset displacement index. The behavioral path compression index is used to measure whether the actual access path of the data object after policy deployment tends to be stable and convergent.

[0107] The behavioral path compression index is based on the ratio of the total number of access path hops to the number of unique paths. By modeling the hopping nature and distribution aggregation of access paths, it reflects whether the system has formed a structurally consistent behavioral path after policy execution. It measures the distribution aggregation and execution trajectory compression of access paths, thereby reflecting the convergence state of behavioral paths under policy guidance.

[0108] If the access path concentration is high, the repetitiveness is strong, and the hop count distribution is stable, it means that the scheduling path has been gradually compressed and a stable access trajectory has been formed. The behavior path compression index is close to 1, indicating that the policy scheduling has a good path aggregation effect. Conversely, if the access path distribution is divergent, changes frequently, and the hop count is inconsistent, the behavior path compression index value approaches 0, indicating that the system has behavioral path oscillation and the policy's guiding effect on behavior is weakened. The role of this index is to assist in determining whether the policy has achieved structural convergence at the path level. It is a comprehensive evaluation indicator of the path consistency of the policy execution results and the stability of the behavioral response.

[0109] The logic for obtaining the behavior path compression index is as follows:

[0110] After the strategy structure is executed, in the specified observation period Collect all access request records associated with the data object. Indicates the starting time point of the observation period, Indicates the duration (time length) of the observation period, i.e. The length of the time window to start backward observation, representing the node jump structure of each visit as an ordered path vector , forming a path set: , u is the number of visits during the observation period;

[0111] Path Collection Count the total number of hops on all paths: ,in, Indicates the path Number of hops in

[0112] Identify the number of unique structures in a path , the expression is: ,in, , Indicates from The unique path set after deduplication is based on the node sequence equality judgment; the behavior path compression index is calculated, and the calculation expression is: .

[0113] The policy-guided offset index is used to measure the degree of topological offset between the target deployment area set by the policy structure and the actual replica scheduling and access behavior;

[0114] The policy-guided offset displacement index calculates the minimum structural displacement distance of the actual replica access or scheduling landing point relative to the policy target node set and constructs a logarithmic proportional measurement of the path displacement to reflect whether policy execution has caused obvious spatial deflection at the physical node level. If actual access still revolves around the policy's expected area and system scheduling remains structurally concentrated, the policy-guided offset displacement index value is low. If the actual operation node gradually moves away from the initial target area after policy deployment, resulting in scheduling drift, the policy-guided offset displacement index rises rapidly, indicating that the policy control power is gradually weakening in the geographical or topological dimension. The policy-guided offset displacement index can be used to identify structural offset problems in policy execution and guide whether policy redirection, node migration, or consistency level adjustment is needed. It is an important spatial dimension indicator of policy control effectiveness.

[0115] The logic for obtaining the strategy-guided offset displacement index is as follows:

[0116] Extract the policy target replica node set when the data object is scheduled from the system deployment log: ,in, For the j-th storage node in the distributed system, within the set policy execution cycle, the actual nodes involved in the scheduling behavior, access behavior and copy management behavior generated by the data object are collected to form the actual operation landing point set , where N is the complete set of callable operation response actions, is the kth storage node in the distributed system;

[0117] Calculate two nodes 、 The structural distance in the system network layer determines the system boot offset: , the calculation strategy guides the offset displacement index, and the calculation expression is: .

[0118] It should be noted that the policy structure contains a deployment location constraint field, which indicates the set of node areas where the policy expects replicas to be deployed, such as "edge node set", "cross-availability zone distribution", "core backbone node", etc. The policy target replica node set is the expected target operation area when the policy is deployed; the actual operation landing point set reflects the actual operation landing point of the system under the guidance of the current policy; the structural distance is determined by the number of network layer hops (such as the three-layer switching distance), which indicates the minimum total aggregate displacement in the topology between the actual operation behavior of the system and the policy target area.

[0119] The obtained behavior path compression index and strategy guidance offset displacement index are combined to generate the data management coefficient. The expression is: , where 、 are the preset proportional coefficients of the behavior path compression index and the strategy guidance offset displacement index, respectively, and 、 Both are greater than 0.

[0120] The specific method of jointly generating the data management coefficient may involve multiple algorithms and models, which depends on the actual situation and application requirements. In this embodiment, a weighted summation method can be used to combine the behavior path compression index and the strategy-guided offset displacement index to generate a comprehensive data management coefficient. This data management coefficient can be used as an input for distinguishing nodes in the distributed storage path and used to determine the final storage path situation.

[0121] It should be noted that the size of the preset proportional coefficient is a specific numerical value obtained by quantifying each parameter. In order to facilitate subsequent comparison, the size of the coefficient depends on the amount of sample data and the preset proportional coefficient initially set by technical personnel in this field for each set of sample data. It is not unique, as long as it does not affect the proportional relationship between the parameter and the quantized numerical value. For example, the behavior path compression index is proportional to the data management coefficient. The behavior path compression index and the strategy-guided offset displacement index are normalized to have the same dimension and range. This can be achieved by subtracting the mean from the original data and dividing it by the standard deviation, or mapping the data to the range of [0, 1].

[0122] The larger the behavior path compression index and the smaller the policy-guided offset displacement index, the larger the jointly generated data management coefficient. This indicates that under the current policy control, the access path of the data object exhibits a high degree of structural aggregation. At the same time, the scheduling behavior is highly concentrated in the target area set by the policy. The distributed storage system achieves good scheduling convergence and policy response consistency at both the behavior path layer and the spatial structure layer. This state reflects the strong correlation between the behavior modeling results and the policy structure, indicating that the policy deployment is accurate and the execution is stable. The storage system has good resource scheduling controllability and behavior management capabilities, which is an ideal embodiment of the behavior-driven policy execution effect.

[0123] The smaller the behavior path compression index and the larger the policy-guided offset displacement index, the smaller the jointly generated data management coefficient, indicating that under the control of the current policy structure, the access path of the data object still has significant jumps or structural divergence. At the same time, the actual scheduling node deviates significantly from the policy target deployment area, and the system scheduling behavior fails to effectively converge to the expected structure. This state indicates that the current behavior characteristics may have evolved, or the policy deployment plan cannot adapt to changes in system resources, resulting in uncontrolled behavior response, path dispersion, and declining policy execution effect. At this time, the behavior re-identification and policy structure reconstruction mechanism should be activated to re-establish a stable and efficient data management chain;

[0124] Comparing the generated data management coefficient with the pre-set data management threshold to generate a stable execution signal and an evolution adjustment signal;

[0125] If the data management coefficient is greater than or equal to the data management threshold, a stable execution signal is generated, indicating that the current access path of the data object has formed a stable aggregation trend under policy control, and the actual scheduling behavior is highly consistent with the policy deployment area. The policy control effect is good and the resource allocation is reasonable. The system does not need to adjust the policy, maintains the current policy structure and corresponding scheduling path, and continues to execute the original scheduling plan;

[0126] If the data management coefficient is less than the data management threshold, an evolution adjustment signal is generated, indicating that the current behavior path has a clear divergence trend, or the actual replica scheduling area has produced an offset that causes policy guidance failure, resulting in reduced system scheduling efficiency and policy adaptability. The behavior re-identification mechanism is triggered, and the scheduling plan is stopped. The behavior feature vector is recalculated, the behavior label is redistributed, and the new policy structure is mapped. At the same time, the replica deployment and resource path are updated, and the policy self-evolution process begins.

[0127] Through the signal linkage mechanism, the system can achieve closed-loop judgment and differentiated response control of the policy execution status based on the comparison results of the threshold. The system can dynamically judge whether the current policy is still effective and adaptable without relying on manual rules, thereby timely triggering policy reconstruction, resource rescheduling or behavior re-identification, improving the accuracy and explainability of data management decisions, and enhancing the intelligent regulation capabilities of distributed storage systems in multiple scenarios and multiple behavior modes, which helps to achieve more efficient and reliable data storage strategy optimization goals.

[0128] It should be noted that the relevant threshold information in this embodiment is pre-set by professionals and will not be explained in detail here. Some parameter English letters in the embodiments have the same situation, but different meanings are explained when used, and will not be explained one by one here.

[0129] The present invention achieves dynamic perception and differentiated control of data access behavior in distributed storage systems by building a data management mechanism based on behavior perception and policy linkage. By collecting and standardizing access logs, extracting access mutation rate, periodicity, and access span indicators, and generating behavioral feature vectors that integrate contextual semantics, the present invention identifies behavioral patterns based on clustering methods and maps them to generate a policy structure with the number of replicas, compression strategy, heat level, and consistency requirements. A scheduling cost function is constructed in combination with node resource status to achieve optimal deployment, hierarchical storage, and consistent configuration of replicas.

[0130] After the policy is executed, the compressibility and deviation trend of the behavior path are analyzed, the management and scheduling features are extracted, the data management coefficient is constructed and a stable execution or policy adjustment signal is generated, thereby realizing behavior-driven closed-loop policy control and dynamic resource scheduling, reducing the policy execution mismatch problem caused by the lack of behavior perception in the distributed storage system, and significantly improving the accuracy of data management of the distributed system and the stability of system operation.

[0131] The above formulas are all dimensionless and numerical calculations. The formulas are obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters in the formulas are set by technicians in this field according to actual conditions.

[0132] The above embodiments may be implemented in whole or in part through software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product.

[0133] Those skilled in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0134] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0135] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0136] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data management method based on a distributed storage system, characterized by: The steps include: Collect and standardize data access logs in distributed storage systems, construct time-ordered access behavior sequences, extract access mutation rate, access periodicity, and access span indicators to generate a fused behavior feature vector; Based on the fused behavioral feature vector, behavioral pattern clustering and labeling are performed to build a mapping relationship between corresponding behavioral categories and strategic intents, and a policy structure is generated to control resource allocation. Schedule and calculate the node resource status based on the policy structure, build a cost function to select the optimal node, complete replica creation, compression configuration and layered deployment, establish a consistent path and record the scheduling results, and generate replica operation status information for strategy evolution; Analyze the behavioral path response process after strategy deployment, obtain and analyze the management scheduling information in the behavioral path compression and execution displacement trends, and perform scheduling management based on the different signals generated by the analysis results; Based on the fused behavioral feature vector, behavioral patterns are clustered and labeled, a mapping relationship between corresponding behavioral categories and strategic intent is constructed, and a policy structure is generated to control resource allocation. The specific steps include: Use logarithmic functions to normalize the values of different feature dimensions, and use a clustering method based on density gradient optimization to divide the feature space; Define behavior categories corresponding to different access patterns and business semantics. Specific behavior categories include: periodic access, sudden hotspot, long-term cold storage, collaborative sharing, and unstable. Each data object is assigned a unique behavior label; Map the behavior label to a set of policy parameters. The policy structure includes the following fields: number of replicas, storage heat level identifier, whether compression is enabled, replica deployment area, and consistency level requirement. Different rule mappings are performed based on different behavior labels; For periodic access, set up regular compression and delayed replica refresh policies; For sudden hotspots, set the primary replica to reside and enable cache-level snapshots; For long-term cold storage, compression, copy redundancy, and archival storage are performed; For collaborative sharing, set up replica distribution and enable request scheduling; For the unstable type, trigger periodic behavior identification; The node resource status is scheduled and calculated based on the policy structure, and a cost function is constructed to select the optimal node. Replica creation, compression configuration, and layered deployment are completed. A consistent path is established and the scheduling results are recorded. Replica operation status information is generated for policy evolution. The following steps are included: A set of dynamic state parameters is constructed for each storage node to form a resource state vector. The resource state vector includes CPU usage, disk space usage, network bandwidth usage, real-time packet loss rate, and node error history. Based on satisfying policy constraints, replica deployment is performed and a cost function is constructed. The cost function measures the cost of each storage node as a candidate replica deployment point, and storage nodes are selected based on the cost function. Sort candidate storage nodes based on the cost function result, and select the node combination with the minimum total cost for replica deployment; Construct an ascending sorted candidate set based on the cost function, select nodes based on the ascending sorted candidate set and form a target deployment set; Issue deployment instructions to each node in the target deployment set to execute the deployment operation: The deployment operation refers to the initialization of physical writes to data replicas. If the compression strategy is 1, the compressor is enabled to perform structural compression on the data replicas. The replicas are then stored in the corresponding storage media based on the storage popularity level. Then, according to the consistency level requirements, a data synchronization path and confirmation mechanism are established between the master and slave replicas. After the deployment is complete, a scheduling task execution log is generated, binding the task number and behavior label; Analyze the behavioral path response process after policy deployment, obtain and analyze the management scheduling information in the behavioral path compression and execution displacement trends, including the following steps: Obtaining management scheduling information generated during the behavior path compression and execution displacement trend analysis process, the management scheduling information including the behavior path compression index and the strategy guidance offset displacement index; The behavior path compression index indicates whether the actual access path of the data object tends to be stable and convergent after the policy deployment; The policy-guided offset index indicates the degree of topological offset between the target deployment area set by the policy structure and the actual replica scheduling and access behavior; The obtained behavior path compression index and strategy guidance offset displacement index are jointly generated into a data management coefficient; The behavior path compression index is positively correlated with the data management coefficient, while the strategy guidance offset displacement index is negatively correlated with the data management coefficient; Scheduling management is performed based on different signals generated by the analysis results, including the following steps: Compare the generated data management coefficient with the set data management threshold; If the data management coefficient is greater than or equal to the data management threshold, a stable execution signal is generated, and the current policy structure and corresponding scheduling path are maintained, and the original scheduling plan continues to be executed; If the data management coefficient is less than the data management threshold, an evolution adjustment signal is generated, the scheduling plan is stopped, the behavior re-identification mechanism is triggered, the behavior label is reallocated, and a new policy structure is mapped.

2. The data management method based on a distributed storage system according to claim 1, characterized in that: Collect and standardize data access logs in distributed storage systems and construct a chronological sequence of access behaviors. The specific steps include: Collect all access events of data objects in the data access log. Each access event record includes the access time, initiating node, operation type, and data volume. Standardize the collected data, including unifying timestamp accuracy to seconds in UTC format, normalizing operation types to a unified identifier, filtering out non-business access, and merging repeated access requests within a set time interval; The access logs of the standardized data objects are transformed into a time-ordered event sequence, and the preprocessed event sequence is converted into a structured access behavior sequence.

3. The data management method based on a distributed storage system according to claim 2, characterized in that: Extract the access mutation rate, access periodicity, and access span indicators to generate a fused behavior feature vector. The specific steps include: Each valid access event in the event sequence is regarded as a behavior node of the data object, and a continuous behavior trajectory is formed; Extract access rate fluctuation characteristics, count the access volume difference and time interval difference between adjacent accesses, and determine whether there is a rate mutation in the access behavior. If there is a rate mutation, obtain it and use it as the access mutation rate indicator; Extract periodic features, perform frequency domain analysis on access time series, observe activities over time periods and use them as indicators of access concentration; Extract access concentration and span. The time interval between the first and last visits is used as the behavior span. The ratio of the access concentration to the behavior span is used as the access concentration. The access concentration and span are summed up to form the access span indicator. Frequency domain analysis is used to detect the periodic trend of access behavior sequences. The amplitude of behavior fluctuation corresponding to each frequency is calculated. The period component with the largest amplitude of behavior fluctuation is taken as the dominant period. A continuous time series composed of the trend of the number of visits per unit time is used as the trajectory of access density changes. Analyze the contextual tag information of data object binding. The contextual tag information dimensions include data source module, tenant or project code, data function role label, whether the data is accessed collaboratively, and task priority. Context vector representation is performed based on the context embedding function, and the context label is projected to adjacent positions in the Euclidean space according to the semantic structure; After completing the access behavior modeling, a fused behavior feature vector is generated.

Images