Distributed storage shadow copy migration method and device

Abstract

The application provides a distributed storage shadow copy migration method and device, relates to the technical field of data migration and storage, and the method comprises the following steps: obtaining monitoring data of a distributed storage system, wherein the monitoring data is related to the access condition of data in the distributed storage system; generating an access gradient graph based on the monitoring data, wherein the access gradient graph is formed by a three-dimensional access gradient field used for representing the change of data access heat, and the three-dimensional access gradient field is related to the time gradient and the space gradient of data access; calling a pre-constructed migration model, processing the access gradient graph based on the migration model, obtaining a target shadow copy to be migrated, and the migration model is used for determining the shadow copy needing to be migrated according to the input access gradient graph; and generating a migration trigger signal to trigger the migration of the target shadow copy.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of data migration, and in particular to a distributed storage shadow copy migration method and device. Background Art

[0002] With the rapid development of technologies such as cloud computing, big data, and artificial intelligence, the global volume of data is growing exponentially. Distributed storage systems, with their exceptional scalability, high reliability, and cost-effectiveness, have become the core infrastructure supporting the storage and processing of massive amounts of data. In these large-scale systems, the physical location of data has a crucial impact on its access performance. However, business workloads are often dynamic, resulting in frequent shifts in data access hotspots between different nodes and storage tiers. To ensure consistent quality of service (QoS) for upper-layer applications, maintain load balancing across the entire system, and optimize storage costs (for example, migrating less frequently accessed cold data to lower-cost media), the dynamic and efficient migration of data or its shadow copies between storage nodes has become a key research topic with significant research significance and core commercial value in the field of distributed storage management. Therefore, developing a method that can autonomously perceive data access trends, make intelligent decisions, and efficiently execute migration tasks is crucial for improving the overall operational performance of modern data centers.

[0003] Currently, the industry has proposed and implemented a number of technical solutions to address data migration issues in distributed storage systems. However, existing solutions have inherent limitations when dealing with highly dynamic and complex access patterns, leading to delayed decision-making and inefficient execution. This in turn results in delayed migration of data replicas, impacting access speed to hotspot data. Summary of the Invention

[0004] In order to solve the above technical problems, an embodiment of the present invention provides a distributed storage shadow copy migration method, including:

[0005] Obtaining monitoring data of the distributed storage system, wherein the monitoring data is related to access status of data in the distributed storage system;

[0006] generating an access gradient map based on the monitoring data, wherein the access gradient map is formed by a three-dimensional access gradient field for characterizing changes in data access popularity, wherein the three-dimensional access gradient field involves a temporal gradient and a spatial gradient of data access;

[0007] Calling a pre-built migration model, processing the access gradient graph based on the migration model, and obtaining a target shadow copy to be migrated, wherein the migration model is used to determine the shadow copy to be migrated based on the input access gradient graph;

[0008] A migration trigger signal is generated to trigger migration of the target shadow copy.

[0009] In one embodiment, generating an access gradient map based on the monitoring data includes:

[0010] Aggregating the monitoring data in a time dimension to form a temporal access matrix, wherein the temporal access matrix includes access status of each data in a continuous time slice;

[0011] determining a time gradient component for characterizing a change in data access frequency based on the temporal access matrix;

[0012] Determining a spatial gradient component for characterizing a change trend in spatial distribution of data access heat based on the temporal access matrix and logically adjacent data blocks;

[0013] The temporal gradient component and the spatial gradient component are synthesized to form the three-dimensional access gradient field.

[0014] In one embodiment, synthesizing the temporal gradient component and the spatial gradient component to form the three-dimensional access gradient field includes:

[0015] The temporal gradient component and the spatial gradient component are synthesized based on the following formula to form the three-dimensional access gradient field:

[0016]

[0017] described is the gradient vector after the fusion of the temporal gradient component and the spatial gradient component, the sqrt((Ψf / Ψt) 2 + (Ψf / Ψs) 2 ) indicates the severity of the change in data access popularity. vector Indicates the direction of change in data access popularity. vector The gradient vector is obtained by normalizing the gradient vector formed by fusing the temporal gradient component and the spatial gradient component.

[0018] In one embodiment, constructing the migration model includes:

[0019] Obtaining historical monitoring data of the distributed storage system;

[0020] Generate a historical access gradient graph based on the historical monitoring data;

[0021] The historical access gradient graph is used to simulate the mechanism of biological neural synaptic signal transmission and is converted into a synaptic signal matrix. The synaptic signal matrix is ​​used to determine the aggregate signal strength of each data on a storage node. The aggregate signal strength is used to quantify the necessity of migrating the shadow copy of the data.

[0022] Determine a dynamic trigger threshold based on a historical migration success rate in the historical monitoring data and a real-time resource status of the distributed storage system, wherein the dynamic trigger threshold is used to measure the aggregate signal strength;

[0023] The migration model is constructed based on the synaptic signal matrix and the dynamic trigger threshold.

[0024] In one embodiment, the mechanism of simulating biological synaptic signal conduction using the historical access gradient map and converting it into a synaptic signal matrix includes:

[0025] Constructing a normalized gradient feature vector based on the historical access gradient map;

[0026] Obtaining migration records of shadow copies during a historical period, and determining an initial synaptic weight matrix based on the migration records of shadow copies during the historical period;

[0027] The normalized gradient eigenvector is weighted by using the initial synaptic weight matrix, and the elements between the initial synaptic weight matrix and the normalized gradient eigenvector are mapped by a nonlinear biological neuron activation function to generate historical synaptic signal strengths, wherein the historical synaptic signal strengths corresponding to different data are combined to form a historical sequence;

[0028] Processing the historical sequence using a time decay function to determine a time series cumulative signal strength;

[0029] The synaptic signal matrix is ​​formed by aggregating the time-series cumulative signal intensities corresponding to different data.

[0030] In one embodiment, each of the time-series accumulated signal strengths corresponds to data of a storage node;

[0031] The aggregating the time series cumulative signal strengths corresponding to different data to form the synaptic signal matrix includes:

[0032] Determine a network distance weight corresponding to each storage node based on the distances between different storage nodes;

[0033] Adjusting the sequential cumulative signal strength of adjacent storage nodes based on the network distance weight of each storage node;

[0034] Fusing the time series cumulative signal strength of each storage node with the adjusted time series cumulative signal strength of adjacent storage nodes to generate a target fusion signal;

[0035] The synaptic signal matrix is ​​formed based on the target fusion signals of each of the storage nodes.

[0036] In one embodiment, calling a pre-built migration model, processing the access gradient graph based on the migration model, and obtaining a target shadow copy to be migrated includes:

[0037] Updating the current dynamic trigger threshold in combination with the current resource status of the distributed storage system;

[0038] Processing the access gradient map based on the synaptic signal matrix to obtain a current aggregate signal strength corresponding to each of the storage nodes;

[0039] Comparing the current aggregate signal strength with the updated dynamic trigger threshold to determine the target storage node;

[0040] The target shadow copy to be migrated is determined based on the data in the target storage node.

[0041] In one embodiment, before migrating the shadow copy, the following steps are included:

[0042] Determine a local gradient extreme point in the access gradient graph, and determine an adaptive slice boundary set based on the local gradient extreme point;

[0043] For each data region divided in the access gradient map based on the adaptive shard boundary set, determining a shard size that is negatively correlated in value according to the gradient value of each data region, so as to generate a shard size mapping table according to the shard size;

[0044] The shadow copy data to be migrated is partitioned based on the adaptive shard boundary set and the shard size mapping table to obtain a plurality of data shards of different sizes, where the size of the data shard is inversely proportional to the data access heat.

[0045] In one embodiment, migrating the shadow copy includes:

[0046] Building a delta compression dictionary that matches the data pattern of the shadow copy;

[0047] Encoding each data slice of the shadow copy based on the incremental compression dictionary, and determining a compression level in the encoding process according to a gradient value in the access gradient map, to obtain encoded data of the shadow copy;

[0048] Encapsulating the encoded data to form a compressed data fragment sequence, and forming fragment metadata based on compression information during the encoding process;

[0049] The data fragment sequence and fragment metadata are transmitted and verified.

[0050] Another embodiment of the present invention also provides a distributed storage shadow copy migration device, including:

[0051] An acquisition module, configured to obtain monitoring data of a distributed storage system, wherein the monitoring data is related to access conditions of data in the distributed storage system;

[0052] a generating module for generating an access gradient map based on the monitoring data, wherein the access gradient map is formed by a three-dimensional access gradient field for characterizing changes in data access popularity, wherein the three-dimensional access gradient field involves a temporal gradient and a spatial gradient of data access;

[0053] A calling module is used to call a pre-built migration model, process the access gradient graph based on the migration model, and obtain a target shadow copy to be migrated, wherein the migration model is used to determine the shadow copy to be migrated based on the input access gradient graph;

[0054] The migration module is used to generate a migration trigger signal to trigger the migration of the target shadow copy.

[0055] Other features and advantages of the present application will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present application. The purposes and other advantages of the present application can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0056] The technical solution of the present application is further described in detail below through the accompanying drawings and examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0058] Figure 1 This is a flow chart of a distributed storage shadow copy migration method in an embodiment of the present invention.

[0059] Figure 2 This is a flow chart of a distributed storage shadow copy migration method in another embodiment of the present invention.

[0060] Figure 3 This is a flow chart of a distributed storage shadow copy migration method in another embodiment of the present invention.

[0061] Figure 4 This is a structural block diagram of a distributed storage shadow copy migration device in an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings, but are not intended to limit the present invention.

[0063] It should be understood that various modifications may be made to the embodiments disclosed herein. Therefore, the following description should not be considered as limiting, but merely as an example of an embodiment. Other modifications within the scope of the present disclosure will occur to those skilled in the art.

[0064] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the general description of the present disclosure given above and the detailed description of the embodiments given below, serve to explain the principles of the present disclosure.

[0065] These and other characteristics of the invention will become apparent from the following description of a preferred form of embodiment given as a non-limiting example with reference to the accompanying drawings.

[0066] It should also be understood that although the invention has been described with reference to certain specific examples, those skilled in the art will be able to realize many other equivalent forms of the invention that have the characteristics recited in the claims and are therefore within the scope of protection defined thereby.

[0067] The above and other aspects, features and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings.

[0068] Specific embodiments of the present disclosure will be described hereinafter with reference to the accompanying drawings; however, it should be understood that the disclosed embodiments are merely examples of the present disclosure, which may be implemented in a variety of ways. Well-known and / or repetitive functions and structures are not described in detail to avoid obscuring the present disclosure with unnecessary or redundant detail. Therefore, the specific structural and functional details disclosed herein are not intended to be limiting, but rather serve merely as a basis and representative basis for the claims to teach those skilled in the art to variously employ the present disclosure with substantially any suitable detailed structure.

[0069] This description may use the phrases "in one embodiment," "in another embodiment," "in a further embodiment," or "in other embodiments," each of which may refer to one or more of the same or different embodiments according to the present disclosure.

[0070] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0071] like Figure 1 As shown, an embodiment of the present invention provides a distributed storage shadow copy migration method, including:

[0072] S1: Obtain monitoring data of a distributed storage system, where the monitoring data is related to access to data in the distributed storage system;

[0073] S2: generating an access gradient map based on the monitoring data, wherein the access gradient map is formed by a three-dimensional access gradient field for characterizing changes in data access popularity, and the three-dimensional access gradient field involves a temporal gradient and a spatial gradient of data access;

[0074] S3: calling a pre-built migration model, processing the access gradient graph based on the migration model, and obtaining a target shadow copy to be migrated, wherein the migration model is used to determine the shadow copy to be migrated based on the input access gradient graph;

[0075] S4: Generate a migration trigger signal to trigger the migration of the target shadow copy.

[0076] In a typical application scenario, such as a large-scale cloud storage system consisting of hundreds of storage nodes, to ensure high data availability and load balancing, the system creates shadow copies for hotspot data. When data access hotspots shift, these shadow copies need to be migrated to new, more suitable storage nodes. In this embodiment, during operation, the distributed storage system continuously generates various types of monitoring data, which may include, but is not limited to, access logs for each node, I / O statistics (such as IOPS and throughput), and request response time records. By collecting this raw monitoring data and aggregating and calculating it, changing trends in data access popularity can be identified. This trend is quantified and used to construct an access gradient graph. Based on the severity of changes in the gradient graph and the current state of the system, the algorithm of this embodiment determines whether to trigger shadow copy migration and which specific shadow copies to migrate. When a shadow copy is determined to be migrated, a migration trigger signal is generated. In response to the migration trigger signal, the algorithm processes the shadow copy data to be migrated based on the access gradient graph, obtaining a sequence of compressed data shards and corresponding shard metadata, and based on this, implements shadow copy migration and verification. The access gradient graph is a data structure that maps each data block in the storage system to its corresponding gradient vector. This gradient vector quantifies the rate and direction of change in the access popularity of the data block in both time and space, and serves as the direct basis for all subsequent intelligent decisions, such as sharding, compression, and scheduling.

[0077] In some optional implementations, the migration of shadow copies in this example is mainly described as asynchronous migration to minimize the impact on online services. However, for certain businesses with extremely high consistency requirements or that allow short maintenance windows, this embodiment can also be configured as a synchronous or semi-synchronous mode. In synchronous mode, source data access will be suspended until the data is fully migrated and verified successfully to ensure zero data difference. In addition, the access gradient-based migration idea of ​​this embodiment is not only applicable to shadow copy migration in distributed storage systems, but its core principles can also be extended to other fields, such as dynamic migration of virtual machines (VMs) based on CPU and memory usage gradients, or dynamic scheduling of virtual network functions (VNFs) based on network traffic gradients.

[0078] Therefore, the solution of this embodiment has a wide range of applications. Through the above solution of this embodiment, the system can intelligently predict data hotspot trends, adaptively process and migrate data, and improve the efficiency, accuracy and automation level of data migration.

[0079] like Figure 2 As shown, generating an access gradient graph based on the monitoring data includes:

[0080] S201: Aggregating the monitoring data in a time dimension to form a time sequence access matrix, where the time sequence access matrix includes access status of each data in a continuous time slice;

[0081] S202: Determine a time gradient component for characterizing a change in data access frequency based on the temporal access matrix;

[0082] S203: Determine a spatial gradient component for characterizing a change trend of data access popularity in spatial distribution based on the temporal access matrix and logically adjacent data blocks;

[0083] S204: Synthesize the temporal gradient component and the spatial gradient component to form the three-dimensional access gradient field.

[0084] For example, the temporal access matrix in this embodiment refers to a two-dimensional data structure, where the rows of the matrix represent logical data blocks in the storage system, and the columns represent consecutive time slices (time windows). Each cell in the matrix stores the access feature vector of the corresponding data block within the corresponding time slice. This vector may include multiple dimensions such as access frequency, read-write operation ratio, average I / O size, and average access latency. The temporal access matrix is ​​a core data structure. For example, each row of the matrix corresponds to a logical data block (e.g., a 64MB data block), and each column corresponds to a time slice (e.g., 5 minutes). Each cell (i, j) in the matrix is ​​a vector containing multiple access features of data block i within time slice j. These features may include: access frequency (e.g., total number of reads and writes within the time slice), read-write ratio, average response latency, etc. To generate this matrix, the system uses a sliding time window aggregation method to continuously process real-time monitoring data streams from the entire distributed storage system (including access logs and I / O statistics from each node) and dynamically update the matrix. Optionally, when aggregating monitoring data along the time dimension to form a sequential access matrix, a mapping conversion from node-level monitoring metrics to data block-level access characteristics is required. Specifically, each storage node maintains an access log table with a record format of [timestamp, data block ID, operation type, I / O size, response time]. The system decomposes the node's I / O operations and attributes them to specific logical data blocks using the data block ID. For example, node N1's total IOPS in time slice T1 is 1000. Access log analysis shows that data block DB001 contributed 150 I / Os, data block DB002 contributed 200 I / Os, and so on. Accordingly, in the sequential access matrix, the access frequency component of element M[DB001][T1] is set to 150, and the access frequency component of element M[DB002][T1] is set to 200. This achieves accurate mapping from node-level aggregate metrics to detailed data block-level metrics.

[0085] To capture the temporal evolution of data heat, that is, whether a data block is becoming hotter or cooler, a time gradient is calculated. Specifically, for any data block in the temporal access matrix, its time gradient component Ψf / Ψt is calculated using a time difference algorithm: Ψf / Ψt = (f(t) - f(t-Δt)) / Δt. Here, Ψ is the partial derivative, f(t) is the access frequency of the data block in the current time slice t, f(t-Δt) is its access frequency in the previous time slice t-Δt, and Δt is the length of the time slice (i.e., the sampling interval). A large positive value for the time gradient component indicates that the data is rapidly becoming hotter, while a large negative value indicates that it is rapidly cooling.

[0086] When calculating spatial gradient components, logically adjacent data blocks refer to data blocks that are continuous at the application or file system level, not necessarily physically contiguous on the storage medium. This can be determined by logical block address (LBA) continuity or data offset continuity within the same file. To identify the spatial clustering of data popularity, namely the boundaries of hotspots, the spatial gradient is calculated. For any data block in the temporal access matrix, its spatial gradient component Ψf / Ψs is calculated using a spatial difference algorithm: Ψf / Ψs = (f(s+Δs) - f(s)) / Δs. Here, f(s) is the access frequency of the current data block in a given time slice s, and f(s+Δs) is the access frequency of its logically adjacent data blocks. Logically adjacent data blocks here can refer to consecutive data blocks within the same file or adjacent data blocks in the address space. Δs is the spatial step size, typically set to 1. A larger spatial gradient value indicates that the corresponding data is closer to the boundary between hotspots and non-hotspots.

[0087] Furthermore, the synthesizing the temporal gradient component and the spatial gradient component to form the three-dimensional access gradient field includes:

[0088] S205: Synthesize the temporal gradient component and the spatial gradient component based on the following formula to form the three-dimensional access gradient field:

[0089]

[0090] described is the gradient vector after the fusion of the temporal gradient component and the spatial gradient component, the sqrt((Ψf / Ψt) 2 + (Ψf / Ψs) 2 ) indicates the severity of the change in data access popularity. vector Indicates the direction of change in data access popularity. vector The gradient vector is obtained by normalizing the gradient vector formed by fusing the temporal gradient component and the spatial gradient component.

[0091] Specifically, in this embodiment, after obtaining the gradient components of the time and space dimensions, they are fused into a gradient vector ▽f through a vector synthesis algorithm. The gradient vectors of all data blocks together constitute the three-dimensional access gradient field. The synthesis method used in this embodiment is shown in the formula described above. The vector calculated by this formula has a scalar size of sqrt((Ψf / Ψt) 2 + (Ψf / Ψs) 2 ) indicates the overall intensity of the change in access popularity, while direction vector It represents the direction of change of data access heat (for example, the direction on the two-dimensional plane of space and time).

[0092] This embodiment collects time-series monitoring data, analyzes temporal and spatial gradients, and ultimately synthesizes them into a three-dimensional access gradient field, achieving a fundamental shift from state perception to trend prediction. In distributed storage migration scenarios, this embodiment's solution no longer focuses solely on the instantaneous value of data block access frequency. Instead, it deeply reveals the rate at which this popularity changes over time (i.e., whether the data is becoming hot or cold) and its spatial spread (i.e., whether the hotspot is spreading to neighboring data blocks). This solves the decision-making lag caused by reliance on static thresholds. The system can preemptively initiate migration for preheated data blocks whose access frequency has not yet reached the threshold but whose temporal gradient is extremely high, achieving preemptive layout optimization. Furthermore, for cooling data blocks with high access frequency but a negative temporal gradient, unnecessary migrations can be avoided, effectively suppressing migration jitter caused by fluctuations in popularity near the threshold. This improves the accuracy and foresight of migration decisions, enhances the internal performance of the entire storage system, and improves its responsiveness to changes in business load. From a user's perspective, this translates to lower data access latency and a more stable service experience.

[0093] In some optional implementations, the heat index f used to calculate the gradient can be a composite index in addition to the access frequency. For example, f = wfreq × access frequency + wsize × average I / O size + wlat × (1 / average latency), where wfreq, wsize, and wlat are weights of different dimensions. Composite heat indexes can more comprehensively reflect the access value of data. Moreover, the gradient field can be extended to higher dimensions. For example, a cost gradient can be introduced to characterize the cost-effectiveness trend of migrating data blocks between different storage media (such as SSD, HDD, and archival storage), thereby realizing cost-aware intelligent migration decisions.

[0094] When calculating the temporal gradient component, the algorithm of this embodiment provides an adaptive adjustment mechanism for the specific window selection strategy of the temporal difference algorithm. For example, the system can preset a baseline time window, such as 5 minutes. Simultaneously, the system calculates the variance of data block access frequencies over a short period of time. When the variance is less than a preset stability threshold, it indicates that the access pattern is stable, and the baseline window is used to obtain a smooth gradient value. When the variance exceeds this threshold, it indicates that the access pattern has undergone a drastic change, and the system automatically shortens the calculation window to a more sensitive size, such as 1 minute, to more accurately capture the sudden change trend. When calculating the spatial gradient component, the algorithm provides at least two operational definitions for the specific definition of spatial adjacency: Logical Block Address (LBA)-based adjacency: In the underlying storage system, each data block has a unique LBA. If the LBAs of two data blocks are consecutive, or their difference is less than a preset proximity threshold (e.g., a distance of 4 blocks), they are defined as spatially adjacent. Adjacency based on file system metadata: By parsing file system metadata, if two data blocks belong to the same file and their logical offsets within the file are consecutive, they are defined as spatially adjacent. This better reflects the relevance of data at the application level.

[0095] Furthermore, the scalar size of the gradient vector refers to the Euclidean length of the gradient vector in the time-space two-dimensional plane. It is a non-negative scalar value that purely represents the intensity or speed of the change in access popularity without containing directional information. This embodiment is used to calculate the modulus of the gradient vector, that is, the overall intensity of the change in access popularity. The specific calculation formula is: |▽f| = sqrt((Ψf / Ψt) 2 + (Ψf / Ψs) 2 ), where Ψf / Ψt and Ψf / Ψs are the temporal and spatial gradient components, respectively, and |▽f| is the scalar size of the gradient vector.

[0096] Determine the direction vector direction representing the gradient direction vector Used to clarify the direction of heat change. The direction vector is defined as a unit vector on the time-space two-dimensional plane. The specific calculation method is to normalize the original vector composed of the time and space gradient components: direction vector = (Ψf / Ψt, Ψf / Ψs) / |▽f|= ( (Ψf / Ψt) / |▽f|, (Ψf / Ψs) / |▽f| ); the components of this unit vector clearly indicate the relative proportions and directions of changes in the time and space dimensions.

[0097] Since the gradient vector ▽f for each data block is defined by its scalar magnitude and direction vector, in practical applications, it can be directly represented by the original vector ▽f = (Ψf / Ψt, Ψf / Ψs), which consists of two components and fully preserves all gradient information. Through the above steps, a precise, quantifiable gradient vector containing directional information is constructed for each data block, providing a solid data foundation for all subsequent gradient-based intelligent decision-making.

[0098] In some optional implementations, if the monitoring dimension of the system is further increased, for example, a business value dimension that measures the importance of data is introduced, then the access gradient field can be expanded to a higher-dimensional field (such as three-dimensional). In this case, the direction vector direction vector It will also be a three-dimensional unit vector, and its calculation method is correspondingly expanded to vector normalization in three-dimensional space, so as to guide the system to make comprehensive migration decisions in the three dimensions of time, space and business value.

[0099] For example, in one application example, the process of generating an access gradient map includes:

[0100] Monitoring data includes three core metrics: access frequency (number of I / O operations per second), throughput (bytes read and written per second), and response latency (average response time in milliseconds). The system collects real-time monitoring data from each node every five seconds, generating a raw data stream.

[0101] Perform aggregation calculations on multi-dimensional monitoring indicators to generate comprehensive access popularity values ​​for data blocks.

[0102] Specifically, for the access popularity H(DBx, t) of data block DBx in time slice t, the weighted aggregation formula is used: H(DBx, t) = w1×F(DBx, t) + w2×T(DBx, t) + w3×(1 / L(DBx, t)). Among them, F(DBx, t) is the access frequency (times / second) of data block DBx in time slice t, and the value range after normalization is [0, 1]; T(DBx, t) is the throughput (MB / second), which is also normalized to [0, 1]; L(DBx, t) is the average response delay (milliseconds), the inverse of which is taken and normalized to reflect the relationship that better performance means higher popularity; w1, w2, and w3 are weight coefficients, satisfying w1+w2+w3=1. In this embodiment, w1=0.5, w2=0.3, and w3=0.2 are set to reflect the focus on access frequency. Among them, the normalization can adopt the minimum-maximum normalization method: X norm = (X - X min ) / (X max -X min ). Where X is the original indicator value, Xmin and X max They are the minimum and maximum values ​​of the indicator in the statistical window, X norm is the normalized value.

[0103] After obtaining the (3D) access gradient field, it can be input into the pre-built migration model to determine whether there are shadow copies that need to be migrated. The migration model in this embodiment needs to be built in advance, and building the migration model includes:

[0104] S5: Obtain historical monitoring data of the distributed storage system;

[0105] S6: generating a historical access gradient graph based on the historical monitoring data;

[0106] S7: Using the historical access gradient graph to simulate the mechanism of biological neural synaptic signal conduction, converting it into a synaptic signal matrix, wherein the synaptic signal matrix is ​​used to determine the aggregate signal strength of each data on a storage node, and the aggregate signal strength is used to quantify the necessity of migrating the shadow copy of the data;

[0107] S8: Determine a dynamic trigger threshold based on a historical migration success rate in the historical monitoring data and a real-time resource status of the distributed storage system, where the dynamic trigger threshold is used to measure the aggregate signal strength;

[0108] S9: Constructing the migration model based on the synaptic signal matrix and the dynamic trigger threshold.

[0109] The synaptic signal matrix of this embodiment is a quantified matrix of migration impulse or urgency in a fully distributed storage system. Each element of the matrix corresponds to the aggregate signal strength of a data block on a node. This strength is the direct input for subsequent comparison with dynamic thresholds to make the final migration decision.

[0110] Specifically, the mechanism of simulating biological synaptic signal conduction by using the historical access gradient map and converting it into a synaptic signal matrix includes:

[0111] S701: Constructing a normalized gradient feature vector based on the historical access gradient map;

[0112] S702: Obtain migration records of shadow copies during a historical period, and determine an initial synaptic weight matrix based on the migration records of shadow copies during the historical period;

[0113] S703: Weighting the normalized gradient eigenvector using the initial synaptic weight matrix, and mapping elements between the initial synaptic weight matrix and the normalized gradient eigenvector using a nonlinear biological neuron activation function to generate historical synaptic signal strengths, where the historical synaptic signal strengths corresponding to different data are combined to form a historical sequence;

[0114] S704: Processing the historical sequence using a time decay function to determine a time series cumulative signal strength;

[0115] S705: Aggregate the time-series cumulative signal intensities corresponding to different data to form the synaptic signal matrix.

[0116] In this embodiment, each of the time-series accumulated signal strengths corresponds to data of a storage node;

[0117] The aggregating the time-series cumulative signal strengths corresponding to different data to form the synaptic signal matrix includes:

[0118] S706: Determine a network distance weight corresponding to each storage node based on the distances between different storage nodes;

[0119] S707: Adjusting the sequential cumulative signal strength of the adjacent storage nodes based on the network distance weight of each storage node;

[0120] S708: Fusing the time-series cumulative signal strength of each storage node with the adjusted time-series cumulative signal strength of the adjacent storage node to generate a target fusion signal;

[0121] S709: Forming the synaptic signal matrix based on the target fusion signal of each storage node.

[0122] For example, in order to eliminate the dimensionality effect of gradient values ​​in different dimensions, it is necessary to normalize the gradient vector to form a standardized gradient feature vector. A preferred normalization formula is: gradient = (gradient -min) / (max - min), where gradient is the original gradient component value, min and max are the minimum and maximum values ​​of the component in the statistical period, and normalized gradient is the normalized gradient feature vector. At the same time, in order to give the model initial prior knowledge, the initial synaptic weight matrix is ​​generated based on the historical migration record. For example, w init = success rate ×sensitivity factor The initial weights are generated in the following way, where success rateis the historical migration success rate involving this type of data, sensitivity factor It can be set according to the importance of the data or the business level, init is the initial synaptic weight matrix.

[0123] The biological neuron activation function refers to a mathematical function used to introduce nonlinear mapping, such as Sigmoid or ReLU. Its function is to simulate the characteristics of real neurons that are either not activated or fully activated, so that the system's response to gradient changes is not smooth and linear, but rather shows stronger sensitivity to signal changes exceeding a certain intrinsic threshold. This embodiment simulates the process of biological neurons receiving signals and being activated. Specifically, the original synaptic signal strength S is calculated by the following formula: S = fact (Σ(w i ×g i )); where g i is the i-th component of the normalized gradient eigenvector, w i is the corresponding synaptic weight, Σ(w i ×g i ) is a weighted sum. fact(x) is a nonlinear activation function, such as the improved Sigmoid function f(x) = 1 / (1+e (-kx) ), k is the steepness parameter. Nonlinear mapping makes the signal intensity more sensitive to gradient changes and avoids a simple linear relationship.

[0124] Taking into account the persistence effect of the signal, it is necessary to apply a time decay function to the historical sequence of the original synaptic signal strength to calculate the time series cumulative signal strength that characterizes the persistence effect of the signal. This embodiment simulates the memory effect of neuronal signals. For each original synaptic signal strength in the historical sequence, multiply it by a weight factor that has an exponential decay relationship from its generation time to the current time, and then sum all the weighted signal strengths to obtain the time series cumulative signal strength. The calculation formula is: S cumulative =Σ(S t ×d (tcurrent - t) ). Where S t is the original synaptic signal strength generated at the historical time point t, d is a decay factor between 0 and 1 (e.g. 0.9), t current is the current time. A persistent signal (even if weak) will have a stronger cumulative effect than a short-lived strong signal, which is more consistent with the inertial characteristics of access popularity in the real world.

[0125] After obtaining the strength of each synaptic signal, it can be aggregated to obtain a synaptic signal matrix. This embodiment uses the synaptic signal matrix to make the decision no longer an isolated behavior of a single node, but a collaborative behavior within a distributed system. Specifically, based on the network distance between storage nodes, the network distance weight is calculated for each storage node pair. For each storage node, its own time-series cumulative signal strength is fused with the time-series cumulative signal strength of the adjacent nodes adjusted by its corresponding network distance weight to generate the final aggregate signal. The calculation formula is: S aggregated = S local + Σ(S neighbor ×w dist ). Where S local is the time series cumulative signal strength of the node itself, S neighbor is the signal strength of its neighboring nodes, w dist It is the weight that reflects the network topological distance. The closer the distance, the higher the weight.

[0126] Specifically, in this embodiment, when forming the synaptic signal matrix, the synaptic weight matrix also needs to be generated at the same time. The synaptic weight wi represents the degree of influence of the i-th gradient feature on the migration decision, and the value range is [0, 1]. The setting of the initial weight is based on the statistical analysis of historical migration data, and the following method is used: extract the records of the last 1000 migration tasks from the historical migration log, and each record contains: the gradient feature vector at the time of triggering, the migration result (success / failure), the migration time, the resource consumption and other information. Secondly, for each gradient feature dimension i, its correlation with the success of the migration is statistically analyzed. Specifically calculate the feature importance score: I i = (μ success，i - μ fail，i ) / σ i , where μ success , i is the average value of the i-th feature in the successful migration case, μ fail，i is the average of the failure cases, σ i is the standard deviation of the feature. Normalize the importance score to the initial weight: wi(0)= (I i - I min ) / (I max - I min ) × 0.7 + 0.15; where I min is the minimum feature importance score, I max is the maximum feature importance score. The base value of 0.15 is added here to ensure that all features have a certain influence in the initial stage. The configuration of the scaling factor of 0.7 can avoid the initial weight being too extreme. For example, for the time gradient feature, statistics show that the average time gradient of successful migration is 0.65, the average time gradient of failed migration is 0.25, and the standard deviation is 0.3, then Itime = (0.65-0.25) / 0.3 = 1.33. Assuming that the I value range of all features is [-0.5, 2.0], then w time (0) = (1.33-(-0.5)) / (2.0-(-0.5)) × 0.7 + 0.15 = 0.66.

[0127] Synaptic weights are not directly determined parameters and require continuous training to ensure accuracy. The objective function of synaptic weight training is defined as follows:

[0128] In this embodiment, the training goal is to maximize the comprehensive utility function U of the migration task, which comprehensively considers the migration success rate, execution efficiency, and resource consumption: U = α × P success + β × E efficiency + γ × (1 -C resource ); where P success is the probability of successful migration, estimated by the logistic regression model: P success = 1 / (1 + exp(-Σ(w i × g i )));E efficiency is the execution efficiency score, defined as E efficiency = V baseline / V actual , V baseline is the baseline migration speed (e.g. 100MB / s), V actual is the actual migration speed; C resource is the normalized resource consumption, including the weighted average utilization of CPU, network, and I / O; α, β, and γ are target weights, satisfying α + β + γ = 1. In this embodiment, α is set to 0.5, β is set to 0.3, and γ is set to 0.2.

[0129] The normalized gradient feature vector is weighted using the initial synaptic weight matrix and mapped through a nonlinear biological neuron activation function to generate the original synaptic signal strength and form a historical sequence. The specific calculation process uses the formula S = f act (Σ(w i ×g i )), where the activation function f act (x) = 1 / (1+e (-2x) ).

[0130] Furthermore, in this embodiment, after each migration task is completed, the system will update the weight according to the actual execution effect. The updating process includes: calculating the actual utility value U of this migration actual For example, a migration is successfully completed (P success=1), actual speed 120MB / s (E efficiency =100 / 120=0.83), resource consumption 45% (C resource =0.45), then U actual = 0.5×1 + 0.3×0.83 + 0.2×(1-0.45) = 0.859. Calculate the reinforcement learning reward value R reward =(U actual -U expected ) / U expected , where U expected is the expected utility based on the current weight prediction. expected =0.75, then R reward = (0.859-0.75) / 0.75 = 0.145. Apply gradient ascent to update weights: w i (t+1) = w i (t) +η rl × R reward × g i × ΨP success / Ψw i Among them, η rl Is the learning rate, using an adaptive strategy: η rl (t) =η initial / (1 + λ×t), η initial = 0.1, λ = 0.001, t is the training round; ΨP success / Ψw i = g i × P success ×(1 - P success ) is the partial derivative of the success probability with respect to the weight. To ensure that the weight remains within the valid range, it is clipped after the update: w i (t+1) = max(0.05, min(0.95, w i (t+1))).

[0131] The time decay mechanism ensures that recent signals have a greater impact on decision-making. Specifically, the strength of each original synaptic signal in the historical sequence is multiplied by a weight factor that has an exponential decay relationship from the time of its generation to the current time to obtain the weighted signal strength; all weighted signal strengths are summed to obtain the time series cumulative signal strength. The value of the attenuation factor d is determined by experiment. In this embodiment, d=0.85, which means that the signal strength of each time slice is attenuated by 15%. For example, assuming that the original signal strengths of the last three time slices are S(t-2)=0.6, S(t-1)=0.75, and S(t)=0.9, the time series cumulative signal strength S cumulative= 0.6×0.852 + 0.75×0.851 + 0.9×0.850 = 0.434 + 0.638+ 0.9 = 1.972.

[0132] This embodiment continuously optimizes synaptic weights based on actual migration results, allowing the system's decision-making capabilities to continuously improve with experience. Compared to traditional fixed-parameter approaches, this approach automatically adapts to different business scenarios and load patterns, achieving truly intelligent migration decision-making.

[0133] In some optional implementations, the weight update can also introduce a momentum term to accelerate convergence: i (t+1) = w i (t) + η rl × ΔW i (t) + momentum × ΔW i (t-1), where the momentum coefficient is usually set to 0.9, ΔW i (t) is the current weight update amount. Alternatively, a batch update strategy can be used to accumulate gradient information from multiple migrations and then update the weights uniformly to improve training stability.

[0134] This embodiment constructs an intelligent, self-learning migration trigger decision engine by converting gradient features into synaptic signals that simulate biological neurons and comparing them with dynamic thresholds that are adaptively adjusted based on historical success rates and real-time system resource status. By introducing nonlinear activation functions, time series accumulation effects, and multi-node signal aggregation, this engine addresses the rigidity of decisions caused by one-size-fits-all static thresholds. Specifically, in complex environments like data centers, fixed trigger thresholds are completely incapable of adapting to changing service loads and network conditions. Dynamic thresholds can self-adjust based on the actual performance (success rate) of historical migrations and the current availability of system resources. When resources are abundant and the historical success rate is high, the threshold is lowered to encourage migration; when resources are scarce or historical performance is poor, the threshold is automatically raised to adopt a conservative strategy. This ensures that each migration decision is the optimal one, taking into account data trends, historical experience, and the current global system status. This improves the intelligence and robustness of migration decisions in complex environments, ensuring that migration operations always occur at the most appropriate time and in the most appropriate manner.

[0135] The dynamic trigger threshold changes in real time. When determining the current dynamic trigger threshold, in this embodiment, the current dynamic trigger threshold is multiplied by an adjustment item representing the deviation between the historical migration success rate and the preset target success rate, and a factor reflecting the current system resource sufficiency, thereby iteratively generating the dynamic trigger threshold for the next cycle. A preferred iterative formula is: T(n+1) = T(n) × (1 + γ × (R target -R actual )) × F resource Where T(n+1) is the dynamic trigger threshold of the next cycle, T(n) is the current threshold, γ is the learning rate, R target and R actual are the target and actual success rates, F resource is a resource abundance factor (for example, when the CPU or network bandwidth is idle, this factor is greater than 1, encouraging migration; when resources are tight, this factor is less than 1, inhibiting migration). Specifically, the resource abundance factor is a single scalar value that integrates and normalizes multiple independent, different-dimensional system resource indicators (such as CPU load and network utilization). This value ranges from [0, 1] and intuitively reflects the overall resource availability of the current system, and is used to inhibit or allow migration decisions. When the aggregate signal strength S of a data block is aggregated When the dynamic trigger threshold T(n+1) is exceeded and system resources allow, a migration trigger signal for the data block will be generated. In this embodiment, biological synapses are the connection structure that transmits signals between neurons. When the electrical signal of the current neuron reaches the threshold, neurotransmitters are released to cause the next neuron to respond. This biological mechanism is simulated by computerization: the gradient feature vector g i Mapped to the input signal of the neuron, the synaptic connection strength coefficient w i (value range [0, 1]) corresponds to the signal transmission efficiency of biological synapses, where w i The larger the value, the stronger the influence of the gradient feature on the migration decision; the activation function f act (x)=1 / (1+e (-kx) ) simulates the threshold response characteristics of neurons, which only produce significant output when the weighted input signal exceeds the intrinsic threshold; the time decay function d (tcurrent-t) (where d∈(0,1)) simulates the natural degradation process of neurotransmitters over time, so that the older the signal, the less influence it has on the current decision. This constructs an intelligent migration trigger decision system that can autonomously learn and adapt to the environment. That is, the dynamic trigger threshold and the above-constructed synaptic signal matrix containing synaptic weights cooperate to form the migration model of this instance.

[0136] like Figure 3As shown, after the migration model is built, the system (i.e., the algorithm) can call the pre-built migration model and process the access gradient graph based on the migration model to obtain the target shadow copy to be migrated, including:

[0137] S301: updating the current dynamic trigger threshold based on the current resource status of the distributed storage system;

[0138] S302: Processing the access gradient map based on the synaptic signal matrix to obtain a current aggregate signal strength corresponding to each of the storage nodes;

[0139] S303: Compare the current aggregate signal strength with the updated dynamic trigger threshold to determine a target storage node;

[0140] S304: Determine the target shadow copy to be migrated based on the data in the target storage node.

[0141] For example, the system first determines the dynamic trigger threshold adapted to the current scenario using the aforementioned method. It then processes the access gradient graph based on the synaptic signal matrix to obtain the current aggregate signal strength for each storage node. This aggregate signal strength is then compared with the latest dynamic trigger threshold. If it is within the threshold range, the corresponding data block's access popularity is normal and migration is not necessary. If it is greater than the threshold range, the corresponding data block's access popularity is showing a significant upward trend, so migration is necessary. Based on this, the system can determine the target storage node and, within the target storage node, identify the data blocks that require shadow copy migration.

[0142] After the shadow copy to be migrated is determined, the system does not directly migrate the shadow copy. Instead, it performs the following operations:

[0143] S10: Determine a local gradient extreme point in the access gradient graph, and determine an adaptive slice boundary set based on the local gradient extreme point;

[0144] S11: for each data region divided in the access gradient map based on the adaptive shard boundary set, determining a shard size that is negatively correlated in value according to the gradient value of each data region, and generating a shard size mapping table according to the shard size;

[0145] S12: Partitioning the shadow copy data to be migrated based on the adaptive shard boundary set and the shard size mapping table to obtain a plurality of data shards of different sizes, wherein the size of the data shard is inversely proportional to the data access popularity.

[0146] Specifically, once a migration trigger is received, the shadow copy data to be migrated and the initial access gradient map are read. Unlike traditional fixed-size chunking, this approach is gradient-aware, enabling more efficient and intelligent resource allocation. Specifically, regions with high gradient values ​​(i.e., hot data with rapidly fluctuating access popularity) are split into finer-grained shards, while regions with low gradient values ​​(cold data) are split into coarser-grained shards. These shards of varying sizes enter the incremental compression encoding pipeline, generating a highly compressed sequence of compressed data shards and shard metadata containing metadata such as original size, checksum, and gradient values. A shadow copy is a functionally independent copy of a data block. It is typically synchronized with the primary copy in the background for data migration, backup, or snapshot creation. After the migration is complete, a seamless switch to the primary copy is achieved without disrupting front-end services.

[0147] When migrating the shadow copy, include:

[0148] S13: Build an incremental compression dictionary that matches the data pattern of the shadow copy;

[0149] S14: Encoding each data slice of the shadow copy based on the incremental compression dictionary, and determining a compression level in the encoding process according to a gradient value in the access gradient map, to obtain encoded data of the shadow copy;

[0150] S15: Encapsulating the encoded data to form a compressed data fragment sequence, and forming fragment metadata according to the compression information in the encoding process;

[0151] S16: Transmit and verify the data fragment sequence and fragment metadata.

[0152] For example, after obtaining compressed data fragments, an asynchronous transmission pipeline is initiated to send these fragments from the source node to the destination node. This transmission process is parallel and multi-channel. To ensure transmission reliability, a bidirectional verification mechanism is implemented, comparing the data checksums of the source and destination nodes in real time during the transmission process. Even if an interruption occurs due to network instability, the breakpoint can be quickly located and incremental transmission can be resumed, avoiding the repeated transmission of large amounts of data. Information such as the progress of the entire process, successful and failed blocks, and performance statistics are recorded in the transmission status log.

[0153] After all data blocks have been transferred, the destination node performs integrity verification on the received data, including decompression and hash verification, to ensure that the data is not corrupted during transmission and compression / decompression. Once verification is successful, the system generates a migration completion status and seamlessly switches services to the new shadow replica. Furthermore, the performance data of the migration (recorded in the transfer status log) and the success or failure results are fed into the migration model as feedback. The model uses this feedback to update and optimize the parameters related to gradient calculation and migration triggering, making the next migration decision more accurate and efficient, thereby forming a continuously self-optimizing closed-loop system. The migration model refers to the set of core parameters and algorithms that control migration decisions, including but not limited to the synaptic weights, dynamic trigger thresholds, and reinforcement learning parameters mentioned in the examples. This model can continuously learn and evolve through a feedback mechanism, and is therefore also called an optimization model.

[0154] In order to better illustrate the solutions of the above embodiments, the following is explained in conjunction with specific application examples:

[0155] For example, the monitoring data of a data block DB001 in time slice t1 is: access frequency 500 times / second, throughput 20MB / second, and average latency 5 milliseconds. Assume that within the statistical window, the access frequency range is [0, 1000], the throughput range is [0, 100], and the latency range is [1, 20]. After normalization, F norm = 500 / 1000 = 0.5, T norm = 20 / 100 = 0.2, L norm =(1 / 5-1 / 20) / (1 / 1-1 / 20) ≈ 0.19. Overall heat H(DB001, t1) = 0.5×0.5 + 0.3×0.2 + 0.2×0.19 = 0.348.

[0156] 1. Based on the heat value sequence of consecutive time slices, construct the initial access gradient graph.

[0157] In this embodiment, the system maintains a sliding time window that contains the most recent 12 time slices (i.e., 1 minute of historical data). For each data block, the gradient value is calculated based on its heat time series H(t-11), H(t-10), ..., H(t). The time gradient uses the linear regression slope: ΨH / Ψt = (12×Σ(i×H i ) - Σi×ΣH i ) / (12×Σi 2 -(Σi) 2), where i is the time slice index. The spatial gradient is calculated by comparing the heat difference between logically adjacent data blocks: ΨH / Ψs = |H(DBx) - H(DBx+1)|. The gradient vector ▽H = (ΨH / Ψt, ΨH / Ψs) for each data block constitutes the initial access gradient map. This gradient map is stored in a hash table, with the key being the data block ID and the value being the gradient vector and its modulus |▽H|.

[0158] This embodiment accurately quantifies the access popularity trends of each data block, providing a precise data foundation for subsequent intelligent migration decisions. Compared to traditional instantaneous indicator judgments, gradient graphs can capture the dynamic changes in popularity, avoiding misjudgments caused by short-term fluctuations.

[0159] 2. In an optional embodiment, the process of initializing the intelligent decision-making mechanism and the dynamic threshold is specifically as follows:

[0160] The gradient change characteristics in the three-dimensional access gradient field are converted into a synaptic signal matrix by simulating the mechanism of biological synaptic signal conduction.

[0161] In this embodiment, the synaptic decision-making mechanism simulates the signal conduction process of the biological nervous system. Specifically, the gradient feature vector of each data block is analogous to the input signal received by a neuron, the synaptic weight represents the influence of different features on the decision, and the activation function simulates the threshold triggering characteristics of the neuron. This mechanism can comprehensively consider multi-dimensional information to make more intelligent migration decisions.

[0162] Determine the initial value T(0) of the dynamic trigger threshold:

[0163] The initial threshold T(0) is determined by benchmarking during the system warm-up phase. The specific method involves executing 100 test migration tasks covering different load modes during the initial system deployment. These test tasks include: hot data migration (30 times), warm data migration (40 times), and cold data migration (30 times). The synaptic signal strength S is recorded when each migration task is triggered. test and the migration result (success or failure). The initial threshold is determined based on the 85% percentile of the migration success rate. For example, the signal strengths of all successful migrations are sorted in ascending order, and the 85th percentile value is taken as T(0). In the test of this embodiment, 88 out of 100 test migrations were successful, and the signal strength range of successful migrations was [1.2, 3.5]. After sorting the signal strengths of the 88 successful cases, the 75th value (88 × 0.85 ≈ 75) is 2.1, so T(0) is set to 2.1.

[0164] Define the target success rate R target and the actual success rate R actual :

[0165] Rtarget Is the migration success rate target value that the system expects to achieve, which is set according to the business SLA requirements. In this embodiment, R is set for key business systems. target =0.95 (95%), for non-critical business, set R target =0.90 (90%). R actual It is the actual success rate of the last W migration tasks under the current threshold T(n), using sliding window statistics. The specific calculation formula is: actual = N success / W, where N success is the number of migrations successfully completed within the window, and W is the window size, which is usually set to 20. For example, at a certain evaluation moment, 17 of the last 20 migration tasks were successfully completed, and 3 failed due to network interruption or insufficient resources. Then R actual = 17 / 20 = 0.85.

[0166] Based on the determination of the dynamic trigger threshold, the current dynamic trigger threshold is multiplied by an adjustment item used to characterize the deviation between the historical migration success rate and the preset target success rate, and a factor reflecting the current system resource abundance, to iteratively generate the dynamic trigger threshold for the next cycle.

[0167] In this embodiment, the dynamic threshold update follows the formula: T(n+1) = T(n) × (1 + γ × (R target -R actual )) × F resource The learning rate γ is set to 0.1 to ensure the smoothness of threshold adjustment. Resource abundance factor F resource The calculation method is: F resource = 1 - (0.4×CPU usage + 0.4×Net usage + 0.2×IO usage ), where CPU usage 、Net usage ,IO usage are the CPU, network, and I / O usage rates, normalized to the range [0, 1]. Specific calculation example: Assume that the current T(n) = 2.1, R target =0.95, R actual =0.85, the system resource usage is CPU 60%, network 40%, I / O 50%. Then F resource= 1 - (0.4×0.6 + 0.4×0.4 + 0.2×0.5) = 0.5. The threshold is updated to: T(n+1) = 2.1 × (1 + 0.1 × (0.95 - 0.85)) × 0.5 = 2.1 × 1.01 × 0.5 = 1.06.

[0168] Through an adaptive adjustment mechanism, when the actual success rate falls below the target, the threshold is lowered to relax the triggering conditions; when system resources are tight, the threshold is raised to reduce the migration frequency. This achieves a dynamic balance between migration decisions and system status.

[0169] In some optional embodiments, the activation function used to map the weighted sum of gradients to signal strength can use, in addition to the sigmoid function, other functions commonly used in deep learning, such as the rectified linear unit (ReLU) function, the leaky rectified linear unit (Leaky ReLU) function, or the hyperbolic tangent (tanh) function, each of which has different computational and gradient propagation characteristics. Furthermore, to aggregate signal strengths from multiple storage nodes, a more complex graph computation model, such as a graph neural network (GNN), can be used instead of a simple weighted summation. GNNs are better able to learn and express complex topological relationships between nodes and their impact on signal propagation.

[0170] In a further embodiment, a specific method for collecting multi-dimensional resource status is provided in a typical Linux server environment in the process of comparing the signal strength in the synaptic signal matrix with the dynamic trigger threshold to generate a migration trigger signal. The system can deploy a monitoring agent (Agent) that periodically collects resource status vectors (for example, every 30 seconds) in the following manner: CPU load: Execute the sar -u 1 1 command and parse the %idle column in the output result. The CPU load rate is (100 - %idle) / 100. Network bandwidth utilization: Read the / proc / net / dev file to obtain the receiving (rx) of a specific network interface (such as eth0) between two sampling points. bytes ) and send (tx bytes ) bytes, divided by the sampling interval and the maximum theoretical bandwidth of the interface to obtain the utilization rate. Storage I / O load: Run the iostat -dx 1 2 command and analyze the %util column of the target disk to directly obtain the I / O utilization rate.

[0171] In addition, regarding the time decay in the historical success rate calculation, the dynamic trigger threshold formula is calculated as T(n+1) = T(n) × (1+γ×(R target - R actual ))×Fresource R in actual When using the exponentially weighted moving average (EWMA) algorithm, it gives higher weight to recent migration records. The specific calculation formula is: actual (t) =λ×R currentmigration +(1-λ) × R actual (t-1); where R actual (t) is the updated weighted historical success rate, R actual (t-1) is the success rate of the last time, R currentmigration is the success or failure of the migration just completed (success is counted as 1, failure is counted as 0), and λ is a decay factor between 0 and 1 (for example, it can be set to 0.1). The smaller the value, the slower the weight of the historical record decays.

[0172] Furthermore, regarding the specific calculation formula of the resource abundance factor, a quantitative calculation method is provided to integrate the multi-dimensional resource utilization into a single adjustment factor. A preferred calculation formula is: F resource = 1 -(w cpu ×CPU load + w net ×Net util + w i0 ×IO util ); where CPU load 、Net util ,IO util is the resource load rate collected and normalized to the interval [0, 1]; w cpu 、w net 、w i0 is the weight coefficient of each resource, the sum of which is 1, and can be configured according to the system bottleneck (for example, if the network is the main bottleneck, w net F resource The closer the value is to 1, the more idle the resources are, and the smaller the inhibitory effect on triggering migration is; the closer the value is to 0, the tighter the resources are, and the stronger the inhibitory effect is.

[0173] Furthermore, regarding the criteria for determining conflicting migrations, the system performs a multi-level conflict check before generating the final migration trigger signal. A conflicting migration occurs when a planned migration task cannot be scheduled because its execution would violate one or more pre-set concurrency or resource constraints. These constraints are designed to prevent performance degradation or state inconsistency caused by excessive migrations. A typical check process is as follows: a global lock service (such as one implemented using Redis or ZooKeeper) is queried to check whether the data block ID to be migrated is locked. If so, an object conflict is detected, and the trigger fails. The task managers of the source and target nodes are queried to obtain the number of currently executing migration tasks. If the number of concurrent tasks on either node has reached the system-defined upper limit (for example, a maximum of three concurrent migration tasks per node), a node conflict is detected, and the trigger fails. A resource pre-check request is sent to the target node, containing the data size required for the migration. The target node checks whether its available storage space meets the requirements. If not, a resource conflict is detected, and the trigger fails. Only when all the above checks are passed will the migration trigger signal be finally confirmed and sent to the execution module.

[0174] In some optional implementations, in addition to the rigid rule-based checks described in this example, a more flexible, cost-based admission control model can be introduced to determine conflicting migrations. In this model, the system estimates the resource overhead (cost) of a new migration task and assesses its potential performance impact on existing online services. A migration task is allowed only if the expected benefits (such as load balancing benefits from the migration) significantly outweigh the expected costs, even if it does not strictly violate the hard concurrency limit.

[0175] 3. To process the shadow copy data to be migrated based on the initial access gradient graph, the data needs to be intelligently partitioned. Specifically, processing the shadow copy data to be migrated based on the initial access gradient graph includes:

[0176] The local extreme value points of the gradient in the initial access gradient map are calculated and used as natural boundaries to define the adaptive fragment boundary set.

[0177] In this embodiment, the physical sharding of the data is matched with the logical boundaries of the access heat. The local extreme point of the gradient refers to the position where the local peak (maximum) and valley (minimum) of the gradient scalar size appear in the gradient field. The peak point usually corresponds to the center of the area where the heat changes most dramatically, while the valley point corresponds to the center of the most stable area. They are used as natural anchor points to define areas with different heat characteristics. The adaptive shard boundary set refers to a set of a series of data offsets that define the start and end positions of each adaptive shard. The set is finally determined by calculating the gradient extreme points and performing alignment optimization. In specific implementation, the gradient field is discretely sampled by an adaptive sampling algorithm, and the sampling density is increased in areas where the gradient changes dramatically to capture more details. For example, the sampling rate is rate Can be compared with the second derivative of the gradient |▽ 2 f| positive correlation. A multi-scale extreme value detection algorithm is used on the set of gradient sampling points to identify local maximum points (hotspot centers) and minimum points (cold zone centers). These extreme points are considered candidate boundary points. To ensure that sharding more closely matches the actual data layout on the storage media, a boundary alignment algorithm can optionally be used to align these candidate boundary points with natural data boundaries, such as file boundaries or underlying storage block boundaries. These optimized boundary points collectively constitute the adaptive shard boundary set, which divides the entire data to be migrated into several intervals with relatively uniform gradient characteristics.

[0178] 4. For each data region divided by the adaptive fragment boundary set, calculate the fragment size that is negatively correlated with it based on the gradient value in each data region, and generate a fragment size mapping table:

[0179] In this embodiment, a specific size is determined for each slice interval. Specifically, a base slice size is divided by a gain term containing the absolute size of the gradient value, where the gain term is a positive correlation function of the absolute size of the gradient value, so that the calculated slice size is inversely proportional to the absolute size of the gradient value. A preferred calculation formula is: Size slice = Size base / (1 + k × |▽f|); where Size slice Size is the calculated fragment size for the region; base is a system-preset benchmark size, such as 16MB; |▽f| is the absolute size of the average gradient value of the area, obtained from the initial access gradient map; k is a positive adjustment coefficient used to control the degree of influence of the gradient on the shard size. According to this formula, the larger the gradient value |▽f| is (hot data), the larger the denominator is, and the calculated shard size is. slice On the contrary, the area with small gradient value (cold data) has a shard size close to Size baseThe calculated shard sizes for all regions are recorded in the shard size map.

[0180] 5. Apply the adaptive shard boundary set and shard size mapping table to partition the shadow copy data to be migrated:

[0181] In this embodiment, the system partitions the actual shadow copy data to be migrated based on shard boundaries (i.e., the start and end locations of each shard) and shard size. This process produces a series of data shards of varying sizes, which are prepared for subsequent compression and transmission. Hot data is partitioned into small chunks for fast, low-latency migration and updates; non-hot data is kept in larger shards, reducing the number of shards and metadata overhead, thereby improving overall migration efficiency.

[0182] In some optional implementations, in addition to finding gradient extremes, shard boundaries can also be determined using data clustering algorithms. For example, the gradient vector (including temporal and spatial components) of each data block can be used as a feature, and clustering algorithms such as K-Means or DBSCAN can be used to automatically group data blocks with similar gradient characteristics into the same cluster. Each cluster naturally forms a shard, ensuring that the access popularity trends within each shard are highly consistent.

[0183] 6. According to one aspect of the present application, obtaining a compressed data fragment sequence and corresponding fragment metadata includes:

[0184] Build an incremental compression dictionary for this migration task based on the data patterns present in the shadow copy data to be migrated.

[0185] Specifically, the incremental compression dictionary refers to a compression dictionary that is dynamically constructed specifically for a single migration task and can be learned and updated online during the migration process. It contains the most frequently occurring repeated data patterns in the current data to be migrated and their corresponding codes, which is essentially different from the general, static compression algorithm dictionary. This embodiment abandons the general, static compression dictionary and instead dynamically constructs an optimal dictionary for each migration task. In specific implementation, repeated data patterns are detected in real time in high-priority data shards to be migrated through sliding window pattern recognition technology. For example, a suffix array or AC automaton algorithm can be used to quickly identify repeated substrings, and their frequency and position can be counted to form a window pattern set. Through the pattern merging algorithm, similar patterns from different windows are clustered (for example, if the similarity of two patterns is greater than a preset threshold of 0.8, they are merged) to form a global pattern library. According to the frequency-based coding allocation algorithm (such as Huffman coding, high-frequency patterns are assigned short codes: code length =-log2(frequency / total count), where code length is the encoding length, frequency is the pattern frequency, total count is the total number of patterns), shorter codes are assigned to patterns with higher frequencies in the global pattern library, and finally an incremental compression dictionary exclusive to this migration task is obtained.

[0186] Furthermore, to enable the dictionary to continuously adapt to data changes, the online learning and update mechanism for the incremental compression dictionary also includes:

[0187] The frequency of new data patterns in the data stream is monitored. When the frequency of a new pattern exceeds a preset inclusion threshold, the new pattern is encoded and added to the incremental compression dictionary. The frequency of existing patterns in the incremental compression dictionary is also periodically evaluated. When the frequency of an existing pattern falls below a preset elimination threshold, the existing pattern is removed from the dictionary. This ensures the timeliness of the compression dictionary, ensuring that it always reflects the latest and most frequent data patterns.

[0188] 7. Use the incremental compression dictionary to encode the shadow copy data to be migrated, and refer to the gradient value in the initial access gradient map to adjust the compression level of the encoding process, and output the encoded data stream:

[0189] Optionally, the output encoded data stream is constructed as a multi-stage pipeline compression process, specifically:

[0190] In the first stage, the incremental compression dictionary is used to replace the identifiable data patterns in the shadow copy data to be migrated with corresponding codes to form a pattern-compressed data stream.

[0191] In this embodiment, dictionary-based rapid pattern replacement can reduce redundant information in data. The pattern-compressed data stream refers to the intermediate product after data has passed through the first stage of the compression pipeline. In this data stream, all longer data patterns recognized by the incremental compression dictionary have been replaced with their corresponding shorter codes, achieving preliminary data compression.

[0192] In the second stage, compression processing is applied to the pattern compression data stream, and the compression level of the compression processing is regulated by the gradient value in the initial access gradient map to obtain a gradient optimized compression stream.

[0193] This embodiment realizes the linking of compression strategy and data importance. Specifically, a base compression level is calculated with an adjustment factor that is positively correlated with the gradient value to determine the final compression level applied to the pattern compression data stream, so that the higher the gradient value, the lower the compression level. The calculation formula of the gradient adaptive compression level is: L comp =L base × (1 + k grad ); where Lcomp is the final compression level; L base is the base compression level; k grad is a regulation factor positively correlated with the gradient value. The underlying logic is that areas with high gradient values ​​are hot data, highly likely to be accessed in the future, and therefore require extremely fast decompression speeds on the target end. By lowering the compression level (for example, switching from heavyweight LZMA to lightweight LZ4, or adjusting the compression algorithm's search depth parameter), some compression ratio can be sacrificed in exchange for decompression performance. Conversely, for cold data with low gradient values, a higher compression level is applied to maximize network transmission bandwidth savings.

[0194] In the third stage, to enhance data reliability, checksum data is embedded into the gradient-optimized compressed stream, forming a checksum-encoded compressed data stream that is then encapsulated as a sequence of compressed data fragments, i.e., the encoded data stream. For example, a CRC32 checksum can be inserted every N compressed blocks.

[0195] 8. Encapsulate the encoded data stream into a compressed data fragment sequence and integrate the statistical information of the compression process to form the fragment metadata:

[0196] In this embodiment, this step is responsible for packaging. The resulting compressed data fragment sequence is a compact binary data stream. The fragment metadata is a structure containing rich information, including, for example, the fragment's original size, compressed size, source-side checksum, the gradient value that triggered the migration, and the version number of the incremental compression dictionary used.

[0197] This embodiment systematically addresses the resource mismatch and efficiency bottlenecks caused by fixed policies through a series of adaptive data processing technologies deeply coupled with access gradients. Gradient-driven adaptive sharding breaks the constraints of fixed-size chunks. For small amounts of critical data with extremely high access gradients (extremely hot data), the system splits it into fine-grained shards for migration, eliminating the significant read / write amplification and bandwidth waste associated with having to move entire 64MB data blocks just to migrate a few KB of metadata. A migration-specific incremental compression dictionary enables the compression algorithm to maximize the locality of the data in this task, achieving higher compression rates than general-purpose compression algorithms. More importantly, the strategy of linking the compression level to the gradient value—using a low compression level for hot data (to ensure fast decompression) and a high compression level for cold data (to maximize bandwidth savings)—addresses the existing technology's inability to strike a balance between transmission efficiency and access performance. This allows the entire data processing and transmission process to tailor the optimal strategy for data of varying hotness, ultimately improving resource utilization and overall efficiency of data migration.

[0198] In some optional implementations, the dictionary pattern elimination mechanism, in addition to frequency thresholds, can also employ more classic cache elimination strategies, such as Least Recently Used (LRU) or Least Frequently Used (LFU), to more finely manage dictionary content, ensuring that the dictionary always contains the freshest or most popular patterns. Furthermore, in the second-level gradient-adaptive compression, in addition to adjusting the compression level of a single compression algorithm, a pool of multiple compression algorithms (such as high-compression LZMA, high-speed LZ4, and balanced ZStandard) can be maintained. The most appropriate compression algorithm for a shard can be directly selected from the pool based on the gradient value.

[0199] 9. According to one aspect of the present application, the step of performing asynchronous transmission and verification includes, before initiating transmission:

[0200] A hierarchical data block structure is constructed at the source end according to the compressed data fragment sequence, and a source-end hierarchical checksum tree is calculated for the hierarchical data block structure.

[0201] Specifically, the source side uses a recursive binary partitioning algorithm to split a large compressed data shard into a tree-like hierarchical data block structure, where leaf nodes are the smallest transmission units that fit within the network's MTU (maximum transmission unit). Based on this structure, a source-side hierarchical checksum tree is calculated for each layer. A preferred implementation involves applying independent hash algorithms to different levels of the hierarchical data block structure. For example, the efficient CRC32 algorithm is used for the bottom-level leaf nodes (data blocks), the MD5 algorithm is used for the middle layers (shards consisting of multiple data blocks), and the more secure SHA256 algorithm is used for the root node (the entire migration task). This multi-algorithm strategy effectively avoids hash collisions at different granularities and enhances the robustness of the checksum. The hierarchical checksum tree is a tree-like data structure inspired by the Merkle tree. The leaf nodes of the tree store the checksum of the smallest data block, while the checksum of each non-leaf node is calculated by concatenating the checksums of all its direct child nodes. This allows for highly efficient integrity verification of any data subset.

[0202] A corresponding target-side checksum framework is pre-created on the target side according to topology information of the hierarchical data block structure.

[0203] In this embodiment, before the transmission officially begins, the source sends the checksum tree information to the target, which contains only the topology and the checksum size of each node, without the data content. Based on this information, the target creates a target-side checksum frame in memory with the same structure but empty content. This is equivalent to pre-building a mold for filling in the checksum.

[0204] The source-side hierarchical checksum tree and the target-side checksum framework are coordinated to establish a two-way check anchor sequence, which is used for incremental check in subsequent asynchronous transmission processes.

[0205] Specifically, a two-way checksum anchor sequence refers to a series of synchronization points for incremental verification that are pre-negotiated between the source and target before transmission. These synchronization points are typically specific nodes in a hierarchical checksum tree. During the transmission process, whenever the data corresponding to an anchor point is fully transmitted, both parties compare the checksum of that anchor point to achieve incremental confirmation. The two-way checksum anchor sequence is a predefined protocol that specifies when and at which level the checksums of the source and target ends are synchronized and compared during data transmission. For example, the protocol may specify that the MD5 checksum of the shard level to which it belongs is compared once every 1MB of data is successfully transmitted. These anchor points provide the basis for subsequent rapid breakpoint location.

[0206] 10. Furthermore, when asynchronous transmission is interrupted, the following intelligent recovery steps are performed:

[0207] The checksums are compared by performing a binary search on the bidirectional checkpoint anchor sequence to locate the last data consistent checkpoint.

[0208] In this embodiment, when a transmission interruption is detected (e.g., a heartbeat timeout), the system does not retransmit from the last simple offset record point. Instead, it performs an efficient binary search algorithm on the checksum tree represented by the established bidirectional checksum anchor point sequence. For example, the SHA256 checksum of the root node is compared. If there is a mismatch, the binary search is recursively continued within the subtree with the mismatching checksum until the deepest node with a consistent checksum is found. This deepest node is the last consistent checkpoint, precisely indicating the starting point of data corruption or loss.

[0209] An incremental retransmission plan is generated based on the checkpoint to resume data transmission.

[0210] Specifically, once a consistency checkpoint is located, the system accurately calculates the minimum data set that needs to be retransmitted: all data blocks from that checkpoint to the interruption point. The resulting incremental retransmission plan only includes the data that actually needs to be retransmitted, minimizing the amount of data retransmitted and improving migration efficiency and success rates in unstable network environments.

[0211] This embodiment improves the reliability and recovery efficiency of the data migration process by collaboratively constructing a hierarchical checksum tree on the source and target sides and utilizing bidirectional checksum anchors for interruption recovery. Simple recovery mechanisms based on fixed checkpoints often require retransmission of large amounts of correctly received data in the event of a transmission interruption. This results in extremely low recovery efficiency, especially in high-latency, unstable network environments. This embodiment uses a binary search algorithm on the checksum tree to precisely locate the last consistent data point, exponentially improving location efficiency (from linear time complexity O(n) to logarithmic time complexity O(log n)). In the case of cross-regional migrations of large data centers, this reduces recovery time due to network jitter from minutes to seconds. This not only saves valuable network bandwidth but, more importantly, reduces the failure rate of migration tasks and improves system availability. For users, this means that data migration can be completed robustly and quickly even under complex network conditions, ensuring business continuity. By establishing a collaborative, hierarchical checksum system on the source and target sides, breakpoints can be quickly and accurately located and incremental retransmissions minimized.

[0212] In some optional implementations, a binary search algorithm for recovery is a preferred solution. As a simpler alternative with lower implementation complexity, the system can also adopt a mechanism based on forward jump confirmation. Under this mechanism, the target end periodically (for example, after successfully receiving N data blocks) calculates and sends back to the source end an aggregate checksum of all data received to date. The source end can confirm that the previous transmission is correct by comparing this value. If an interruption occurs, it will retransmit from the point of the last successful confirmation. In addition, for the hash algorithms used at different levels in the checksum tree, in addition to using different categories of algorithms, it is also possible to use the same high-performance hash algorithm (such as BLAKE3), but configure a different, predefined salt value (Salt) for each level, which can also effectively prevent cross-level hash fraud or collisions.

[0213] 11. In a further embodiment, the asynchronous transmission and bidirectional incremental verification process is further refined, especially the interruption recovery link, specifically:

[0214] Regarding the timeout mechanism in transmission interruption detection, an adaptive timeout calculation method is provided that is linked to the data heat (gradient). The specific relationship formula is as follows: timeout = base timeout × (1 + c / |▽f|); where timeout is the timeout set for a data segment transmission; base timeoutis a system-wide baseline timeout, for example, 5 seconds; |▽f| is the absolute value of the gradient value recorded in the shard's metadata; and c is a positive constant adjustment factor. Hot data with higher gradient values ​​|▽f| is more urgent and important to migrate, so its timeout should be set shorter to allow the system to more quickly detect transmission anomalies and initiate recovery mechanisms. Conversely, for cold data with lower gradient values, a longer timeout can be tolerated.

[0215] Optionally, regarding the specific termination conditions of the binary search algorithm in the two-way checkpoint comparison, it is stipulated as follows: the algorithm is performed on a hierarchical checksum tree. The search process starts from the root node and proceeds recursively downward. Its termination condition is: when a non-leaf node P of the checksum tree is recursively found, it is found that the checksum of the P node itself is completely consistent at the source end and the target end, but among the direct child nodes of the P node, the checksum of at least one child node C is inconsistent at the source end and the target end. At this time, the binary search terminates, and the node P is determined as the last data consistent checkpoint. This means that within the data range represented by the P node, the error occurs within the sub-range corresponding to the C node.

[0216] Furthermore, regarding the precise determination of the minimum data set algorithm in the generation of the incremental transmission plan, the specific implementation is as follows: After locating the last data-consistent checkpoint P and the first inconsistent child node C, the system will accurately construct a minimized retransmission set. The starting position of this set is the starting offset of the original data block corresponding to the inconsistent child node C in the entire replica to be migrated. The ending position of this set is the ending offset of the last data block originally planned to be transmitted. In other words, the minimized retransmission set accurately contains all remaining data blocks from the first known bad data block to the end of the task. This avoids retransmitting any data that has been confirmed to be successfully transmitted.

[0217] Furthermore, regarding the specific definitions and calculation methods of performance indicators in transmission status updates, the transmission status record includes at least the following performance indicators: Effective transmission rate: This is clearly defined as the amount of payload data successfully received per unit time without errors and duplication at the application level. It truly reflects the transmission speed perceived by the user, in contrast to throughput, which typically includes protocol overhead and retransmitted data. Its calculation formula is (the total original size of the data successfully transmitted) / (the total time from the start of migration to the completion of migration). This indicator reflects the end-to-end actual business data transmission efficiency. Retransmission rate: This is calculated as (the total amount of data retransmitted due to interruption or verification failure) / (the total original size of the data successfully transmitted). This indicator is a key parameter for measuring network link quality and transmission protocol efficiency; the lower the value, the better. Recovery efficiency: This is calculated as (the total time from detecting the interruption to successfully locating the breakpoint) / (the total interruption duration). This indicator is used to measure the performance of the breakpoint recovery mechanism of the present invention. The lower the value, the faster the positioning and the higher the recovery efficiency.

[0218] In some optional implementations, to further improve transmission efficiency in harsh network environments (such as wide area networks with high latency and high packet loss rates), in addition to the passive retransmission recovery mechanism of this embodiment, an active fault-tolerance mechanism, namely forward error correction (FEC), can be introduced. When transmitting data, the source end calculates and sends some additional redundant data (error correction code). If a small amount of packet loss occurs in the data received by the destination end, this redundant data can be used to directly recover the lost original data without requesting retransmission from the source end, thereby reducing the round-trip transmission delay.

[0219] 12. According to one aspect of the present application, after a migration task is completed and integrity verified, the system verifies the integrity of the migrated data based on the transferred data blocks and the transfer status record, generates a migration completion status, and updates the initial access gradient graph based on real-time access requests to obtain an optimized gradient model for subsequent migrations. The steps of updating the initial access gradient graph to obtain the optimized gradient model specifically include:

[0220] Based on the transfer status records and migration completion status, the overall performance of this migration task is quantitatively evaluated and a migration effect score is generated.

[0221] In this embodiment, an abstract migration task is converted into a quantifiable score. The migration effect score refers to a normalized value used to quantify the comprehensive performance of a single migration task. It integrates multiple key performance indicators (KPIs) in the migration process, such as speed, success rate, resource consumption, etc., into a single score that is easy to compare and evaluate through weighted summation. Specifically, the system extracts key performance indicators (KPIs) from the transmission status record, such as the total migration time, effective bandwidth, CPU and I / O resource consumption rate, and whether the migration is ultimately successful. The migration effect score Score is calculated using a multi-indicator scoring function. perf A preferred scoring function is: Score perf = α perf × V mig + β perf × R succ + γ perf × E res ; where V mig is the speed indicator, R succ is the success rate (success is 1, failure is 0), E res is the quantitative value of resource efficiency; α perf , β perf , γ perf It is an administrator-configurable weight factor that balances the different priorities of speed, reliability, and cost.

[0222] Compare the transfer effect score with the preset expected value and calculate the reinforcement learning reward value.

[0223] This embodiment generates a reward or penalty signal to drive learning. The reinforcement learning reward value refers to a scalar signal generated based on the comparison between the transfer effect score and the expected value, which is used to guide the update of model parameters. It is usually designed to be a positive value (reward) or a negative value (penalty), which directly reflects the system's evaluation of the previous decision-making behavior. Specifically, the system maintains a preset expected performance score expected score . Reinforcement learning reward value R reward Can be done through R reward =(Score perf - expected score ) / expected score To calculate. If Score perf Higher than expected, then R reward If it is positive, it means that the decision that triggers this migration will be rewarded; otherwise, it is negative, which means that it will be punished.

[0224] Apply the reinforcement learning reward value to iteratively update the synaptic weights that triggered this migration, forming an optimized synaptic weight matrix for subsequent migration triggering decisions.

[0225] In this embodiment, the reward value is applied to the update of the decision model parameters. The system will trace back to which data blocks and which gradient features (such as time gradient and spatial gradient) generated the synaptic signal that ultimately triggered this migration. reward Iteratively update the relevant synaptic weights w(t). A preferred weight update formula, as shown in the gradient descent algorithm: w(t+1) = w(t) +η rl ×R reward ×S signal ×(1 - w(t)); where w(t+1) is the new weight after update; η rl is the learning rate, which controls the update step size; S signal is the signal strength when the migration was initially triggered. If a migration receives a positive reward, the weight w(t) contributing to this trigger is strengthened, making it easier for the system to trigger migration again under similar gradient characteristics in the future. Conversely, if a negative reward is received, the relevant weight is weakened, thereby inhibiting similar, potentially ineffective migration decisions in the future. Through continuous iterations of thousands of migrations, the synaptic weight matrix of the entire system will become increasingly intelligent, and its decisions will increasingly meet the expected performance goals.

[0226] This embodiment introduces a reinforcement learning-based feedback optimization loop, enabling the entire migration decision system to continuously learn and evolve from its own experience. This solves the problem of fixed decision model parameters and inability to adapt to long-term load evolution in existing technologies. In traditional migration solutions, all policies and parameters (such as thresholds and weights) are rarely modified once configured. This embodiment, however, treats each migration as a practice, quantifying its migration effect and comparing it with the expected value to generate a reward or penalty signal. This signal is then used to fine-tune the upstream synaptic weight parameters through algorithms such as gradient descent. This creates a complete perception-decision-execution-feedback loop. The system can automatically and continuously adapt to the unique load patterns of the specific business environment in which it is deployed. For example, if the system finds that migrating a certain type of data consistently yields unexpected performance improvements (high rewards), it will automatically increase the weight that triggers this migration, while suppressing it if it doesn't. This makes the long-term operational efficiency of the present invention far superior to any statically configured system, enabling truly personalized data migration services tailored to each user and business scenario.

[0227] In some optional embodiments, the learning model that drives the weight update can be a more complex model in addition to the direct weight adjustment described in this embodiment. For example, it can be a small neural network or decision tree, whose network weights or tree structure are updated according to the reward value of reinforcement learning. In addition, the design of the reward function can also be nonlinear. For example, for migration tasks whose performance far exceeds expectations, an exponentially growing reward value is given, while for tasks that are slightly below expectations, only a small penalty is given, which can encourage the system to more boldly explore migration strategies that may bring huge benefits.

[0228] In a further embodiment, regarding the migration effect evaluation α perf , β perf , γ perf The value range and setting principle of the weight coefficient are as follows: Value range: α perf , β perf , γ perf All of them are floating point numbers in the range [0, 1], and the sum of the three should always be 1. These weights are configured by the system administrator based on the priority of the business scenario. In performance-sensitive scenarios (such as data migration of online trading systems), the migration speed should be prioritized. You can set α perf (speed weight) is higher, such as α perf =0.7,β perf =0.2,γ perf =0.1. In cost-sensitive or background archiving scenarios, resource efficiency should be prioritized to avoid affecting online business. You can set γ perf (resource efficiency weight) is high, such as α perf =0.2,β perf =0.2,γ perf =0.6.

[0229] Furthermore, the expected performance is expected score During initialization, when the system is first launched, a series of benchmark migration tests (for example, 100 times) covering different data types and network conditions can be performed to calculate the scores of these tests. perf The average value of score In order to make the expected value adapt to the long-term performance evolution of the system during dynamic adjustment, a smooth dynamic adjustment mechanism is adopted. For example, the exponentially weighted moving average (EWMA) is used: expected score (new) = 0.99 × expected score (old) + 0.01 × actual score ; Among them, actual scoreis the actual score of the most recent migration, which allows the expected performance score to slowly and steadily track the overall performance baseline of the system.

[0230] Furthermore, the learning rate η in the weight gradient descent update is proposed rl In order to balance the rapid convergence in the early stage of learning and the stability in the later stage, and to avoid the drastic fluctuation caused by a single abnormal migration after the model matures, an adaptive learning rate strategy with time decay is adopted. A preferred strategy is: η rl (t) = η rl_initial / (1 + decay rate × N migrations ); where η rl (t) is the current actual learning rate; η rl_initial is the initial learning rate, which can be set to a relatively large value, such as 0.1; N migrations is the total number of migrations that the system has completed; decay rate is a small decay rate, such as 0.001. This strategy makes the learning rate larger in the early stages of system operation, enabling rapid learning. As experience accumulates, the learning rate gradually decreases, making the model parameters tend to be stable.

[0231] In some optional embodiments, the learning rate η rl In addition to the attenuation strategy based on the number of training cycles used in this embodiment, more advanced adaptive optimization algorithms, such as Adam or RMSprop, can also be used for adaptive adjustment. These algorithms can maintain an independent, adaptive learning rate for each weight parameter in the model. For parameters that are updated frequently, the learning rate will be appropriately reduced; for parameters that are updated sparsely, the learning rate will be maintained or increased. This more refined adjustment strategy usually allows the model to converge faster and more stably.

[0232] 13. According to one aspect of the present application, the step of performing asynchronous transmission and verification further includes:

[0233] According to the gradient information contained in the shard metadata, priority scheduling is performed on the compressed data shard sequence. The scheduling at least includes placing shards with higher gradient values ​​at the front of the transmission queue or allocating more parallel transmission channels to them.

[0234] Priority scheduling refers to a resource scheduling strategy that allocates transmission resources (such as queue position and the number of available parallel channels) based on the task's preset priority, not the order in which tasks arrive. In this embodiment, priority is uniquely determined by the access gradient value of a data shard. This optimization is motivated by the fact that not all data shards are equally important. Shards with high gradient values ​​are hot data, which users or applications are likely to access immediately after migration. Therefore, prioritizing the transmission of these shards can significantly reduce service-perceived latency. Specific implementations may include the following two or a combination of the following: Priority-based transmission queue: When sending compressed data shards to the transmission queue, the system uses a priority queue instead of a simple first-in-first-out (FIFO) queue. The priority of each shard is determined by the gradient value recorded in its metadata. Shards with higher gradient values ​​are assigned higher priority. Therefore, the transmission engine always selects the highest-priority shard from the head of the queue for transmission, ensuring that hot data blocks are always transmitted before cold data blocks. Gradient-based dynamic allocation of parallel channels: Assume that the system maintains a transmission channel pool consisting of N parallel TCP channels. During scheduling, the system dynamically allocates channels based on the gradient value of each shard. For example, a gradient threshold can be set. For hot data shards with gradient values ​​above the threshold, m (e.g., m = 4) parallel channels are allocated for concurrent transmission; for common or cold data shards with gradient values ​​below the threshold, only n (e.g., n = 1) channels are allocated. Hot data is not only logically prioritized but also receives more bandwidth resources at the physical transmission level, reaching the target node several times faster than cold data. By introducing a priority scheduling mechanism, the entire migration process is more user-friendly for upper-layer applications. While the total completion time of the migration may remain unchanged, the availability of critical data is significantly improved, optimizing user experience and system responsiveness.

[0235] In some optional implementations, in addition to strict priority queue scheduling, a more fair Weighted Fair Queuing (WFQ) algorithm can also be used. In WFQ, each shard is assigned a weight based on its gradient value, and the system allocates bandwidth resources to it in proportion to the weight. This allows even cold data shards with very low gradient values ​​to obtain a minimum transmission bandwidth, avoiding the problem of starvation (i.e., long periods of no transmission) in extreme cases. Furthermore, the scheduling priority can be a composite value, for example, Priority = w grad × gradient value + w size × (1 / shard size), where w gradis the weight coefficient of the gradient. This allows us to prioritize the transmission of hot data while also tending to complete smaller and easier-to-transmit shards first, thereby improving the overall task turnover rate of the system.

[0236] In a specific embodiment, the process of generating synaptic signals is shown. The scenario is set as follows: current node: N1; data block to be evaluated: DB A ; Input parameters include: at the current time point t current , data block DB to be evaluated A The normalized gradient eigenvector is g = [g t , g s ] = [0.8, 0.5], where g t is the time gradient component, g s is the spatial gradient component. The weight in the synaptic weight matrix corresponding to the normalized gradient feature vector g is read from the system as w = [w t , w s ] =[0.6,0.4]. The activation function used by the system is f act (x) = 1 / (1+e (-2x) ).DB A The historical raw synaptic signal strength in the past two time slices is recorded as S(t current -2) = 0.5, S(t current -1) = 0.6. The time decay factor d is set to 0.8. The neighbor nodes of node N1 are N2 and N3, and the S reported by them for the same logical data (or its copy) is cumulative The values ​​are S cumulative (N2) = 0.7, S cumulative (N3) = 0.4. The network distance weight calculated based on the network topology is w dist (N1, N2) = 0.2, w dist (N1, N3) = 0.1. The dynamic trigger threshold T of the current system is 1.70. Calculate the original synaptic signal strength S(t current ): Calculate the weighted sum: Σ(w i × g i ) = (w t × g t ) + (w s × g s ) =(0.6 × 0.8) + (0.4 × 0.5) = 0.48 + 0.20 = 0.68. Through the activation function mapping: S(t current ) =f act (0.68) = 1 / (1 + e (-2 × 0.68)) = 1 / (1 + e (-1.36) ) ≈ 1 / (1 + 0.2567) ≈ 0.7957. Compute DB on node N1 A The time series cumulative signal strength S cumulative (N1): Apply the time decay formula: S cumulative = Σ(S t × d (tcurrent - t) ). Substitute the value: S cumulative (N1) = (S(t current -2) × d 2 ) + (S(t current -1) × d 1 ) + (S(t current ) × d 0 ) = (0.5 × 0.8 2 ) + (0.6 × 0.8 1 ) + (0.7957 × 0.8 0 ) = (0.5 × 0.64) + (0.6 × 0.8) + (0.7957 × 1) = 0.32 + 0.48 + 0.7957 = 1.5957. A The final aggregate signal intensity S aggregated (N1): Apply the multi-node signal aggregation formula: S aggregated = S local + Σ(S neighbor × w dist ). Substitute the value: S aggregated (N1) = S cumulative (N1) +(S cumulative (N2) × w dist (N1,N2)) + (S cumulative (N3) × w dist (N1, N3)) = 1.5957 + (0.7 × 0.2) + (0.4 × 0.1) = 1.5957 + 0.14 + 0.04 = 1.7757. The calculated final aggregate signal strength is compared with the dynamic trigger threshold, S aggregated (N1) (1.7757)>T (1.70). Since the signal strength exceeds the current threshold, the system will A Generate a migration trigger signal to start the subsequent migration process.

[0237] It should be noted that all specific values ​​given in this embodiment, such as the weight coefficient w = [0.6, 0.4] and the attenuation factor d = 0.8, are illustrative values, intended to clearly illustrate the calculation process. In actual system deployment, the optimal values ​​of these parameters are typically determined through extensive offline training and online tuning based on actual business loads, or adaptively adjusted using the reinforcement learning mechanism described in this invention.

[0238] In this solution, adaptive sharding and incremental compression achieve a synergistic effect. This synergistic effect occurs when two or more technical features (such as adaptive sharding and incremental compression) combine to produce technical results (such as improved network bandwidth utilization) that exceed the sum of the effects achieved by applying these features independently. In this solution, gradient-driven adaptive sharding and migration-specific incremental compression are not simply a front-and-rear process, but rather a deeply coupled and synergistic mechanism. Specifically, the adaptive sharding mechanism divides data into two main regions based on access gradients: hot data regions with high gradient values ​​(split into small shards) and cold data regions with low gradient values ​​(split into large shards). This creates an ideal prerequisite for the incremental compression mechanism. The synergistic effect is reflected in the fact that for small shards of hot data, whose content changes frequently, the online learning dictionary update in the incremental compression mechanism can quickly capture these emerging high-frequency patterns. Small shards also allow for smaller and more focused dictionaries, resulting in extremely fast pattern matching and encoding replacement, perfectly meeting the requirements for low-latency access and fast migration of hot data. For large shards of cold data, the content is stable and rarely changes. When processing this data, the compression pipeline can invest more computing resources in deep compression (for example, using a larger sliding window and search depth) to achieve the ultimate compression rate. Because cold data is insensitive to migration and decompression speeds, trading time for space (higher compression rates) is the optimal strategy. As can be seen, adaptive sharding pre-classifies data, while incremental compression applies optimized, differentiated compression strategies for different data categories. This combination of pre-classification and post-compression strategies enables the system to achieve significantly improved network bandwidth utilization beyond what could be achieved with a single technology alone.

[0239] Furthermore, intelligent triggering and reliable transmission achieve a complementary fault-tolerance effect. This refers to a system-level synergy where the strengths of one subsystem (such as reliable transmission) can offset the potential weaknesses of another subsystem (such as intelligent decision-making) in certain scenarios, and vice versa. This results in the overall system exhibiting greater robustness in the face of anomalies and uncertainty. In this invention, the intelligent synaptic triggering mechanism and the bidirectional incremental verification reliable transmission mechanism form a complementary fault-tolerant closed loop of intelligent decision-making and reliable execution. This complementary relationship is reflected in the fact that in the early stages of learning or when faced with sudden changes in access patterns, the synaptic model may make suboptimal or even erroneous triggering decisions due to insufficient learning. In such cases, the reliable transmission mechanism acts as a safety net. Its efficient breakpoint-resume capability allows tasks to be completed or aborted with minimal effort, even when unnecessary migrations are triggered, reducing the waste of network resources and system jitter that can result from erroneous decisions. Conversely, when executing tasks, the transmission system perceives the true state of the underlying network. For example, frequent transmission interruptions to a target node are a crucial signal of environmental conditions. This state information can be fed back into the decision-making mechanism. For example, it can serve as a negative adjustment factor to temporarily and specifically raise the dynamic trigger threshold for migration to the unstable node, thereby inhibiting migration to that node. This forms a higher-dimensional feedback loop, allowing migration decisions to consider not only the access popularity of the data itself but also the physical reliability of the migration path, further improving the intelligence level of the entire system and the migration success rate.

[0240] In some optional implementations, in addition to the synergistic effects described in this embodiment, there are also strong coupling relationships between other technical features. For example, there is synergy between the reinforcement learning feedback loop and the adaptive timeout mechanism. The reinforcement learning model can analyze the historical migration records and find that the network delay to certain target nodes is generally high. As a learning result, in addition to adjusting the trigger weight, the model can also actively increase the baseline timeout parameters sent to these nodes, thereby making the timeout detection mechanism itself adaptive.

[0241] like Figure 4 As shown, another embodiment of the present invention also provides a distributed storage shadow copy migration device, including:

[0242] An acquisition module, configured to obtain monitoring data of a distributed storage system, wherein the monitoring data is related to access conditions of data in the distributed storage system;

[0243] a generating module for generating an access gradient map based on the monitoring data, wherein the access gradient map is formed by a three-dimensional access gradient field for characterizing changes in data access popularity, wherein the three-dimensional access gradient field involves a temporal gradient and a spatial gradient of data access;

[0244] A calling module is used to call a pre-built migration model, process the access gradient graph based on the migration model, and obtain a target shadow copy to be migrated, wherein the migration model is used to determine the shadow copy to be migrated based on the input access gradient graph;

[0245] The migration module is used to generate a migration trigger signal to trigger the migration of the target shadow copy.

[0246] Another embodiment of the present invention further provides an electronic device, including:

[0247] one or more processors;

[0248] a memory configured to store one or more programs;

[0249] When the one or more programs are executed by the one or more processors, the one or more processors implement the distributed storage shadow copy migration method as described in any one of the above.

[0250] Furthermore, an embodiment of the present invention provides a storage medium storing a computer program that, when executed by a processor, implements the distributed storage shadow copy migration method described above. It should be understood that each solution in this embodiment has the corresponding technical effects of the aforementioned method embodiments and will not be further elaborated here.

[0251] Furthermore, an embodiment of the present invention also provides a computer program product, which is tangibly stored on a computer-readable medium and includes computer-readable instructions, which, when executed, enable at least one processor to perform a distributed storage shadow copy migration method such as that in the embodiment described above.

[0252] It should be noted that the computer storage medium of the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable media may include, but are not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage medium, a magnetic storage medium, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. In the present invention, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program configured for use by or in conjunction with an instruction execution system, system, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, antenna, optical cable, RF, or any suitable combination thereof.

[0253] Furthermore, those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage) containing computer-usable program code.

[0254] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1a process or multiple processes and / or boxes Figure 1 A system that specifies the functions of a box or boxes.

[0255] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction system that is implemented in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0256] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of protection of the present application is limited to these examples. In line with the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

Claims

1. A distributed storage shadow copy migration method, characterized in that: include: Obtaining monitoring data of the distributed storage system, wherein the monitoring data is related to access status of data in the distributed storage system; generating an access gradient map based on the monitoring data, wherein the access gradient map is formed by a three-dimensional access gradient field for characterizing changes in data access popularity, wherein the three-dimensional access gradient field involves a temporal gradient and a spatial gradient of data access; Calling a pre-built migration model, processing the access gradient graph based on the migration model, and obtaining a target shadow copy to be migrated, wherein the migration model is used to determine the shadow copy to be migrated based on the input access gradient graph; Generate a migration trigger signal to trigger migration of the target shadow copy; Generating an access gradient graph based on the monitoring data includes: Aggregating the monitoring data in a time dimension to form a temporal access matrix, wherein the temporal access matrix includes access status of each data in a continuous time slice; determining a time gradient component for characterizing a change in data access frequency based on the temporal access matrix; Determining a spatial gradient component for characterizing a change trend in spatial distribution of data access heat based on the temporal access matrix and logically adjacent data blocks; synthesizing the temporal gradient component and the spatial gradient component to form the three-dimensional access gradient field; The synthesizing the temporal gradient component and the spatial gradient component to form the three-dimensional access gradient field includes: The temporal gradient component and the spatial gradient component are synthesized based on the following formula to form the three-dimensional access gradient field: described is the gradient vector after the fusion of the temporal gradient component and the spatial gradient component, the sqrt((Ψf / Ψt) 2 +(Ψf / Ψs) 2 ) represents the severity of the change in data access heat, the Ψf / Ψt and Ψf / Ψs are the time gradient and space gradient components respectively, the direction vector Indicates the direction of change in data access popularity. vector The gradient vector formed by fusing the temporal gradient component and the spatial gradient component is normalized; Constructing the migration model includes: Obtaining historical monitoring data of the distributed storage system; Generate a historical access gradient graph based on the historical monitoring data; The historical access gradient graph is used to simulate the mechanism of biological neural synaptic signal transmission and is converted into a synaptic signal matrix. The synaptic signal matrix is ​​used to determine the aggregate signal strength of each data on a storage node. The aggregate signal strength is used to quantify the necessity of migrating the shadow copy of the data. Determine a dynamic trigger threshold based on a historical migration success rate in the historical monitoring data and a real-time resource status of the distributed storage system, wherein the dynamic trigger threshold is used to measure the aggregate signal strength; The migration model is constructed based on the synaptic signal matrix and the dynamic trigger threshold.

2. The distributed storage shadow copy migration method according to claim 1, characterized in that: The mechanism of simulating biological synaptic signal conduction by using the historical access gradient map and converting it into a synaptic signal matrix includes: Constructing a normalized gradient feature vector based on the historical access gradient map; Obtaining migration records of shadow copies during a historical period, and determining an initial synaptic weight matrix based on the migration records of shadow copies during the historical period; The normalized gradient eigenvector is weighted by using the initial synaptic weight matrix, and the elements between the initial synaptic weight matrix and the normalized gradient eigenvector are mapped by a nonlinear biological neuron activation function to generate historical synaptic signal strengths, wherein the historical synaptic signal strengths corresponding to different data are combined to form a historical sequence; Processing the historical sequence using a time decay function to determine a time series cumulative signal strength; The synaptic signal matrix is ​​formed by aggregating the time-series cumulative signal intensities corresponding to different data.

3. The distributed storage shadow copy migration method according to claim 2, characterized in that: Each of the time series accumulated signal strengths corresponds to data of a storage node; The aggregating the time series cumulative signal strengths corresponding to different data to form the synaptic signal matrix includes: Determine a network distance weight corresponding to each storage node based on the distances between different storage nodes; Adjusting the sequential cumulative signal strength of adjacent storage nodes based on the network distance weight of each storage node; Fusing the time series cumulative signal strength of each storage node with the adjusted time series cumulative signal strength of adjacent storage nodes to generate a target fusion signal; The synaptic signal matrix is ​​formed based on the target fusion signals of each of the storage nodes.

4. The distributed storage shadow copy migration method according to claim 2, characterized in that: The calling of the pre-built migration model, processing the access gradient graph based on the migration model, and obtaining the target shadow copy to be migrated includes: Updating the current dynamic trigger threshold in combination with the current resource status of the distributed storage system; Processing the access gradient map based on the synaptic signal matrix to obtain a current aggregate signal strength corresponding to each of the storage nodes; Comparing the current aggregate signal strength with the updated dynamic trigger threshold to determine the target storage node; The target shadow copy to be migrated is determined based on the data in the target storage node.

5. The distributed storage shadow copy migration method according to claim 1, characterized in that: Before migrating the shadow copy, the following steps are performed: Determine a local gradient extreme point in the access gradient graph, and determine an adaptive slice boundary set based on the local gradient extreme point; For each data region divided in the access gradient map based on the adaptive shard boundary set, determining a shard size that is negatively correlated in value according to the gradient value of each data region, so as to generate a shard size mapping table according to the shard size; The shadow copy data to be migrated is partitioned based on the adaptive shard boundary set and the shard size mapping table to obtain a plurality of data shards of different sizes, where the size of the data shard is inversely proportional to the data access heat.

6. The distributed storage shadow copy migration method according to claim 5, characterized in that: Migrating the shadow copy includes: Building a delta compression dictionary that matches the data pattern of the shadow copy; Encoding each data slice of the shadow copy based on the incremental compression dictionary, and determining a compression level in the encoding process according to a gradient value in the access gradient map, to obtain encoded data of the shadow copy; Encapsulating the encoded data to form a compressed data fragment sequence, and forming fragment metadata based on compression information during the encoding process; The data fragment sequence and fragment metadata are transmitted and verified.

7. A distributed storage shadow copy migration device, characterized in that: include: An acquisition module, configured to obtain monitoring data of a distributed storage system, wherein the monitoring data is related to access conditions of data in the distributed storage system; a generating module for generating an access gradient map based on the monitoring data, wherein the access gradient map is formed by a three-dimensional access gradient field for characterizing changes in data access popularity, wherein the three-dimensional access gradient field involves a temporal gradient and a spatial gradient of data access; A calling module is used to call a pre-built migration model, process the access gradient graph based on the migration model, and obtain a target shadow copy to be migrated, wherein the migration model is used to determine the shadow copy to be migrated based on the input access gradient graph; A migration module, configured to generate a migration trigger signal to trigger the migration of the target shadow copy; Generating an access gradient graph based on the monitoring data includes: Aggregating the monitoring data in a time dimension to form a temporal access matrix, wherein the temporal access matrix includes access status of each data in a continuous time slice; determining a time gradient component for characterizing a change in data access frequency based on the temporal access matrix; Determining a spatial gradient component for characterizing a change trend in spatial distribution of data access heat based on the temporal access matrix and logically adjacent data blocks; synthesizing the temporal gradient component and the spatial gradient component to form the three-dimensional access gradient field; The synthesizing the temporal gradient component and the spatial gradient component to form the three-dimensional access gradient field includes: The temporal gradient component and the spatial gradient component are synthesized based on the following formula to form the three-dimensional access gradient field: described is the gradient vector after the fusion of the temporal gradient component and the spatial gradient component, the sqrt((Ψf / Ψt) 2 +(Ψf / Ψs) 2 ) represents the severity of the change in data access heat, the Ψf / Ψt and Ψf / Ψs are the time gradient and space gradient components respectively, the direction vector Indicates the direction of change in data access popularity. vector The gradient vector formed by fusing the temporal gradient component and the spatial gradient component is normalized; Constructing the migration model includes: Obtaining historical monitoring data of the distributed storage system; Generate a historical access gradient graph based on the historical monitoring data; The historical access gradient graph is used to simulate the mechanism of biological neural synaptic signal transmission and is converted into a synaptic signal matrix. The synaptic signal matrix is ​​used to determine the aggregate signal strength of each data on a storage node. The aggregate signal strength is used to quantify the necessity of migrating the shadow copy of the data. Determine a dynamic trigger threshold based on a historical migration success rate in the historical monitoring data and a real-time resource status of the distributed storage system, wherein the dynamic trigger threshold is used to measure the aggregate signal strength; The migration model is constructed based on the synaptic signal matrix and the dynamic trigger threshold.

Images