Digital enterprise management data storage method and system based on cloud computing

Abstract

The invention provides a digital enterprise management data storage method and system based on cloud computing, and the method comprises the steps: obtaining a real-time management data set in an enterprise data flow, generating a virtualized storage node, determining a target encrypted fragment index matched with the virtualized storage node according to a preset storage path optimization model, and storing the target encrypted fragment index into a database; based on a dynamic storage path corresponding to the target encrypted fragment index, segmenting the structured data block and the unstructured data block into encrypted data units distributed across regions, and mapping the encrypted data units to corresponding cloud storage partitions; dynamically distributing storage resources of the cloud storage partitions through a load balancing algorithm, and adjusting priority weights of dynamic storage paths according to real-time flow characteristics; and when an access request of the encrypted data unit is detected, redistributing the concurrent connection number of the cloud storage partitions. According to the method, the collaborative optimization problem of resource allocation and safety efficiency can be solved.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a digital enterprise management data storage method and system based on cloud computing. Background Art

[0002] In the field of cloud computing, the storage of enterprise digital management data typically relies on distributed architectures and static sharding strategies. For example, data is divided into encrypted units according to fixed rules and stored in designated areas. In related technologies, the selection of cloud storage paths is often based on preset regional division rules, such as allocating fixed partitions based on data type or storage capacity, and processing the sharded data using a unified encryption algorithm. In addition, existing load balancing solutions often use real-time connection number monitoring and threshold triggering mechanisms. For example, when the number of concurrent connections in a partition exceeds a preset threshold, some connections are migrated to a lower-load area.

[0003] However, the above method has significant flaws in practical applications. The first is the rigidity of the static sharding strategy. The fixed encryption sharding rules cannot adapt to the dynamically changing regional access frequency and security requirements. For example, if high-security data is allocated to a low-access frequency area, it will cause a surge in access latency; and the storage resources in hot spots may be quickly exhausted due to the fixed shard size, requiring frequent expansion. Then, there is the problem of separating security and efficiency. The encryption level and storage path selection are usually decided independently. For example, high-encryption data uses a complex algorithm but the storage path is not optimized, making it difficult to balance encryption overhead and access efficiency. In addition, there are lag issues with load balancing and elastic expansion. The threshold trigger mechanism based on the current number of connections cannot predict traffic trends. For example, sudden high-concurrency requests may cause the system to time out before the expansion is completed. In addition, the expansion process is not linked to the storage path priority, resulting in low migration efficiency.

[0004] These shortcomings make it difficult for existing technologies to achieve efficient storage resource utilization and dynamic security management. This results in low resource utilization, large fluctuations in response latency, and high hardware maintenance costs. This is especially true when processing highly concurrent and multimodal enterprise management data, severely limiting system stability and scalability. Therefore, a method for collaboratively optimizing resource allocation and security efficiency is urgently needed. Summary of the Invention

[0005] The purpose of the present invention is to provide a digital enterprise management data storage method and system based on cloud computing. The embodiment of the present invention is implemented as follows:

[0006] In a first aspect, the present invention provides a digital enterprise management data storage method based on cloud computing, the method comprising: obtaining a real-time management data set in an enterprise data stream, and generating a virtualized storage node based on structured data blocks and unstructured data blocks in the real-time management data set; the virtualized storage node includes encrypted shard indexes associated with different regions, and each encrypted shard index corresponds to a dynamic storage path; according to a preset storage path optimization model, determining a target encrypted shard index that matches the virtualized storage node; the storage path optimization model is generated by training based on regional access frequency and encryption level characteristics in historical storage logs; based on the dynamic storage path corresponding to the target encrypted shard index, dividing the structured data blocks and unstructured data blocks into encrypted data units distributed across regions, and mapping the encrypted data units to corresponding cloud storage partitions; dynamically allocating storage resources of the cloud storage partitions through a load balancing algorithm, and adjusting the priority weight of the dynamic storage path according to real-time traffic characteristics; when an access request for the encrypted data unit is detected, reallocating the number of concurrent connections of the cloud storage partitions according to the updated priority weight and the storage path optimization model to achieve elastic expansion.

[0007] In a second aspect, the present invention provides a data storage system comprising: one or more processors; a memory; and one or more computer programs; wherein the one or more computer programs are stored in the memory and configured to be executed by the one or more processors, and when the one or more computer programs are executed by the processors, the method described above is implemented.

[0008] The digital enterprise management data storage method based on cloud computing provided by the present invention realizes efficient utilization and security management of storage resources by dynamically generating virtualized storage nodes and binding cross-region encrypted sharding indexes, and combining storage path optimization models to intelligently segment and map the structured and unstructured content of real-time management data. Specifically, virtualized storage nodes can automatically select the optimal storage path according to real-time traffic characteristics by associating dynamic encrypted sharding indexes with regional characteristics, solving the resource waste and security vulnerability problems caused by traditional static sharding strategies; the storage path optimization model is trained and generated based on regional access frequency and encryption level characteristics in historical storage logs, which can dynamically balance data access efficiency and security requirements, avoiding access delay bottlenecks caused by fixed paths for high-security data; by segmenting structured and unstructured data blocks into encrypted data units distributed across regions, and combining load balancing algorithms to dynamically adjust the priority weights of storage paths, it can achieve elastic expansion of storage resources in high-concurrency scenarios, significantly improving system response speed and stability. In addition, the dynamic binding mechanism of the encrypted shard index and the real-time mapping relationship of the cloud storage partition form a closed-loop management, ensuring that the data storage location always matches the regional security level and access load status, thereby reducing hardware maintenance costs and enhancing the system's adaptability to sudden traffic fluctuations. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments of the present invention.

[0010] Figure 1 This is a flowchart of a digital enterprise management data storage method based on cloud computing provided by an embodiment of the present invention.

[0011] Figure 2 This is a schematic diagram of the composition of a storage system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0012] The following describes the embodiments of the present invention in conjunction with the accompanying drawings. The terms used in the implementation methods of the embodiments of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention.

[0013] In the embodiment of the present invention, the execution subject of the digital enterprise management data storage method based on cloud computing is a data storage system, such as a storage server. Figure 1 As shown, the method includes:

[0014] Step S100: Obtain a real-time management data set in the enterprise data stream, and generate a virtualized storage node based on the structured data blocks and unstructured data blocks in the real-time management data set; the virtualized storage node contains encrypted shard indexes associated with different areas, and each encrypted shard index corresponds to a dynamic storage path.

[0015] A real-time management dataset refers to a collection of data that is continuously collected from the enterprise's operations and contains real-time dynamic update features. Its data sources include but are not limited to business system logs, sensor monitoring data streams, user interaction behavior records, and cross-platform data transmission interfaces. Structured data blocks are data units with a predefined data model and can be parsed using a fixed field format, such as relational data table records in a database, key-value pairs in JSON or XML format; unstructured data blocks are data units without a fixed format or predefined model, such as image files, audio and video streams, natural language text, or binary log files. The process of generating a virtualized storage node includes: separating the structured and unstructured data blocks of the real-time management dataset through a data classification engine, allocating initial storage resource tags based on data block type characteristics (such as data volume, format identifier, and access frequency), and binding storage resource tags to geographically distributed or logically partitioned cloud storage areas according to preset regional association policies. During this process, an encrypted shard index is a unique identifier generated for each virtualized storage node. It indicates the logical distribution of the encrypted shards associated with that node within the cloud storage region. For example, a data shard index for the North American region is associated with the US West Coast cloud storage partition, and a data shard index for the Asia Pacific region is associated with the Singapore cloud storage partition. Each encrypted shard index corresponds to a dynamic storage path. Dynamic storage paths are data transmission routing rules that are dynamically adjusted based on real-time network status, regional load conditions, or security policies. For example, if a storage node in a certain region is detected to be overloaded, the dynamic storage path can reroute the data shard to a backup storage node in a less-loaded region.

[0016] Specifically, the operation of generating a virtualized storage node is achieved, for example, through the following process: first, the format of the data blocks in the real-time management data set is parsed, and the structured data blocks and unstructured data blocks are distinguished based on the metadata characteristics of the data blocks (such as file header identifiers, content type tags); second, an independent data fingerprint (such as a hash value or digital summary) is generated for each data block, and the data fingerprint is matched with the encrypted shard index template in the storage area policy library to determine the regional association relationship to which the data block should be bound; finally, a virtualized storage node instance is created based on the matching result, and the mapping relationship between the encrypted shard index and the dynamic storage path is written into the node metadata. For example, when it is detected that the structured data block is a database transaction log, its encrypted shard index will be marked as "high priority - European region", and the corresponding dynamic storage path will point to the high-performance storage cluster of the Frankfurt cloud storage partition; and unstructured data blocks such as surveillance video streams may be marked as "medium priority - Asia Pacific region", and the dynamic storage path will point to the object storage service of the Tokyo cloud storage partition.

[0017] Step S200: Determine the target encrypted shard index that matches the virtualized storage node according to a preset storage path optimization model; the storage path optimization model is generated based on regional access frequency and encryption level characteristics in historical storage logs.

[0018] The storage path optimization model is a predictive model trained using a machine learning algorithm. Its input includes multiple regional association features recorded in historical storage logs, and its output is the optimal encryption shard index and dynamic storage path configuration parameters adapted to the virtualized storage node. Regional access frequency refers to the statistical value of the frequency of data access requests received by a specific cloud storage partition within a historical time period, such as the number of access requests received by the US East cloud storage partition during the daily peak period. The encryption level feature is used to describe the security level that data shards must meet during storage. For example, shards using the AES-256 encryption algorithm must be associated with a high-security storage path, while shards using the TLS1.2 transport protocol must be bound to a low-latency storage path. The process of training the storage path optimization model includes: extracting time series data of regional access frequency, hierarchical labels of encryption level features, measured values ​​of storage latency parameters, and statistical distribution data of concurrent connection peaks from historical storage logs, and constructing association rules between regional features and storage path performance through a supervised learning algorithm.

[0019] In this process, the following steps are required to determine the target encrypted shard index: First, the metadata of the virtualized storage node (such as data block type, region association label, encryption level requirement) is input into the storage path optimization model; second, the model calculates the matching scores of different encrypted shard indexes by analyzing the periodic characteristics of regional access frequency (such as the difference in access patterns between weekdays and holidays) and the compatibility constraints of encryption level characteristics (such as the need to avoid cross-region transmission of high-security level shards); finally, the encrypted shard index with the highest matching score is selected as the target encrypted shard index. For example, if the encryption level characteristic associated with a virtualized storage node is "highest security level" and historical storage logs show that the access frequency of the European region during the target time period is low, the model may point the target encrypted shard index to a dedicated encrypted storage area of ​​the Zurich cloud storage partition and assign it a dynamic storage path that only allows intranet transmission to reduce exposure risks.

[0020] As an implementation method, the training process of the storage path optimization model includes the following steps:

[0021] Step S201: extract multiple region-related features from historical storage logs; the region-related features include region access frequency, encryption level features, storage delay parameters, and concurrent connection peaks.

[0022] Historical storage logs are data sets recorded by cloud storage systems over historical time periods that contain storage operation metadata and performance metrics. These data fields include storage request timestamps, target partition identifiers, encryption algorithm types, transport protocol versions, response latency in milliseconds, and concurrent connection statistics. Regional features are extracted from historical storage logs and are used to characterize the behavior patterns of different cloud storage partitions. These include: regional access frequency (e.g., the average number of daily accesses to the US East partition between 8:00 AM and 10:00 AM), encryption level characteristics (e.g., the percentage of shards in the Asia-Pacific partition using the AES-256 encryption algorithm), storage latency parameters (e.g., the 95th percentile latency of data write operations for the Europe-Central partition), and peak concurrent connections (e.g., the number of concurrent connections to the Singapore partition during periods of historical peak load). Extracting regional features requires a log parsing engine, which generates structured feature data from raw logs based on predefined feature extraction rules (e.g., time window sliding average calculations, encryption level classification statistics). For example, for regional access frequency, the engine can count the number of access requests received by the West Coast partition every hour in the past 30 days and generate a time series dataset; for encryption level features, the engine counts the ratio of the number of shards using the TLS1.3 protocol to the total number of shards in the Tokyo partition and maps it to a normalized feature value in the range of 0-1.

[0023] Step S202: Generate an initial training set based on the regional visit frequency and encryption level characteristics, and input the initial training set into the initial path optimization network for the first round of training.

[0024] The initial training set refers to the basic dataset used to train the storage path optimization model. It consists of time series data on regional access frequency and classification labels for encryption level features. The process of generating the initial training set includes: standardizing the regional access frequency data (such as Z-score normalization), converting the encryption level features into one-hot encoding (such as encoding "high encryption level" as [1,0,0] and "medium encryption level" as [0,1,0]), and aligning the two along the time dimension before merging them into a multidimensional feature vector. The initial path optimization network is an untrained neural network architecture. The number of nodes in its input layer matches the feature dimensions of the initial training set, the hidden layer uses a fully connected structure, and the number of nodes in the output layer matches the configuration parameter dimensions of the dynamic storage path. The goal of the first round of training is to enable the network to initially learn how regional access frequency and encryption level features influence storage path selection through supervised learning. For example, when the input feature is "access frequency of the East US region 0.8, encryption level feature [1,0,0]", the network needs to output the priority weight (such as 0.9) pointing to the high-security storage path and the corresponding resource allocation ratio (such as 70% of storage resources are allocated to this path).

[0025] Step S203: During the first round of training, the node weight distribution of the initial path optimization network is determined by a cross-validation algorithm, and an intermediate path optimization model is constructed according to the node weight distribution.

[0026] The cross-validation algorithm is a data partitioning strategy used to evaluate model generalization. Its core operation involves partitioning the initial training set into multiple mutually exclusive subsets (e.g., five subsets). One of these subsets is then selected as the validation set, and the remaining subsets are used as the training set. The node weight distribution is optimized through multiple iterations of training and validation. During the first round of training, the node weight distribution of the initial path optimization network is gradually adjusted through forward propagation and backpropagation. During the forward propagation phase, input features are passed through hidden layer activation functions (e.g., ReLU) to generate predicted outputs. During the validation phase, node weights are updated using a gradient descent algorithm by calculating the mean squared error between the predicted outputs and the actual path configuration parameters (e.g., the prediction error of path priority weights). When the path prediction error meets a preset convergence condition (e.g., the error change rate is less than 1% over three consecutive iterations), the current node weight distribution is determined to be optimal, and an intermediate path optimization model is constructed based on this distribution. For example, if cross-validation shows that the high-visit frequency feature in the Western United States is most strongly correlated with the weight of the third hidden layer node (weight 0.85), this node will be given higher importance in the intermediate model to capture the path selection patterns of the frequently visited region.

[0027] As an implementation manner, in step S203, determining the node weight distribution of the initial path optimization network by a cross-validation algorithm may specifically include:

[0028] Step S2031: Divide the initial training set into multiple training subsets and validation subsets, and perform forward propagation training on the initial path optimization network based on the training subsets.

[0029] The operation of dividing the training subset and validation subset must follow the K-fold cross-validation rule. For example, the initial training set is evenly divided into 5 subsets (subsets 1 to 5), and subset 1 is selected as the validation set, and subsets 2-5 are selected as the training sets. After completing a round of training, the validation set is replaced with subset 2. This process is repeated until all subsets are used as validation sets. During the forward propagation training process, input features are loaded into the network input layer in batches from the training subset. After linear combination and nonlinear activation (such as the Sigmoid function) of the hidden layer nodes, predicted values ​​of priority weights and resource allocation ratios are generated. For example, when the training subset contains 1,000 historical access records in the East US region, the forward propagation will calculate the path priority prediction value corresponding to each record one by one and compare it with the true label to initialize the weight adjustment direction.

[0030] Step S2032: After each forward propagation training, the path prediction error of the initial path optimization network is calculated using the validation subset, and the node weight distribution is adjusted according to the path prediction error.

[0031] Path prediction error is a measure of the difference between the predicted value output by the model on the validation subset and the actual configuration parameters. Mean squared error (MSE) or mean absolute error (MAE) is typically used as an evaluation metric. For example, if the model predicts a priority weight of 0.7 for a record in the validation subset, while the actual value is 0.8, the single-point error is 0.1. The overall path prediction error is obtained by averaging the errors across all validation samples. When adjusting the node weight distribution, the stochastic gradient descent (SGD) algorithm is used to update the weight matrix based on the direction of the error gradient. For example, if the second node in the hidden layer contributes the most to the error (gradient value of 0.15), its weight will be significantly reduced to reduce subsequent prediction bias.

[0032] Step S2033: When the path prediction error satisfies a preset convergence condition, determining that the node weight distribution of the initial path optimization network is the optimal weight distribution.

[0033] The preset convergence condition is typically set as the error rate of change or an absolute error threshold. For example, it requires that the overall path prediction error fluctuate by less than 0.5% over three consecutive iterations, or that the absolute value of the error drops below 0.02. When these conditions are met, the model has fully learned the distribution of the training data, and the node weight distribution no longer changes significantly with continued training. At this point, the optimal weight distribution is solidified. For example, the weight of the first node in the hidden layer is fixed at 0.68, and the weight of the third node is fixed at 0.73, forming a stable feature mapping relationship.

[0034] Step S2034: Generate a weight update function corresponding to the encryption level feature based on the optimal weight distribution and the region access frequency in the initial training set.

[0035] The weight update function is a mathematical expression used during the model inference phase to dynamically adjust node weights based on real-time region access frequency to adapt to the path selection requirements of different encryption level characteristics. For example, if the optimal weight distribution shows that high encryption level characteristics (such as [1,0,0]) are strongly correlated with the fifth node in the hidden layer (weight 0.85), the weight update function can be defined as follows: when the encryption level is detected to be high, the weight of the fifth node is increased by 10%; when the encryption level is low, the weight of the fifth node is decreased by 5%. This function is encoded into the model inference logic to ensure that the path optimization strategy can adapt to changes in encryption level.

[0036] Step S2035: iteratively update the node weights of the initial path optimization network through the weight update function until the path prediction error reaches the minimum threshold.

[0037] The iterative update process requires repeatedly loading the initial training set data into the training environment, applying the weight update function to adjust node weights, and re-evaluating the path prediction error. For example, if the initial error is 0.1, after three iterative updates, the error drops to 0.05, and the weight of the fifth node is adjusted from 0.85 to 0.89, indicating that the function effectively improves prediction accuracy in high-encryption scenarios. When the error reaches a minimum threshold (e.g., 0.03), the iteration is terminated and the final weight distribution is imported into the intermediate path optimization model.

[0038] Step S204: Input the storage delay parameters and the concurrent connection peak into the intermediate path optimization model to generate a path optimization parameter set corresponding to the cloud storage partition; the path optimization parameter set includes the priority weight of the dynamic storage path, the update cycle of the encrypted shard index, and the storage resource allocation ratio.

[0039] Storage latency parameters are measured metrics extracted from historical storage logs that reflect the response speed of each partition (e.g., the average read latency for the Singapore partition is 120ms). Peak concurrent connections refer to the number of concurrent connections for a partition during historical peak load periods (e.g., the Frankfurt partition's peak concurrent connections are 5000). When these parameters are fed into the intermediate path optimization model, the model performs a nonlinear transformation on the latency and concurrency characteristics through hidden layer nodes to generate a set of path optimization parameters. For example, when given "storage latency parameters of 150ms and peak concurrent connections of 4500," the model might output a dynamic storage path priority weight of 0.75 (indicating the recommended priority for this path under the current load), an encrypted shard index update period of 24 hours (indicating that shard index bindings are reevaluated every 24 hours), and a storage resource allocation ratio of 60% (i.e., 60% of storage resources are allocated to the partition corresponding to this path). This process relies on the model's generalization capabilities to ensure that it can generate reasonable parameter configurations for unseen latency and concurrency combinations.

[0040] Step S205: Perform back propagation optimization on the intermediate path optimization model according to the path optimization parameter set to generate a storage path optimization model; wherein the output of the storage path optimization model includes the priority weight and storage resource allocation ratio of the dynamic storage path corresponding to the target encryption shard index.

[0041] Backpropagation optimization involves using the target output in the path optimization parameter set (e.g., priority weight 0.8, resource allocation ratio 65%) as a supervisory signal. The model calculates the loss function (e.g., cross-entropy loss) between the predicted output and the target value, and then propagates the gradient back through the network to adjust the hidden layer weight parameters. During this phase, storage latency parameters are used to calculate the latency loss (e.g., the difference between the predicted and measured latency), and the peak concurrent connections are used to constrain the resource allocation ratio calculation range (e.g., when the peak concurrent connections exceed 4000, the resource allocation ratio must not fall below 50%). When the latency loss reaches a preset optimization threshold (e.g., the loss drops below 0.05), the hidden layer weight parameters are fixed to the final weights. The output layer parameters of the storage path optimization model are generated based on the final weights and the output layer activation function (e.g., softmax). For example, after optimization, the model might strongly associate the priority weight calculation logic for the US West partition with the fourth hidden layer node (weight 0.92), ensuring that the priority of this path is automatically increased in high-concurrency scenarios.

[0042] As an implementation method, step S205, performing back propagation optimization on the intermediate path optimization model according to the path optimization parameter set, may specifically include:

[0043] Step S2051: input the priority weight and storage resource allocation ratio in the path optimization parameter set into the hidden layer of the intermediate path optimization model to generate hidden layer output features.

[0044] Hidden layer output features are intermediate representations generated by nonlinearly transforming the input parameters through the hidden layer nodes. Their dimensionality is the same as the number of hidden layer nodes. For example, when the input priority weight is 0.8 and the storage resource allocation ratio is 65%, the hidden layer may generate a 128-dimensional feature vector, where each dimension corresponds to the abstract characteristics of a different path optimization mode (for example, the 12th dimension represents "high priority-low latency path preference" and the 45th dimension represents "balanced resource allocation tendency").

[0045] Step S2052: Calculate the delay loss value of the intermediate path optimization model based on the hidden layer output features and the stored delay parameters.

[0046] The latency penalty is used to quantify the deviation of the model's predicted storage path from its actual latency performance. It is calculated as the mean squared error between the predicted latency (calculated based on the hidden layer output features) and the measured storage latency parameters. For example, if the hidden layer output features indicate a path with a predicted latency of 130ms, and the actual latency is 145ms, the single-point latency penalty is (145-130)^2 = 225. The overall latency penalty is then calculated by averaging the latency values ​​across all samples.

[0047] Step S2053: Back-propagating the delay loss value through the gradient descent algorithm to adjust the hidden layer weight parameters of the intermediate path optimization model.

[0048] The gradient descent algorithm determines the direction and magnitude of weight adjustments by calculating the partial derivative (gradient) of the delayed loss with respect to the hidden layer weights. For example, if the gradient calculation shows that increasing the weight of the third node in the hidden layer by 0.1 will reduce the loss by 5%, the optimizer will gradually increase that weight at a learning rate (e.g., 0.001). This process requires multiple iterations until the improvement in loss from the weight adjustment stabilizes.

[0049] Step S2054: When the delay loss value reaches a preset optimization threshold, the hidden layer weight parameters of the intermediate path optimization model are fixed as final weight parameters.

[0050] The preset optimization threshold is typically set to an acceptable performance deviation range, for example, a latency loss value of less than 10ms mean absolute error. When the loss value drops below the threshold (e.g., 8ms), the hidden layer weight parameters are no longer updated and are marked as final. For example, the weight of the seventh node in the hidden layer is fixed at 0.72, indicating that its contribution to low-latency path selection has reached its optimal state.

[0051] Step S2055: Generate output layer parameters of the storage path optimization model based on the final weight parameters and the concurrent connection peak value, so that the output of the storage path optimization model matches the priority weight and storage resource allocation ratio of the dynamic storage path.

[0052] The generation of output layer parameters requires combining the final weight parameters with the constraints of the peak concurrent connections. For example, if the peak concurrent connections are 5000, the output layer activation function must map the hidden layer features to a priority weight range of [0, 1] and a resource allocation ratio range of [0%, 100%], while ensuring that the resource allocation ratio does not fall below a preset lower limit (e.g., 30%) at high concurrency. Through linear regression or the Softmax function, the model ultimately outputs parameters that fully match the dynamic storage path configuration, such as a priority weight of 0.78 and a resource allocation ratio of 58%.

[0053] Step S300: Based on the dynamic storage path corresponding to the target encryption shard index, the structured data blocks and the unstructured data blocks are divided into encrypted data units distributed across regions, and the encrypted data units are mapped to the corresponding cloud storage partitions.

[0054] A cross-regionally distributed encrypted data unit refers to a data entity that is created by splitting the original data block into multiple sub-blocks according to a preset sharding rule and then distributing them across cloud storage partitions in different geographical or logical locations through encryption algorithms and region binding strategies. The process of splitting structured and unstructured data blocks must follow the dynamic storage path rules defined in the target encrypted shard index. For example, for structured data blocks, if their dynamic storage path requires a shard size of no more than 1MB and storage across at least three regions, the data must be split according to a fixed block size (e.g., 1MB) and a region identifier must be attached to each sub-block. For unstructured data blocks, if the dynamic storage path requires the shard size to be dynamically adjusted to accommodate network bandwidth fluctuations, the shard boundaries must be dynamically calculated based on the real-time transmission rate. The generation of encrypted data units is implemented through a shard encryption protocol, which includes a data compression algorithm (e.g., ZIP or Snappy), an encryption algorithm type (e.g., AES or RSA), and shard metadata encapsulation rules (e.g., combining the shard sequence number, region identifier, and checksum into a shard header).

[0055] Specifically, the operation of mapping encrypted data units to cloud storage partitions includes the following stages: first, assigning an initial target partition to each encrypted data unit based on the partition priority weight defined in the dynamic storage path (e.g., the weight of the US West Coast partition is 0.8, and the weight of the Singapore partition is 0.6); second, transmitting the encrypted data unit to the target partition via a cross-region transmission protocol (e.g., HTTP / 3 or a dedicated tunneling protocol), and monitoring the partition availability status in real time during the transmission process; finally, recording the shard index mapping relationship of the encrypted data unit in the storage service of the target partition to ensure that subsequent access requests can quickly locate the shard location through the index. For example, an unstructured data block (e.g., a video file) may be divided into 100 encrypted data units, of which the first 30 units are mapped to the Tokyo cloud storage partition, the middle 40 units are mapped to the Sydney cloud storage partition, and the remaining 30 units are mapped to the Mumbai cloud storage partition, and the encrypted shard index of each unit contains the access key and geographic location code of the corresponding partition.

[0056] As an embodiment, step S300, based on the dynamic storage path corresponding to the target encryption shard index, splits the structured data block and the unstructured data block into encrypted data units distributed across regions, which may specifically include:

[0057] Step S310: Determine the number of encrypted shards for each cloud storage partition based on the regional distribution characteristics of the dynamic storage paths.

[0058] The regional distribution characteristics of a dynamic storage path refer to a set of attributes, parsed from the target encrypted shard index, that describe the geographic location or logical grouping of cloud storage partitions. These include, but are not limited to, the partition's physical location code (e.g., longitude and latitude coordinates), logical hierarchy labels (e.g., core storage zone or edge cache zone), and storage capacity limits (e.g., the US West partition supports up to 10PB of storage). Determining the number of encrypted shards requires integrating the storage capacity and load balancing strategy within the regional distribution characteristics. First, the theoretical maximum number of shards is calculated based on the partition's storage capacity limit (e.g., a 10TB partition with a capacity of 1GB per shard can accommodate 10,000 shards). Second, the actual number of shards is dynamically adjusted based on the shard redundancy factor defined in the load balancing strategy (e.g., each data unit must be replicated in at least three partitions). For example, if the dynamic storage path requires the US West partition to store the primary replica, the Tokyo partition to store the primary backup replica, and the Frankfurt partition to store the secondary backup replica, the number of encrypted shards should be allocated based on the total number of data units multiplied by the replication factor of 3, ensuring that the number of shards in each partition matches its capacity.

[0059] Step S320: Generate a shard encryption protocol corresponding to each cloud storage partition based on the number of encrypted shards and the data type of the structured data block.

[0060] A shard encryption protocol is a set of data sharding and encryption rules customized for a specific cloud storage partition. Its core elements include a shard size range (e.g., shard size for unstructured data blocks is limited to 64MB to 256MB), an encryption algorithm type (e.g., AES-256 or ChaCha20), a compression scheme (e.g., ZIP or Snappy), and a metadata encapsulation format (e.g., a shard header containing a shard sequence number, region identifier, and checksum). When generating a shard encryption protocol, the data compression scheme must first be determined based on the data type of the structured data block. For highly structured database transaction logs (e.g., MySQL binlog), the low-latency LZ4 compression algorithm is used; for semi-structured JSON data streams, the Zstandard algorithm, which balances compression rate and speed, is used. Subsequently, the shard size range for unstructured data blocks is divided based on the number of encrypted shards. For example, when the number of encrypted shards is 1000, the initial shard size for an unstructured video file can be set to 100MB, with dynamic adjustment allowed through a size fluctuation factor (e.g., ±20%) in the shard encryption protocol.

[0061] As an embodiment, step S320 generates a shard encryption protocol corresponding to each cloud storage partition based on the number of encrypted shards and the data type of the structured data block, which may specifically include:

[0062] Step S321: Determine a data compression mode according to the data type of the structured data block, and select an encryption algorithm type that matches the data compression mode from a preset encryption algorithm library.

[0063] The data type of a structured data block is defined by a format identifier in its metadata (e.g., "type=sql" represents a relational database log, or "type=json" represents semi-structured JSON data). When determining the data compression mode, the system queries a preset compression strategy table based on the data type. For example, "sql" type data uses the LZ4 compression algorithm by default for low-latency processing, "json" type data uses the Zstandard algorithm to balance compression ratio and speed, and "csv" type data uses gzip compression to maximize storage efficiency. The encryption algorithm type must be compatible with the compression mode. For example, LZ4-compressed data is suitable for high-strength AES-256 encryption due to its smaller size. However, gzip-compressed data may require the ChaCha20 algorithm, which has lower computational overhead, due to its longer processing time. When selecting an algorithm from the encryption algorithm library, hardware acceleration support for the target partition (e.g., whether Intel QAT supports the AES-NI instruction set) must also be considered.

[0064] Step S322: Divide the fragment size range of the unstructured data block based on the number of encrypted fragments, and generate an initial fragmentation rule according to the fragment size range.

[0065] When dividing the shard size range, the system calculates the theoretical optimal shard size based on the number of encrypted shards and the average network bandwidth of the target partition. For example, if the number of encrypted shards is 1000 and the target partition bandwidth is 1Gbps, the shard size range can be initialized to 1MB to 10MB to ensure that the shard transmission time (1MB / 125MBps=8ms) is well below the TCP timeout threshold. When generating the initial sharding rules, it is necessary to define the shard boundary determination logic (such as cutting by fixed size or splitting by content sensitivity), the allowed range of shard overlap (such as adjacent shards can overlap by 5% to prevent data loss), and the shard metadata format (such as the shard header contains a version number and a shard checksum). For example, the initial sharding rules for unstructured data blocks may stipulate that the size of each shard is 5MB±10%, the overlapping area between shards does not exceed 512KB, and the shard header contains a SHA-256 checksum.

[0066] Step S323: Adapt the encryption strength of the fragment size range in the initial fragmentation rule by the encryption algorithm type to generate an intermediate fragmentation rule.

[0067] Encryption strength adaptation refers to adjusting the shard size range based on the computational overhead and security requirements of the selected encryption algorithm. For example, the AES-256 encryption algorithm requires shard sizes to be integer multiples of 16 bytes due to its block cipher nature. Therefore, the initial sharding rule of 5MB±10% needs to be adjusted to 4.992MB (i.e., 4992KB, divisible by 16). The ChaCha20 algorithm, on the other hand, does not have strict alignment requirements for shard size due to its stream cipher nature, so the initial sharding rule can remain unchanged. When generating intermediate sharding rules, the system encodes the encryption algorithm constraints (such as shard padding method and initialization vector length) into the shard metadata and updates the legal range of shard sizes (e.g., the adjusted size range is 4.992MB to 10.016MB).

[0068] Step S324: According to the encryption strength adaptation result in the intermediate fragmentation rule, the unstructured data block is divided into multiple initial encryption fragments.

[0069] When performing sharding, the system cuts unstructured data blocks according to the size range and encryption constraints in the intermediate sharding rules. For example, a 10GB video file is split into 2048 initial encrypted shards with a fixed size of 4.992MB (10*1024 / 4.992≈2048). Each shard is padded to a 16-byte boundary to meet AES encryption requirements, and a shard header containing an initialization vector (IV) and a timestamp is appended. During this process, the sharding tool needs to calculate the shard hash value (such as SHA-256) in real time and write metadata to facilitate subsequent verification of shard integrity.

[0070] Step S325: Perform compression mode verification on the initial encrypted slice, and adjust the matching relationship between the data compression mode and the slice size range according to the verification result.

[0071] The compression mode check is designed to verify the compatibility of the compression algorithm and the shard size. For example, if the compression rate of a shard compressed with Snappy is less than 20% due to its small size (such as 1MB), the shard size range needs to be increased to more than 2MB to improve compression efficiency; conversely, if the compression time of a single shard exceeds 50ms due to the size of a shard compressed with Zstandard (such as 10MB), the size range needs to be lowered to 5MB to reduce latency. The check result is comprehensively evaluated through three indicators: compression rate, compression time, and shard volume fluctuation rate: if the indicator does not reach the preset threshold (such as the compression rate needs to be ≥30%), the system will automatically trigger the re-matching process of the compression mode and shard size. For example, when the compression rate of LZ4 compression is only 25% for a 2MB shard, the system may switch to the Zstandard algorithm or increase the shard size to 4MB.

[0072] Step S326: Generate a final sharding rule based on the adjusted matching relationship, and bind the final sharding rule to the encryption algorithm type to generate a sharding encryption protocol.

[0073] The final sharding rules integrate the complete configuration of compression mode, shard size range, encryption algorithm parameters and shard metadata format. For example, the adjusted rules may stipulate that the shard size is fixed at 4MB, using Zstandard compression (compression level 5) and AES-256-GCM encryption (IV length 12 bytes), and the shard header contains the shard sequence number, region identifier, volume before and after compression, and SHA-256 checksum. When generating the shard encryption protocol, the system binds the final sharding rules to the encryption algorithm type (such as "AES-256-GCM") and writes it to the protocol configuration file (such as XML or JSON format) for sharding tools and storage services to call. For example, the protocol file may contain the following entries:

[0074] {

[0075] "Fragment size range": "4MB±5%",

[0076] "Compression algorithm": "Zstandard-level5",

[0077] "Encryption algorithm": "AES-256-GCM",

[0078] "Metadata fields": ["Shard ID", "Region Code", "Size Before Compression", "SHA-256"]

[0079] }

[0080] As an implementation method, the generation process of the sharding encryption protocol further includes dynamically adjusting the final sharding rules through a sharding optimization model, which may specifically include:

[0081] Step S3261: Extract shard performance characteristics associated with encryption algorithm type, data compression mode, and shard size range from historical shard logs.

[0082] The historical sharding log records detailed information about past data sharding and encryption processes, including the encryption algorithm used, the effectiveness of the data compression scheme, and the appropriateness of the shard size range. The encryption algorithm type is selected from a preset encryption algorithm library in step S321. The data compression scheme is determined based on the data type of the structured data block. The shard size range is determined through multiple adjustments in the previous steps. Sharding performance characteristics are a series of metrics related to the encryption algorithm type, data compression scheme, and shard size range, such as encryption time, compression ratio, and data integrity.

[0083] The goal of extracting shard performance characteristics associated with encryption algorithm types, data compression modes, and shard size ranges from historical shard logs is to gain practical experience and data. By analyzing historical shard logs, we can understand shard performance under different combinations of encryption algorithm types, data compression modes, and shard size ranges. For example, certain encryption algorithms may have higher encryption efficiency within specific shard size ranges, while certain data compression modes may achieve better compression ratios when processing specific data types. Extracting these shard performance characteristics provides data support for subsequent dynamic adjustments using the shard optimization model.

[0084] For example, historical shard logs record key performance indicators of past shard operations, including shard compression time (such as the average time of 12ms for Zstandard to compress a 4MB shard), encryption throughput (such as AES-256-GCM encryption rate of 550MB / s), shard transmission success rate (such as 99.98%), and storage partition response latency (such as the average write latency of 45ms for the US West Coast partition). When extracting shard performance characteristics, the system aggregates log data based on three dimensions: encryption algorithm type, data compression mode, and shard size range to generate a feature vector. For example, for the combination of "AES-256-GCM+Zstandard+4MB", the feature vector may contain [compression rate 35%, encryption rate 550MB / s, shard retransmission rate 0.02%].

[0085] Step S3262: Input the sharding performance characteristics into the sharding optimization model to generate sharding adjustment parameters corresponding to the final sharding rules.

[0086] The shard optimization model is a machine learning-based regression model whose input is a shard performance feature vector and whose output is a shard adjustment parameter (such as a recommended shard size fluctuation range of ±8% and a compression level adjustment of 4). The model predicts the impact of adjustment parameters on shard efficiency by analyzing performance bottlenecks in historical data (such as high compression levels causing CPU overload). For example, when the input features show that the transmission success rate of 4MB shards in the Tokyo partition has dropped to 97%, the model may output a shard size adjustment parameter of "reducing to 3.5MB" to adapt to the network fluctuation characteristics of that partition.

[0087] Step S3263: Based on the fragment overlap threshold in the fragment adjustment parameter, perform overlapping area detection on the boundaries of the initial encrypted fragments.

[0088] The shard overlap threshold is used to control the data duplication ratio of adjacent shards (such as allowing a maximum overlap of 5%) to prevent data loss or verification failure. When detecting overlapping areas, the system compares the start and end offsets of adjacent shards and calculates the percentage of overlapping bytes. For example, if the end offset of shard N is 1000000 bytes and the start offset of shard N+1 is 999500 bytes, the overlapping area is 500 bytes, accounting for 0.49% of the shard size (1MB), which is below the 5% threshold and is considered compliant.

[0089] Step S3264: When it is detected that the boundary overlap area of ​​the initial encrypted fragments exceeds the fragment overlap threshold, the fragment size range is recalculated and the final fragmentation rule is updated.

[0090] If the overlap in a batch of shards reaches 6%, the system will recalculate the shard size based on the sharding adjustment parameters. For example, if the original shard size is 4MB and the overlap exceeds the limit, network packet loss may lead to inaccurate sharding. In this case, the size will be reduced to 3.8MB and the number of shards will be increased. The size range in the final sharding rule will also be updated to "3.8MB ± 3%." The updated rule will be synchronized with the sharding encryption protocol to ensure that subsequent sharding operations meet the new constraints.

[0091] Step S3265: Perform secondary sharding on the unstructured data block based on the updated final sharding rule to generate an optimized encrypted sharding set.

[0092] The secondary sharding process re-segments unstructured data blocks using updated sharding rules. For example, a 10GB video file is split into 2632 shards of 3.8MB each (10*1024 / 3.8≈2632). Each shard is compressed using Zstandard and encrypted with AES to generate optimized encrypted shards. During this process, the system records the actual size, compression ratio, and encryption time of each shard and writes it into a historical sharding log for subsequent model training.

[0093] Step S3266: Based on the cross-combination result of the optimized encryption shard set and the structured data block, the binding relationship between the shard size range and the encryption algorithm type in the shard encryption protocol is corrected.

[0094] When revising the sharding encryption protocol, parameter bindings must be adjusted based on the optimized sharding performance data. For example, if Zstandard compression level 4 improves the compression ratio to 38% for 3.8MB shards after secondary sharding, the compression level parameter in the protocol will be adjusted from "level 5" to "level 4." Simultaneously, the shard size range will be updated to "3.8MB ± 3%," while the encryption algorithm type will remain unchanged at "AES-256-GCM." The revised protocol will serve as the new baseline configuration for all subsequent sharding operations until the optimization process is triggered again.

[0095] Step S330: Slice and compress the unstructured data block using a slicing encryption protocol to generate multiple compressed encrypted sub-blocks.

[0096] When performing shard compression, the compression mode and encryption algorithm in the shard encryption protocol are sequentially applied to unstructured data blocks: first, the original data is split into multiple sub-blocks according to the shard size range (for example, splitting a 10GB video file into 100 100MB sub-blocks); second, each sub-block is compressed using a specified compression algorithm (for example, using Snappy compression to reduce the sub-block size to 70MB); finally, the compressed sub-block is encrypted using an encryption algorithm (for example, using AES-256-GCM mode encryption and appending an authentication tag). During this process, the shard encryption protocol requires that each compressed and encrypted sub-block be accompanied by shard metadata, including the shard sequence number (for data reassembly), the region identifier (for example, "us-west-1"), and the size check code before and after compression (for example, CRC32 hash value) to ensure shard integrity and traceability. For example, a 100MB sub-chunk of a Snappy-compressed and AES-encrypted video might be labeled "slice 023-us-west-1-CRC32-0x8a3e1d," where "023" indicates its sequential position in the original file.

[0097] Step S340: Cross-combining the compressed encrypted sub-blocks with the structured data blocks to generate encrypted data units distributed across regions.

[0098] Cross-combining involves mixing and encapsulating structured data blocks (such as database records) with unstructured compressed and encrypted sub-blocks (such as video segments) according to pre-set rules into a unified encrypted data unit. Specifically, each structured data block is assigned a primary storage partition based on the priority weight of the dynamic storage path (e.g., a weight of 0.8 for the US West partition and 0.6 for the Tokyo partition), and is then bound to the unstructured sub-blocks belonging to the same logical group. Subsequently, a cross-reference table is inserted during the encapsulation process (e.g., recording "structured data block ID-001" associating "video sub-block 023" with "video sub-block 045") to ensure that the associated data can be quickly located during subsequent access. For example, a structured data block (ID-001) containing a sales order record might be combined with three video feedback sub-blocks (023, 045, and 067) into a single encrypted data unit, labeled "unit ID-7a3b" and stored across the US West, Tokyo, and Frankfurt partitions.

[0099] Step S350: Map the encrypted data units to corresponding cloud storage partitions in sequence according to the priority weights of the dynamic storage paths.

[0100] The process of mapping encrypted data units to cloud storage partitions strictly adheres to the priority weights and zone binding policies defined in the dynamic storage path. Priority weights indicate the priority of different partitions in storage resource competition (e.g., a partition with a weight of 0.8 is prioritized for high-bandwidth links), while zone binding policies specify mandatory storage locations for specific data types (e.g., financial transaction data must be stored in a GDPR-compliant partition). The system queries the dynamic storage path configuration table based on the encrypted data unit's type tag (e.g., "High Security Level - Video Data"), matches a list of eligible target partitions, and sorts them in descending priority order (e.g., 0.9 for the Frankfurt partition and 0.7 for the Singapore partition). The system then attempts to write the encrypted data unit to the target partition in the order listed. If the target partition's resources are insufficient (e.g., storage space falls below a threshold), the unit is automatically downgraded to a less optimal partition. For example, an encrypted data unit labeled "High Security Level" might be preferentially mapped to the Zurich partition (weight 0.95). If that partition is full, it is then relocated to the Dublin partition (weight 0.85).

[0101] Step S400: Dynamically allocate storage resources of cloud storage partitions through a load balancing algorithm, and adjust the priority weights of dynamic storage paths according to real-time traffic characteristics.

[0102] A load balancing algorithm is a decision-making mechanism used to optimize resource allocation. Its core function is to dynamically adjust the storage location and connection allocation ratio of data shards based on the real-time load status of cloud storage partitions (such as CPU utilization, storage I / O throughput, and network bandwidth utilization). The process of dynamically allocating storage resources involves periodically collecting performance metrics from each partition (such as the current number of connections and data throughput fluctuation range), generating a partition load score using a weighted calculation formula (e.g., load balancing coefficient = 0.6 × connection stability factor + 0.4 × throughput fluctuation factor), and migrating some storage tasks from high-load partitions to low-load partitions based on the score. In this process, real-time traffic characteristics refer to instantaneous data flow properties captured from the network transport layer (such as request rate, packet size distribution, and transport protocol type). These characteristics are used to dynamically adjust the priority weights of storage paths. For example, if a sudden increase in the HTTP request rate for a partition is detected, the weight of the corresponding dynamic storage path may be increased from 0.7 to 0.9 to prioritize high-traffic requests.

[0103] Adjusting priority weights requires the following steps: First, the traffic analysis engine identifies key patterns in real-time traffic (such as sudden access spikes and sustained low-bandwidth transmission); second, the pattern characteristics are input into the storage path optimization model to recalculate the weight values ​​of each path; finally, the dynamic storage path configuration table is updated to reflect the new weight assignments. For example, if a US West Coast cloud storage partition experiences a surge in access requests due to a promotion, the load balancing algorithm may reduce its path weight from 0.8 to 0.5, divert some requests to the Singapore cloud storage partition, whose weight has been increased to 0.7, and delay the transmission of non-urgent data shards to off-peak processing times.

[0104] As an implementation method, step S400, dynamically allocating storage resources of cloud storage partitions through a load balancing algorithm, may specifically include:

[0105] Step S410: monitor the current number of connections and data throughput of each cloud storage partition in real time.

[0106] The current number of connections refers to the number of data access sessions processed simultaneously by a specific cloud storage partition within the real-time monitoring window (such as 1-second granularity), including active read and write request connections, data transmission connections, and metadata query connections; data throughput refers to the volume of data traffic passing through the partition per unit time (such as MB / s or GB / min), including the sum of upstream (write) and downstream (read) traffic. The real-time monitoring process is implemented through monitoring agents deployed in each partition. The agent periodically collects connection counters (such as the ESTABLISHED count in the TCP connection status table) and throughput meters (such as byte transmission and reception statistics at the network interface layer), and aggregates the data to the central load balancing controller. For example, the monitoring agent in the US West partition collects the current number of connections (such as 2450) and data throughput (such as 1.2GB / s) every 500ms, and uploads the indicators to the controller database via the HTTPS protocol.

[0107] Step S420: Calculate the load balancing coefficient of each cloud storage partition based on the current number of connections and data throughput.

[0108] The load balancing coefficient is a normalized value used to quantify the load status of a partition. Its calculation process combines two core indicators: connection stability and throughput volatility. Specifically, the system first obtains the partition's historical connection number sequence (such as the number of concurrent connections recorded per minute in the past 24 hours) and the current data throughput fluctuation range (such as the difference between the maximum and minimum throughput values ​​in the last 5 minutes). Through statistical analysis, it generates a first load factor reflecting the stability of the connection number (such as a coefficient of variation of 0.15) and a second load factor reflecting the amplitude of throughput changes (such as a range of 0.4). Subsequently, the two factors are input into the weighted fusion model, which determines the fusion ratio (such as a solid-state drive storage resource type or a mechanical hard drive storage resource type) based on the partition's storage resource type (such as a solid-state drive storage resource type or a mechanical hard drive storage resource type) and the priority weight of the dynamic storage path (such as a weight of 0.8 for the US West Coast partition), and outputs the initial load coefficient (such as 0.72). Finally, the initial load coefficient is mapped to a normalized load coefficient (such as 0.68) in the range of 0-1 through a dynamic range compression function (such as the Sigmoid function), which serves as the final evaluation indicator of the partition load status.

[0109] As an implementation, step S420, calculating the load balancing coefficient of each cloud storage partition, may specifically include:

[0110] Step S421: Obtain the historical connection number sequence and current data throughput fluctuation range of the cloud storage partition; the historical connection number sequence includes concurrent connection number statistics within multiple time windows.

[0111] The historical connection count series is a time series dataset extracted from partition monitoring logs. Its time window granularity can be configured to the minute level (for example, recording the number of concurrent connections once every minute) or the hourly level (for example, recording the peak number of connections every hour). For example, the historical connection count series for the US West partition may contain 1440 records (24 hours x 60 minutes), each of which includes a timestamp (for example, 2023-10-05 14:30:00) and the corresponding number of concurrent connections (for example, 2350). The current data throughput fluctuation range is calculated using a sliding time window (for example, the last 5 minutes). The difference between the maximum throughput value (for example, 1.5 GB / s) and the minimum throughput value (for example, 0.9 GB / s) within the window is the fluctuation range (0.6 GB / s).

[0112] Step S422: Generate a first load factor reflecting the stability of the number of connections based on the distribution characteristics of each statistical value in the historical number of connections sequence.

[0113] The quantification of connection number stability is achieved by analyzing the statistical distribution of the historical connection number sequence, specifically including calculating the variance, standard deviation or coefficient of variation (standard deviation / mean) of the sequence. For example, if the historical connection number mean of the West Coast region is 2000 and the standard deviation is 150, then the coefficient of variation is 150 / 2000=0.075. The smaller the coefficient of variation, the smoother the fluctuation of the connection number and the higher the stability. The first load factor can be designed as an inverse proportional function of the coefficient of variation (such as 1-coefficient of variation). When the coefficient of variation is 0.075, the first load factor is 0.925, indicating high stability.

[0114] Step S423: Based on the difference between the upper and lower limits of the current data throughput fluctuation range, generate a second load factor reflecting the throughput variation range.

[0115] The throughput change amplitude is calculated by the ratio of the difference between the upper and lower limits of the current data throughput fluctuation range (e.g., 0.6 GB / s) to the partition's maximum theoretical throughput (e.g., 10 GB / s). For example, if the difference is 0.6 GB / s and the maximum theoretical throughput is 10 GB / s, the change amplitude is 0.6 / 10 = 0.06. The second load factor can be designed as an inverse function of the amplitude value (e.g., 1-amplitude value). When the amplitude value is 0.06, the second load factor is 0.94, indicating that the throughput fluctuation is small and the system load is relatively stable.

[0116] Step S424: input the first load factor and the second load factor into a preset weight fusion model to generate an initial load coefficient; the weight fusion model determines the fusion ratio based on the storage resource type of the cloud storage partition and the priority weight of the dynamic storage path.

[0117] The weighted fusion model dynamically adjusts the weights of the first and second load factors based on the partition's storage resource type. For example, for partitions with SSD storage resources, the model assigns a first load factor weight of 0.7 and a second load factor weight of 0.3 because its low-latency characteristics are more sensitive to connection stability. For partitions with HDD storage resources, the model assigns a first load factor weight of 0.4 and a second load factor weight of 0.6 because its throughput limitations are more significant. If the first load factor of the US West partition (SSD) is 0.925 and the second load factor is 0.94, then the initial load factor = 0.7 × 0.925 + 0.3 × 0.94 = 0.9295.

[0118] Step S425: Dynamic range compression is performed on the initial load coefficient to generate a normalized load coefficient; the intensity of the dynamic range compression is dynamically adjusted according to the regional security level of the cloud storage partition.

[0119] Dynamic range compression uses a nonlinear function (such as a logarithmic function or a piecewise linear function) to map the initial load factor to a range of 0-1. For example, when the security level of a zone is "high," a strong compression strategy is used (for example, an initial load factor of 0.9295 is mapped to 0.72 using a sigmoid function) to avoid frequent migrations caused by small fluctuations. When the security level is "medium," a linear mapping strategy is used (for example, 0.9295 is directly used as the normalized value). This design ensures more conservative load assessments for high-security zone zones, reducing the risk of misjudgment.

[0120] Step S426: Determine the division boundary between the saturated load partition and the non-saturated load partition based on the comparison result of the normalized load coefficient and the preset load threshold; wherein the division boundary is used to control the migration ratio of the concurrent connection number and the update range of the priority weight.

[0121] Preset load thresholds are set based on the partition's hardware performance and business needs. For example, the threshold for partitions with SSD storage resources is set to 0.8, and for partitions with HDD storage resources is set to 0.7. If the normalized load factor of the US West partition is 0.72 and its threshold is set to 0.8, the partition is marked as an unloaded partition. If the normalized load factor of the Frankfurt partition is 0.85 (threshold 0.8), it is marked as a loaded partition. The partition boundary further influences the calculation logic of the migration ratio: when the coefficient exceeds the threshold, the excess is migrated according to a linear ratio (e.g., (0.85-0.8) / 0.85≈5.88%) or a nonlinear ratio (e.g., a square relationship). At the same time, the priority weight is updated proportionally to the excess value (e.g., the weight is reduced by 0.5% for every 0.01 excess).

[0122] Step S430: sorting the load balancing coefficients by a load balancing algorithm to determine saturated load partitions and non-saturated load partitions.

[0123] The load balancing algorithm sorts all partitions in descending order based on the normalized load coefficient, and divides them into saturated load partitions (coefficient ≥ 0.75) and non-saturated load partitions (coefficient < 0.75) based on a preset load threshold (such as 0.75). For example, if the normalized load coefficient of the Frankfurt partition is 0.82 and that of the Tokyo partition is 0.68, the former is marked as a saturated load partition and the latter as a non-saturated load partition. The hardware performance differences of the partitions need to be considered during the division process: for partitions configured with high-performance solid-state drive storage resource types, the load threshold can be appropriately increased (such as 0.8); while the threshold for partitions with mechanical hard drive storage resource types is lowered (such as 0.7) to avoid performance degradation caused by hardware bottlenecks.

[0124] Step S440: Migrate the number of concurrent connections of the saturated load partition to the non-saturated load partition in proportion, and update the priority weight of the dynamic storage path.

[0125] The proportion of concurrent connections that are migrated is determined by the extent to which the load factor of the saturated partition exceeds the threshold. For example, if the normalized load factor of the US West partition is 0.85 (threshold 0.75) and the excess is 0.1, the migration proportion is the ratio of the excess to the total load capacity (e.g., 0.1 / 0.85 ≈ 11.76%). The migration operation is implemented by the load balancing controller sending redirect instructions to the client: new connection requests are directly assigned to the unsaturated partition; established connections are gradually redirected to the target partition. Simultaneously, the priority weights of dynamic storage paths are dynamically adjusted based on the migration proportion: if the original weight of the US West partition was 0.8, after migrating 11.76% of connections, its weight decreases to 0.8 × (1 - 0.1176) = 0.706, while the weight of the Singapore partition increases from 0.6 to 0.6 + 0.1176 × 0.8 = 0.694, reflecting the rebalancing of resource allocation.

[0126] Step S450: reallocate the storage paths of the encrypted data units according to the updated priority weights to achieve elastic expansion of the cloud storage partitions.

[0127] The reallocation of storage paths must be combined with the updated priority weights and the real-time available capacity of the partitions. For example, if the weight of the Singapore partition rises to 0.694 and its remaining storage capacity is 5TB, and the volume of the newly generated encrypted data unit is 200GB, the system will prioritize allocating it to the Singapore partition and update the regional binding relationship in the shard index. The elastic expansion mechanism is automatically triggered in this process: when the resource utilization of an unsaturated partition exceeds the preset expansion threshold (such as 80%), the system dynamically applies for additional storage resources through the cloud service API (such as expanding the storage cluster nodes by 10%) to ensure that the connection and data load after migration can be effectively carried.

[0128] Step S500: When an access request for an encrypted data unit is detected, the number of concurrent connections of the cloud storage partition is reallocated according to the updated priority weight and storage path optimization model to achieve elastic expansion.

[0129] An access request is a read or write operation instruction for a specific encrypted data unit initiated by a client or application, which contains a target data identifier (such as a shard index hash value), an access region identifier (such as a user's geographic location code), and an operation type tag (such as a read / write permission level). The process of reallocating the number of concurrent connections needs to combine the updated priority weights (reflecting the current load status of each partition) and the prediction results of the storage path optimization model (such as the growth trend of the number of connections in the next 5 minutes), and dynamically adjust the maximum connection threshold and resource quota of the partition through the elastic expansion controller. For example, when the storage path optimization model predicts that the Frankfurt partition will receive more than 5,000 concurrent connections in the next 10 minutes, the elastic expansion controller can temporarily expand the number of virtual server instances in the partition and increase the maximum number of connections from 3,000 to 8,000 to cope with peak pressure.

[0130] The specific process for achieving elastic expansion includes: first, parsing the target data identifier in the access request and locating the corresponding cloud storage partition through the encrypted shard index; second, calculating the concurrent connection quota that needs to be added based on the current number of connections of the partition, the historical peak number of connections, and the expansion recommendations generated by the storage path optimization model; finally, dynamically adjusting the partition resources (such as adding load balancer nodes and expanding the storage cluster size) through the cloud service provider's API interface. During this process, if it is detected that the overflow risk level of the partition exceeds the preset threshold (such as the connection occupancy rate exceeds 95%), the system will automatically create a temporary storage node to take over the excess requests and gradually release redundant resources after the traffic subsides. For example, when the Tokyo cloud storage partition encounters a DDoS attack, the elastic expansion mechanism may be triggered to expand the number of concurrent connections from 2,000 to 10,000, and the number of connections will be restored to normal levels through a reverse merge operation after the attack ends.

[0131] As an embodiment, in step S500, when an access request for an encrypted data unit is detected, the number of concurrent connections of the cloud storage partition is reallocated according to the updated priority weight and storage path optimization model, which may specifically include:

[0132] Step S510: Parse the target data identifier and access area information in the access request.

[0133] An access request is a read or write operation instruction initiated by a client or application targeting a specific encrypted data unit. Its packet header contains a target data identifier and access region information. The target data identifier is composed of the encrypted data unit's globally unique identifier (e.g., a UUID) and a shard index hash value (e.g., a SHA-256 hash), uniquely identifying the target data unit and its corresponding shard set. The access region information is determined by geo-location resolution of the client's IP address or a user-specified region code (e.g., "EU-Central-1"), indicating the logical or physical region of the request's origin. The parsing process is implemented by a request parsing engine: first, the target data identifier field (e.g., "DataID=7a3b-c41d") is extracted from the packet header and split into a UUID (7a3b) and a shard index hash (c41d) according to a preset format. Second, the client IP address (e.g., 203.0.113.45) is converted to a standard region identifier (e.g., "AP-Southeast-1") using an IP address library or region code mapping table. For example, a video access request from a Singaporean user generates a target data identifier "7a3b-c41d" and access area information "AP-Southeast-1" after parsing, which is used for subsequent path decision-making.

[0134] Step S520: Locate the corresponding encrypted data unit from the encrypted shard index according to the target data identifier, and obtain the storage path history record of the encrypted data unit.

[0135] The encrypted shard index is a distributed key-value storage structure whose key is the shard index hash value and whose value is a collection of metadata for the encrypted data unit (such as the shard storage location, access key, and historical path records). When locating an encrypted data unit, the system queries the encrypted shard index using the shard index hash value (c41d) in the target data identifier to obtain the metadata for the data unit to which the shard belongs (such as the primary storage partition is "AP-Southeast-1" and the backup shard is located in "EU-Central-1"). The storage path history record is a log data set extracted from the metadata that describes the historical storage location changes of the data unit. It includes a timestamp (such as 2023-10-05 14:30:00), the source partition identifier (such as "AP-Southeast-1"), the target partition identifier (such as "EU-Central-1"), the reason for the migration (such as load balancing), and performance indicators after the migration (such as response latency reduced from 120ms to 80ms). For example, the historical records of the encrypted data unit "7a3b" show that it was migrated to the European partition three times in the past week due to peak access in the Asia-Pacific region, and the latency was reduced by about 30% after each migration.

[0136] Step S530: Generate a new priority weight through a storage path optimization model based on the storage path history records and access area information.

[0137] When generating new priority weights, the storage path optimization model performs a multi-dimensional correlation analysis between the performance indicators in the storage path history (such as average latency and throughput fluctuations) and the geographic location characteristics in the access region information (such as inter-regional network latency and compliance constraints). For example, if the access region is "AP-Southeast-1" and the historical records show that the inter-continental transmission delay from this region to "EU-Central-1" is high (200ms), the model may reduce the priority weight of the European partition (from 0.7 to 0.5) and increase the weight of the local partition in the Asia-Pacific region (from 0.8 to 0.9). During the specific calculation process, the model quantifies the influence coefficient of the inter-regional network quality on the weight through a logistic regression algorithm, and dynamically adjusts the output value based on real-time load data (such as the current number of connections in the European partition is 3,800) to ensure that the weight distribution conforms to historical rules and adapts to real-time status.

[0138] Step S540: Adjust the storage path distribution of the encrypted data unit according to the new priority weight, and reallocate the number of concurrent connections of the cloud storage partition.

[0139] Adjusting the storage path distribution requires updating the priority weight field in the dynamic storage path configuration table and triggering a shard migration task. For example, when the weight of the Asia-Pacific partition of the encrypted data unit "7a3b" rises to 0.9, the system migrates its primary storage replica from the European partition to the Singapore partition and updates the region binding relationship in the shard index. When redistributing the number of concurrent connections, the load balancing controller calculates the connection quota for each partition based on the new weight: if the Singapore partition weight of 0.9 corresponds to 70% of the connection quota (for example, 3500 out of a total of 5000 concurrent connections), and the European partition weight of 0.5 corresponds to 30% of the quota (1500), the system will proportionally migrate the excess connections in the European partition (for example, the current number of connections is 2000) (migrating 500 connections to the Singapore partition). The migration operation is implemented through the session redirection protocol (such as HTTP 307 Temporary Redirect) to ensure that the client switches to the new partition seamlessly.

[0140] Step S550: When the number of concurrent connections exceeds the elastic expansion threshold, a temporary storage node is created to take over excess connection requests.

[0141] The elastic scaling threshold is a preset upper limit on the number of connections based on the partition's hardware specifications and business needs (e.g., 5000 for SSD storage resource types and 3000 for HDD storage resource types). When the system detects that the number of concurrent connections in a partition exceeds the threshold (e.g., 5100 connections for the Singapore partition), the system automatically calls the cloud service provider's API to create a temporary storage node. Temporary storage nodes use on-demand elastic instances (e.g., AWS EC2 Spot Instances), whose configuration (e.g., number of vCPUs, memory size) is dynamically determined based on the number of excess connections (e.g., a 2-core, 4GB instance is required for 100 excess connections). After the node is created, the system routes excess connection requests (100 connections) to the temporary node and marks its lifecycle in the node metadata (e.g., automatically releasing it after 12 hours). For example, during the "Double 11" (Singles' Day) promotion, the number of connections in the Singapore partition surged to 6000. The system created three temporary nodes (each accommodating 300 connections) to ensure that the number of connections in the primary partition remained below 5100. The temporary resources were then gradually released after traffic subsided.

[0142] In an optional derivative implementation, after reallocating the number of concurrent connections to the cloud storage partitions in step S500, the method may further include:

[0143] Step S600: monitor the real-time response delay and throughput fluctuation characteristics of the elastically expanded cloud storage partition, and generate an extended performance indicator set.

[0144] Real-time response latency refers to the time interval from receiving a request to returning a response when processing a data access request for an elastically scaled cloud storage partition (e.g., the average latency of the US West Coast partition increased from 50ms to 70ms). This is monitored by distributed probe nodes periodically sending simulated requests (e.g., HTTP GET / POST) to the target partition and recording the round-trip time. Throughput fluctuation refers to the magnitude of the change in partition data traffic per unit time (e.g., the throughput of the Singapore partition increased from 1.2GB / s to 2.5GB / s within 5 minutes and then fell back to 1.8GB / s). This is calculated by counting the maximum, minimum, and standard deviation of the traffic over a sliding time window (e.g., 10-second granularity). When generating the extended performance indicator set, the system combines the time series data of real-time response latency and the throughput fluctuation range into a structured dataset, appending a timestamp, partition identifier, and an extended operation type label (e.g., "temporary node expansion"). For example, the extended performance indicator set for the elastically scaled Frankfurt partition may include the following fields:

[0145] {

[0146] "Partition ID":"eu-central-1",

[0147] "Average Latency": "75ms",

[0148] "Throughput fluctuation range": "1.5-2.3GB / s",

[0149] "Extension Type": "Temporary Node Expansion",

[0150] "Time Range": "2023-10-05 15:00:00 to 15:05:00"

[0151] }

[0152] Step S700: adjusting the priority weight update frequency in the storage path optimization model according to the response delay threshold and throughput fluctuation range in the extended performance indicator set.

[0153] The response delay threshold is the maximum allowable delay value (such as 100ms) preset according to the service level agreement (SLA), and the throughput fluctuation range threshold is defined as the allowed throughput range percentage (such as not exceeding 30% of the partition's maximum theoretical throughput). When adjusting the priority weight update frequency, the system compares the extended performance indicator with the threshold: if the real-time response delay continues to exceed the threshold (such as three consecutive monitoring cycle delays ≥ 110ms), the priority weight update frequency is increased (such as from once an hour to once every 10 minutes) to speed up the path optimization response; if the throughput fluctuation range is lower than the threshold (such as a range of 20%), the update frequency is reduced (such as from once every 15 minutes to once every 30 minutes) to reduce the model calculation overhead. For example, after the temporary node expansion, the throughput fluctuation range of the Tokyo partition dropped to 15%, and the system adjusted its weight update frequency from once every 10 minutes to once every 20 minutes, while maintaining the response delay threshold at 90ms.

[0154] Step S800: Input the adjusted priority weight update frequency into the storage path optimization model, and recalculate the encrypted shard index mapping relationship of the dynamic storage path.

[0155] The storage path optimization model dynamically adjusts its internal time step control logic based on the new update frequency parameters. For example, when the update frequency is increased to once every 10 minutes, the model shortens the time window of historical data (such as from 24 hours to 6 hours) and increases the weight of real-time features (such as the current number of connections) to 70% to respond more quickly to recent load changes. When recalculating the encrypted shard index mapping relationship, the model combines the latest priority weight (such as the Singapore partition weight increased from 0.7 to 0.85) and the available capacity of the partition (such as 5TB of remaining storage space) to generate a new shard index binding relationship. For example, the encrypted data unit "7a3b" originally mapped to the Frankfurt partition was remapped to the Singapore partition because its weight was reduced to 0.6, and the regional code field in the shard index was updated to "ap-southeast-1".

[0156] Step S900: Based on the updated encrypted shard index mapping relationship, the encrypted data units of the cloud storage partition are redistributed across regions, and the shard location metadata of the virtualized storage node is synchronously updated.

[0157] Cross-region redistribution involves migrating encrypted data units from the original storage partition to the target partition based on the new encrypted shard index mapping. The migration is performed by the shard transfer service: first, the shard contents of the encrypted data unit (e.g., the encrypted data block of shard 023) are read from the original partition; second, the shard integrity is verified (e.g., by checking the SHA-256 hash value); finally, the shard is written to the target partition and the metadata of the virtualized storage node is updated (e.g., the location field of shard 023 is changed from "eu-central-1" to "ap-southeast-1"). During the synchronization process, the system uses a two-phase commit protocol to ensure data consistency: in the preparation phase, the original shard is locked from writes; in the commit phase, the original shard resources are released after confirming the successful write to the target partition. For example, after the 200 shards of the encrypted data unit "7a3b" were migrated from the Frankfurt partition to the Singapore partition, the "storage path" field in their metadata was updated in batches, and the migration timestamp and operation ID were recorded for audit purposes.

[0158] As an implementation method, step S900, redistributing the encrypted data units of the cloud storage partition across regions, may specifically include:

[0159] Step S910: extracting the access frequency characteristics of the encrypted data unit and the regional security level change trend.

[0160] Access frequency characteristics refer to the distribution pattern of the number of times an encrypted data unit is accessed within a specified time window (e.g., 24 hours) (e.g., shard 023 accounts for 80% of accesses during the Asia-Pacific working hours (09:00-18:00)). This is generated by counting the request logs of each shard. Regional security level change trends refer to the dynamic adjustment records of the security policies of the storage partitions where the shards are located (e.g., the data encryption standard of the Singapore partition was upgraded from AES-128 to AES-256). This is extracted from security policy update logs and compliance audit reports. For example, the access frequency characteristics of encrypted data unit "7a3b" show a 60% drop in accesses during the early morning hours (00:00-06:00) in the European region. The regional security level change trend indicates that the security level of the Frankfurt partition was downgraded from "high" to "medium" due to the GDPR audit.

[0161] Step S920: Generate a hotspot area tag based on the access frequency characteristics, and calculate the matching degree between the regional security level change trend and the encrypted shard index of the dynamic storage path.

[0162] Hotspot area tags are used to identify logical or physical areas (e.g., "ap-southeast-1" is marked as a hotspot area) with an access frequency exceeding a preset threshold (e.g., an average daily visit volume exceeding 10,000). The matching degree is calculated by comparing the changing trend of the regional security level with the security requirements in the encrypted shard index: if the shard index requires a "high" security level but the current level of the area is "medium", the matching degree score decreases (e.g., from 0.9 to 0.6). The calculation can use cosine similarity or Euclidean distance algorithms. For example, when the security level of the Singapore partition changes from "high" to "medium", its matching degree with the shard index requirements drops from 0.95 to 0.75.

[0163] Step S930: When the matching degree calculation result is lower than the preset security threshold, the encrypted shard index with a low security level is screened out from the virtualized storage node, and the shard encryption protocol is triggered to rotate the key of the encrypted shard index.

[0164] The preset security threshold is typically set to 0.7. When the match falls below this value, the system scans the virtualized storage nodes and filters out all encrypted shard indices bound to low-security partitions (e.g., shards 023 and 045 in the Singapore partition). Key rotation is performed by the key management service in the shard encryption protocol: first, a new key pair (e.g., RSA-2048 public / private key) is generated; second, the shard data is re-encrypted using the new key; and finally, the key identifier field in the shard index is updated. For example, the original key "KeyID-001" for shard 023 is replaced with "KeyID-002," and the rotation timestamp and operator ID are recorded in the shard header.

[0165] Step S940: Based on the encrypted shard index after key rotation, dynamically mask the hotspot area mark of the encrypted data unit to generate a temporary storage path.

[0166] Dynamic masking refers to temporarily removing or replacing the hotspot region tag from the shard metadata with an obfuscated identifier (such as changing "ap-southeast-1" to "region-xx") to prevent malicious attackers from inferring the location of critical data through access patterns. When generating a temporary storage path, the system binds the masked region tag to the backup partition (such as the Mumbai partition) and sets the path validity period (such as 24 hours). For example, the temporary storage path of the encrypted data unit "7a3b" points to the Mumbai partition, and the region field in its metadata is masked to "region-5a", retaining the real partition mapping only in the internal routing table.

[0167] Step S950: Verify the temporary storage path and the conflict avoidance path by superposition, and update the priority weight allocation strategy of the cloud storage partition according to the verification result.

[0168] Overlay verification involves applying both temporary storage paths and conflict avoidance paths simultaneously to test the system's performance under mixed routing conditions. The verification process includes: 1) distributing test requests to the temporary path and conflict avoidance path (e.g., allocating 50% of traffic to each); 2) monitoring response latency, throughput, and error rate; 3) if the temporary path meets performance requirements (e.g., latency ≤ 120ms), its weight is increased to 0.7, and the conflict avoidance path weight is reduced to 0.3; otherwise, the system rolls back to the original path configuration. For example, the temporary path in the Mumbai partition performed exceptionally well in the test (latency 105ms), so the system increased its weight from 0.5 to 0.7 and updated the priority weight allocation policy document.

[0169] Step S1000: When a conflict in the access path of the redistributed encrypted data unit is detected, a conflict avoidance path is generated according to the storage path optimization model, and a load balancing algorithm is triggered to redistribute the number of concurrent connections of the conflicting path.

[0170] An access path conflict occurs when multiple clients simultaneously request different replica shards of the same encrypted data unit, resulting in a surge in the number of connections to the target partition or data transmission congestion (e.g., shard 023 of the Singapore partition is accessed simultaneously by 1,000 concurrent connections). Conflict detection is achieved by real-time monitoring of the source of access requests to the shards and the load of the target partition: when the same shard receives more than the preset conflict threshold (e.g., 500 requests) within 1 second and the target partition load factor exceeds 0.8, it is marked as a conflict path. When generating a conflict avoidance path, the storage path optimization model analyzes the performance indicators of the alternative partitions (e.g., the Mumbai partition has a latency of 110ms and a load factor of 0.65), selects the optimal backup partition, and generates temporary routing rules (e.g., diverting 50% of the requests to the Mumbai partition).

[0171] As an implementation method, in step S1000, triggering a load balancing algorithm to redistribute the number of concurrent connections on the conflicting path may specifically include:

[0172] Step S1001: Identify the target storage partition with an excessive number of concurrent connections in the conflict path, and obtain the historical peak number of connections and the current encryption shard load ratio of the target storage partition.

[0173] The target storage partition is identified based on real-time connection monitoring data and elastic expansion thresholds (such as the Singapore partition's threshold of 5000). When it is detected that its concurrent connections reach 5100, it is marked as an over-limit partition. The historical connection peak is extracted from the performance log (such as the highest number of connections in the past 7 days is 5200), and the current encryption shard load ratio is calculated as the percentage of occupied connections to the total connection quota (such as 5100 / 5000=102%). For example, the historical peak of the Singapore partition is 5200, and the current load ratio is 102%, triggering the secondary distribution condition.

[0174] Step S1002: Generate a connection number prediction curve based on the historical connection number peak value, and calculate the overflow risk level of the target storage partition in combination with the current encryption shard load ratio.

[0175] The connection count prediction curve is generated using a time series prediction algorithm (such as ARIMA or LSTM). It inputs historical connection count peaks and time characteristics (such as weekdays / holidays) and outputs an estimated connection count for the next hour (e.g., 5,300 connections expected in 30 minutes). The overflow risk level is calculated based on the predicted value and the current load ratio: if the predicted value exceeds 120% of the elastic expansion threshold (e.g., 5,000 x 120% = 6,000), the risk level is "high"; if it is between 110% and 120%, the risk level is "medium." For example, the predicted value for the Singapore zone is 5,300 (106% of the threshold), and the risk level is marked as "medium."

[0176] Step S1003: When the overflow risk level exceeds a preset risk threshold, a spare storage partition is dynamically allocated from the virtualized storage node, and the encrypted data units of the target storage partition are migrated in a hierarchical manner according to the overflow risk level.

[0177] The default risk threshold is typically set to "medium." When the risk level is ≥ "medium," the system selects the backup partition with the lowest load factor from the virtualized storage node pool (e.g., a Mumbai partition with a load factor of 0.4). Tiered migration prioritizes shards based on their overflow risk level: high-risk shards (e.g., the top 10% of access frequency) are migrated first, followed by medium-risk shards. For example, 200 high-risk shards from the Singapore partition are migrated first to the Mumbai partition, with a migration rate of 100 shards / minute.

[0178] Step S1004: During the hierarchical migration process, the encrypted shard index binding relationship of the standby storage partition is adjusted in real time according to the availability status of the temporary storage path.

[0179] The availability status of the temporary storage path is evaluated in real time through heartbeat detection and performance probes (e.g., the latency of the Mumbai partition has dropped from 130ms to 115ms). If the backup partition is stable (e.g., the latency is ≤ 150ms for 5 consecutive detections), the number of shard index bindings is gradually increased (e.g., from 50 to 150). If the status is abnormal (e.g., the packet loss rate is ≥ 5%), the migration is suspended and the secondary backup partition (e.g., the Sydney partition) is switched. For example, after migrating 50 shards, the latency of the Mumbai partition stabilizes at 120ms. The system increases the number of bound shards to 100 and updates the region code in the shard index.

[0180] Step S1005: When the overflow risk level of the target storage partition is reduced to a safe range, the storage path optimization model is triggered to reversely merge the encrypted data units of the backup storage partition and release the redundant storage resources generated during the migration process.

[0181] Reverse merging involves gradually migrating encrypted data units temporarily relocated to the backup partition back to the target storage partition. When the overflow risk level of the Singapore partition drops to "low" (predicted number of connections: 4,800), the system initiates the relocation task based on shard priority (low risk precedes high risk) and releases the temporary storage instances in the backup partition (e.g., shutting down three EC2 Spot instances). For example, 150 shards from the Mumbai partition were migrated back to the Singapore partition in batches over a 24-hour period. After the migration was complete, the 5TB of storage space occupied by the temporary nodes was released.

[0182] The embodiment of the present invention also provides a data storage system, such as Figure 2 As shown, data storage system 100 includes a processor 101 and a memory 103. Processor 101 and memory 103 are connected, for example, via bus 102. Optionally, data storage system 100 may further include a transceiver 104. It should be noted that in practical applications, the number of transceivers 104 is not limited to one, and the structure of data storage system 100 does not constitute a limitation on the embodiments of the present invention.

[0183] An embodiment of the present invention provides a data storage system. The data storage system in the embodiment of the present invention includes: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory and configured to be executed by the one or more processors. When the one or more programs are executed by the processor, the method provided in the above embodiment is implemented.

Claims

1. A digital enterprise management data storage method based on cloud computing, characterized in that: The method comprises: Acquire a real-time management data set in an enterprise data stream, and generate a virtualized storage node based on structured data blocks and unstructured data blocks in the real-time management data set; the virtualized storage node includes encrypted shard indexes associated with different regions, and each encrypted shard index corresponds to a dynamic storage path; Determine a target encrypted shard index that matches the virtualized storage node based on a preset storage path optimization model; the storage path optimization model is generated based on regional access frequency and encryption level characteristics in historical storage logs; Based on the dynamic storage path corresponding to the target encrypted shard index, split the structured data block and the unstructured data block into encrypted data units distributed across regions, and map the encrypted data units to corresponding cloud storage partitions; Dynamically allocate storage resources of the cloud storage partitions through a load balancing algorithm, and adjust the priority weights of the dynamic storage paths according to real-time traffic characteristics; When an access request for the encrypted data unit is detected, the number of concurrent connections of the cloud storage partition is reallocated according to the updated priority weight and the storage path optimization model to achieve elastic expansion.

2. The method according to claim 1, characterized in that The training process of the storage path optimization model includes the following steps: Extracting multiple region-related features from the historical storage log; the region-related features include region access frequency, encryption level features, storage delay parameters, and concurrent connection peaks; Generating an initial training set according to the regional visit frequency and encryption level characteristics, and inputting the initial training set into the initial path optimization network for a first round of training; During the first round of training, a node weight distribution of the initial path optimization network is determined by a cross-validation algorithm, and an intermediate path optimization model is constructed according to the node weight distribution; Inputting the storage delay parameter and the concurrent connection peak into the intermediate path optimization model to generate a path optimization parameter set corresponding to the cloud storage partition; the path optimization parameter set includes the priority weight of the dynamic storage path, the update period of the encrypted shard index, and the storage resource allocation ratio; The intermediate path optimization model is back-propagated and optimized according to the path optimization parameter set to generate the storage path optimization model; wherein the output of the storage path optimization model includes the priority weight and storage resource allocation ratio of the dynamic storage path corresponding to the target encryption shard index.

3. The method according to claim 2, characterized in that The determining of the node weight distribution of the initial path optimization network by a cross-validation algorithm includes: Dividing the initial training set into a plurality of training subsets and a validation subset, and performing forward propagation training on the initial path optimization network based on the training subsets; After each forward propagation training, calculating the path prediction error of the initial path optimization network through the verification subset, and adjusting the node weight distribution according to the path prediction error; When the path prediction error satisfies a preset convergence condition, determining that the node weight distribution of the initial path optimization network is an optimal weight distribution; generating a weight update function corresponding to the encryption level feature according to the optimal weight distribution and the region visit frequency in the initial training set; Iteratively updating the node weights of the initial path optimization network by using the weight update function until the path prediction error reaches a minimum threshold; The performing back propagation optimization on the intermediate path optimization model according to the path optimization parameter set includes: Inputting the priority weights and storage resource allocation ratios in the path optimization parameter set into the hidden layer of the intermediate path optimization model to generate hidden layer output features; Calculating a delay loss value of the intermediate path optimization model according to the hidden layer output feature and the stored delay parameter; Backpropagating the delay loss value through a gradient descent algorithm to adjust the hidden layer weight parameters of the intermediate path optimization model; When the delay loss value reaches a preset optimization threshold, fixing the hidden layer weight parameters of the intermediate path optimization model to final weight parameters; According to the final weight parameter and the concurrent connection peak value, the output layer parameter of the storage path optimization model is generated to match the output of the storage path optimization model with the priority weight and storage resource allocation ratio of the dynamic storage path.

4. The method according to claim 1, wherein The step of dividing the structured data block and the unstructured data block into encrypted data units distributed across regions based on the dynamic storage path corresponding to the target encryption shard index includes: Determining the number of encrypted shards for each cloud storage partition based on the regional distribution characteristics of the dynamic storage paths; Generating a shard encryption protocol corresponding to each cloud storage partition based on the number of encryption shards and the data type of the structured data block; Slice-compress the unstructured data block using the slicing encryption protocol to generate a plurality of compressed encrypted sub-blocks; Cross-combining the compressed encrypted sub-blocks with the structured data blocks to generate the cross-region distributed encrypted data units; According to the priority weights of the dynamic storage paths, the encrypted data units are mapped to corresponding cloud storage partitions in sequence.

5. The method according to claim 4, characterized in that The step of generating a shard encryption protocol corresponding to each cloud storage partition based on the number of encryption shards and the data type of the structured data block includes: Determining a data compression mode according to the data type of the structured data block, and selecting an encryption algorithm type that matches the data compression mode from a preset encryption algorithm library; Dividing the fragment size range of the unstructured data block based on the number of encrypted fragments, and generating an initial fragmentation rule according to the fragment size range; Adapting the encryption strength of the fragment size range in the initial fragmentation rule by using the encryption algorithm type to generate an intermediate fragmentation rule; Splitting the unstructured data block into a plurality of initial encryption fragments according to the encryption strength adaptation result in the intermediate fragmentation rule; Performing compression mode verification on the initial encrypted slices, and adjusting the matching relationship between the data compression mode and the slice size range according to the verification result; A final sharding rule is generated based on the adjusted matching relationship, and the final sharding rule is bound to the encryption algorithm type to generate the sharding encryption protocol.

6. The method according to claim 5, characterized in that The generation process of the shard encryption protocol also includes dynamically adjusting the final sharding rules through a sharding optimization model, including: Extracting shard performance characteristics associated with the encryption algorithm type, data compression mode, and shard size range from historical shard logs; Inputting the sharding performance characteristics into the sharding optimization model to generate sharding adjustment parameters corresponding to the final sharding rule; Performing overlapping area detection on the boundaries of the initial encrypted slices according to the slice overlap threshold in the slice adjustment parameter; When it is detected that the boundary overlap area of ​​the initial encrypted fragments exceeds the fragment overlap threshold, recalculating the fragment size range and updating the final fragmentation rule; Performing secondary sharding on the unstructured data block based on the updated final sharding rule to generate an optimized encrypted shard set; According to the cross-combination result of the optimized encryption shard set and the structured data block, the binding relationship between the shard size range and the encryption algorithm type in the shard encryption protocol is modified.

7. The method according to claim 1, characterized in that The dynamically allocating storage resources of the cloud storage partitions by using a load balancing algorithm includes: Monitor the current number of connections and data throughput of each cloud storage partition in real time; Calculate a load balancing coefficient for each cloud storage partition based on the current number of connections and data throughput; sorting the load balancing coefficients by using the load balancing algorithm to determine saturated load partitions and non-saturated load partitions; Migrating the number of concurrent connections of the saturated load partition to the non-saturated load partition in proportion, and updating the priority weight of the dynamic storage path; The storage paths of the encrypted data units are reallocated according to the updated priority weights to achieve elastic expansion of the cloud storage partitions.

8. The method according to claim 7, characterized in that Calculating the load balancing coefficient of each cloud storage partition includes: Obtaining a historical connection count sequence and a current data throughput fluctuation range of the cloud storage partition; the historical connection count sequence includes concurrent connection count statistics within multiple time windows; generating a first load factor reflecting the stability of the number of connections according to the distribution characteristics of each statistical value in the historical number of connections sequence; generating a second load factor reflecting a throughput variation range based on a difference between an upper limit and an lower limit of the current data throughput fluctuation range; Inputting the first load factor and the second load factor into a preset weight fusion model to generate an initial load coefficient; the weight fusion model determines a fusion ratio based on the storage resource type of the cloud storage partition and the priority weight of the dynamic storage path; Performing dynamic range compression on the initial load coefficient to generate a normalized load coefficient; the intensity of the dynamic range compression is dynamically adjusted according to the regional security level of the cloud storage partition; Based on the comparison result of the normalized load coefficient and the preset load threshold, the division boundary between the saturated load partition and the non-saturated load partition is determined; wherein the division boundary is used to control the migration ratio of the concurrent connection number and the update range of the priority weight.

9. The method according to claim 1, characterized in that When an access request to the encrypted data unit is detected, reallocating the number of concurrent connections to the cloud storage partition according to the updated priority weight and the storage path optimization model includes: parsing the target data identifier and access area information in the access request; Locate the corresponding encrypted data unit from the encrypted shard index according to the target data identifier, and obtain a storage path history record of the encrypted data unit; Based on the storage path history and access area information, generating a new priority weight through the storage path optimization model; Adjusting the storage path distribution of the encrypted data unit according to the new priority weight, and reallocating the number of concurrent connections of the cloud storage partition; When the number of concurrent connections exceeds the elastic expansion threshold, a temporary storage node is created to take over the excess connection requests.

10. A data storage system, characterized in that: include: one or more processors; Memory; One or more computer programs; wherein the one or more computer programs are stored in the memory and configured to be executed by the one or more processors, and when the one or more computer programs are executed by the processors, implement the method according to any one of claims 1 to 9.

Images