Digital enterprise management data storage method and system based on cloud computing

Abstract

The application provides a kind of digital enterprise management data storage method and system based on cloud computing, method includes: obtaining real-time management data set in enterprise data stream, generating virtualization storage node, according to the preset storage path optimization model, determine the target encryption fragment index matched with virtualization storage node, based on the dynamic storage path corresponding to target encryption fragment index, structured data block and unstructured data block are segmented into cross-regional distribution encryption data unit, and encryption data unit is mapped to corresponding cloud storage partition;Through load balancing algorithm, the storage resource of cloud storage partition is dynamically allocated, and the priority weight of dynamic storage path is adjusted according to real-time traffic characteristics;When detecting the access request of encryption data unit, the number of concurrent connections of cloud storage partition is re-allocated.The application can solve the method of collaborative optimization problem of resource allocation and security efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, in particular to a cloud computing-based digital enterprise management data storage method and system. BACKGROUND

[0002] In the field of cloud computing, the storage of enterprise digital management data usually relies on distributed architecture and static sharding strategy, for example, the data is divided into encrypted units according to fixed rules and stored in designated areas. In related technologies, the selection of cloud storage paths is usually based on preset area division rules, such as allocating fixed partitions according to data types or storage capacity, and processing the sharded data through a unified encryption algorithm. In addition, the existing load balancing schemes mostly use real-time connection number monitoring and threshold triggering mechanisms, such as migrating part of the connections to low-load areas when the number of concurrent connections in a certain partition exceeds the preset threshold.

[0003] However, the above methods have significant defects in practical application. First, the rigidity of the static sharding strategy, the fixed encryption sharding rules cannot adapt to the dynamic changes of regional access frequency and security requirements. For example, high-security-level data allocated to a low-access-frequency region will cause a sharp increase in access delay; and the storage resources of a hot region may be quickly exhausted due to fixed sharding size, requiring frequent expansion. Then, there is the problem of fragmented processing of security and efficiency, such as the independent decision of encryption level and storage path selection, for example, high-encryption-level data uses complex algorithms but the storage path is not optimized, resulting in a difficult balance between encryption overhead and access efficiency. In addition, there is the problem of hysteresis of load balancing and elastic expansion, such as the threshold triggering mechanism based on the current number of connections cannot predict traffic trends, for example, a sudden high-concurrency request may cause the system to have a response timeout before the expansion is completed, and the expansion process is not linked with the storage path priority, resulting in low migration efficiency.

[0004] The above defects result in the difficulty of the existing technology to achieve efficient use and dynamic security control of storage resources, resulting in low resource utilization, large response delay fluctuations, and high hardware maintenance costs, especially when processing high-concurrency and multi-modal enterprise management data, the system stability and scalability are severely limited. Therefore, there is an urgent need for a method to solve the problem of coordinated optimization of resource allocation and security efficiency. SUMMARY

[0005] The present application aims to provide a cloud computing-based digital enterprise management data storage method and system. The present application is implemented as follows:

[0006] In a first aspect, the present application provides a cloud computing-based digital enterprise management data storage method, which comprises: obtaining real-time management data sets in an enterprise data stream, and generating a virtual storage node according to structured data blocks and unstructured data blocks in the real-time management data sets; the virtual storage node contains encrypted shard indexes associated with different regions, and each encrypted shard index corresponds to a dynamic storage path; determining a target encrypted shard index matched with the virtual storage node according to a preset storage path optimization model; the storage path optimization model is generated by training based on region access frequency and encryption level characteristics in a historical storage log; based on the dynamic storage path corresponding to the target encrypted shard index, the structured data blocks and unstructured data blocks are segmented into encrypted data units distributed across regions, and the encrypted data units are mapped to corresponding cloud storage partitions; the storage resources of the cloud storage partitions are dynamically allocated through a load balancing algorithm, and the priority weight of the dynamic storage path is adjusted according to real-time traffic characteristics; when an access request of the encrypted data unit is detected, the number of concurrent connections of the cloud storage partition is re-allocated according to the updated priority weight and the storage path optimization model, so as to realize elastic expansion.

[0007] In a second aspect, the present application provides a data storage system, which comprises: one or more processors; a memory; one or more computer programs; wherein the one or more computer programs are stored in the memory and configured to be executed by the one or more processors, and when the one or more computer programs are executed by the processor, the method described above is realized.

[0008] The application provides a cloud computing-based digital enterprise management data storage method, which realizes efficient utilization and safe control of storage resources by dynamically generating a virtualized storage node, binding a cross-region encrypted fragment index, intelligently segmenting and mapping structured and unstructured contents of real-time management data in combination with a storage path optimization model. Specifically, the virtualized storage node can automatically select an optimal storage path according to real-time traffic characteristics by associating dynamic encrypted fragment indexes with regional characteristics, solving the problems of resource waste and security vulnerabilities caused by traditional static fragmentation strategies; the storage path optimization model is trained and generated based on regional access frequency and encryption level characteristics in historical storage logs, and can dynamically balance data access efficiency and security requirements to avoid access delay bottlenecks of high-security-level data caused by fixed paths; by segmenting structured and unstructured data blocks into cross-region distributed encrypted data units and dynamically adjusting the priority weight of the storage path in combination with a load balancing algorithm, the elastic expansion of storage resources can be realized in a high-concurrency scenario, and the system response speed and stability are significantly improved. In addition, the dynamic binding mechanism of the encrypted fragment index and the real-time mapping relationship of the cloud storage partition form a closed-loop management, ensuring that the data storage location is always matched with the regional security level and access load state, thereby reducing the hardware maintenance cost and enhancing the adaptability of the system to sudden traffic fluctuations. BRIEF DESCRIPTION OF DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the application.

[0010] Figure 1 is a flowchart of a cloud computing-based digital enterprise management data storage method provided by an embodiment of the application.

[0011] Figure 2 is a composition schematic diagram of a storage system provided by an embodiment of the application. DETAILED DESCRIPTION

[0012] The embodiments of the application will be described below in combination with the drawings in the embodiments of the application. The terms used in the embodiment part of the application are only used to explain the specific embodiments of the application, and are not intended to limit the application.

[0013] The execution subject of the cloud computing-based digital enterprise management data storage method in the embodiment of the application is a data storage system, for example, a storage server. As shown in Figure 1 The method comprises the following steps.

[0014] Step S100: obtaining a real-time management data set in an enterprise data stream, and generating a virtualized storage node according to structured data blocks and unstructured data blocks in the real-time management data set; the virtualized storage node contains encrypted shard indexes associated with different regions, and each encrypted shard index corresponds to a dynamic storage path.

[0015] The real-time management data set refers to a data collection continuously collected from enterprise operation processes and containing real-time dynamic update features, and the data sources include but are not limited to business system logs, sensor monitoring data streams, user interaction behavior records, and cross-platform data transmission interfaces. The structured data blocks are data units with a predefined data model and capable of being parsed through a fixed field format, such as relational data table records in a database, JSON or XML format key-value pair data; the unstructured data blocks are data units without a fixed format or a predefined model, such as image files, audio / video streams, natural language texts, or binary log files. The process of generating the virtualized storage node includes: separating the real-time management data set into structured and unstructured data blocks through a data classification engine, assigning initial storage resource labels based on data block type features (such as data volume, format identifier, and access frequency), and binding the storage resource labels with cloud storage regions in geographical distribution or logical partitioning according to a preset region association strategy. In this process, the encrypted shard index is a unique identifier generated for each virtualized storage node, used to indicate the distribution logic of the associated encrypted shard in the cloud storage region, for example, associating the data shard index in the North America region with the West US cloud storage partition, and associating the data shard index in the Asia Pacific region with the Singapore cloud storage partition. Each encrypted shard index corresponds to a dynamic storage path, which refers to a data transmission routing rule dynamically adjusted according to real-time network status, regional load conditions, or security strategies, for example, when it is detected that a certain regional storage node is overloaded, the dynamic storage path can re-route the data shard to a backup storage node in a low-load region.

[0016] Specifically, the operation of generating a virtualized storage node is implemented by the following process: first, format parsing is performed on the data blocks in the real-time management dataset, and structured data blocks and unstructured data blocks are distinguished according to the metadata characteristics (such as file header identification, content type label) of the data blocks; second, an independent data fingerprint (such as a hash value or a digital digest) is generated for each data block, and the data fingerprint is matched with the encrypted shard index template in the storage area strategy library to determine the area association relationship to which the data block should be bound; finally, a virtualized storage node instance is created according to the matching result, and the mapping relationship between the encrypted shard index and the dynamic storage path is written into the node metadata. For example, when a structured data block is detected as a database transaction log, its encrypted shard index will be marked as "high priority - European region", and the corresponding dynamic storage path points to the high-performance storage cluster of the Frankfurt cloud storage partition; while unstructured data blocks such as monitoring video streams may be marked as "medium priority - Asia-Pacific region", and the dynamic storage path points to the object storage service of the Tokyo cloud storage partition.

[0017] Step S200: determining a target encrypted shard index matching the virtualized storage node according to a preset storage path optimization model; the storage path optimization model is generated by training based on the area access frequency and encryption level characteristics in the historical storage log.

[0018] The storage path optimization model is a prediction model trained by a machine learning algorithm, which inputs include multiple area association characteristics recorded in the historical storage log, and the output is the optimal encrypted shard index and dynamic storage path configuration parameters adapted to the virtualized storage node. The area access frequency refers to the frequency statistical value of data access requests received by a specific cloud storage partition in a historical time period, for example, the number of access requests of the US East cloud storage partition during the daily peak period; the encryption level characteristic is used to describe the security level required to be met by the data shard during storage, for example, the shard using the AES-256 encryption algorithm needs to be associated with a high-security level storage path, and the shard using the TLS1.2 transmission protocol needs to be bound to a low-latency storage path. The process of training the storage path optimization model includes: extracting the time series data of the area access frequency, the hierarchical label of the encryption level characteristic, the measured value of the storage delay parameter, and the statistical distribution data of the concurrent connection peak from the historical storage log, and constructing the association rules between the area characteristics and the storage path performance through a supervised learning algorithm.

[0019] In this process, the operation of determining the target encryption shard index needs to perform the following steps: first, input the metadata of the virtualized storage node (such as data block type, region association label, encryption level requirement) into the storage path optimization model; second, the model calculates the matching scores of different encryption shard indexes by analyzing the periodic characteristics of region access frequency (such as the difference in access mode between weekdays and holidays) and the compatibility constraints of encryption level characteristics (such as high security level shards need to avoid cross-region transmission); finally, select the encryption shard index with the highest matching score as the target encryption shard index. For example, if the encryption level characteristic associated with a virtualized storage node is "highest security level", and the historical storage log shows that the access frequency of the Europe region is low in the target time period, the model may point the target encryption shard index to the dedicated encryption storage area of the Zurich cloud storage partition, and assign it a dynamic storage path that only allows intranet transmission to reduce exposure risk.

[0020] As an implementation, the training process of the storage path optimization model includes the following steps:

[0021] Step S201: Extract a plurality of region association characteristics from the historical storage log; the region association characteristics include region access frequency, encryption level characteristics, storage delay parameters, and concurrent connection peak value.

[0022] The historical storage log refers to the data set recorded by the cloud storage system in the historical time period, which contains storage operation metadata and performance indicators, and its data fields cover storage request timestamp, target partition identifier, encryption algorithm type, transmission protocol version, response delay milliseconds, and concurrent connection number statistics. Region association characteristics are a set of key indicators extracted from historical storage logs to represent the behavior patterns of different cloud storage partitions, including: region access frequency (such as the average number of access times of the US East partition during 08:00-10:00 every day), encryption level characteristics (such as the proportion of shards using AES-256 encryption algorithm in the Asia-Pacific partition), storage delay parameters (such as the 95th percentile delay value of data write operation in the Europe-Middle East partition), and concurrent connection peak value (such as the number of concurrent connections in the Singapore partition during the highest historical load period). The process of extracting region association characteristics needs to be realized through a log parsing engine, which generates structured feature data from raw logs according to pre-defined feature extraction rules (such as time window sliding average calculation, encryption level classification statistics). For example, for region access frequency, the engine can count the number of access requests received by the US West partition every hour in the past 30 days and generate a time series data set; for encryption level characteristics, the number of shards using TLS1.3 protocol in Tokyo partition is counted and the ratio to the total number of shards is mapped to a normalized feature value in the range of 0-1.

[0023] Step S202: Generate an initial training set according to the regional access frequency and encryption level features, and input the initial training set into the initial path optimization network for the first round of training.

[0024] The initial training set refers to the basic data set used to train the storage path optimization model, which is composed of time series data of regional access frequency and classification labels of encryption level features. The process of generating the initial training set includes: standardizing the regional access frequency data (such as Z-score normalization), converting the encryption level features into one-hot encoding (such as encoding "high encryption level" as [1, 0, 0] and "medium encryption level" as [0, 1, 0]), and merging them into a multi-dimensional feature vector after aligning them by time dimension. The initial path optimization network is a neural network architecture that has not been trained, with the number of input layer nodes consistent with the feature dimension of the initial training set, the hidden layer adopting a fully connected structure, and the number of output layer nodes matching the configuration parameter dimension of the dynamic storage path. The goal of the first round of training is to make the network preliminarily learn the influence of regional access frequency and encryption level features on storage path selection through supervised learning. For example, when the input features are "East US regional access frequency 0.8, encryption level features [1, 0, 0]", the network needs to output the priority weight (such as 0.9) of the high-security level storage path and the corresponding resource allocation proportion (such as 70% of the storage resources allocated to the path).

[0025] Step S203: In the first round of training, determine the node weight distribution of the initial path optimization network through the cross-validation algorithm, and construct the intermediate path optimization model according to the node weight distribution.

[0026] The cross-validation algorithm is a data segmentation strategy used to evaluate the generalization ability of the model, and its core operation is to divide the initial training set into multiple mutually exclusive subsets (such as 5 subsets), and sequentially select one subset as the validation set and the remaining subsets as the training set, and optimize the node weight distribution through multiple iterations of training and validation. In the first round of training, the node weight distribution of the initial path optimization network is gradually adjusted through forward propagation and error backpropagation: in the forward propagation stage, the input features pass through the hidden layer activation function (such as ReLU) to generate the predicted output; in the validation stage, the mean squared error (such as the prediction error of the path priority weight) between the predicted output and the true path configuration parameters is calculated, and the gradient descent algorithm is used to update the node weights. When the path prediction error meets the preset convergence condition (such as the error change rate being less than 1% for three consecutive iterations), the current node weight distribution is determined as the optimal weight distribution, and the intermediate path optimization model is constructed based on it. For example, if the cross-validation shows that the high access frequency feature of the West US region has the strongest correlation with the weight of the third node in the hidden layer (the weight value is 0.85), then this node will have higher importance in the intermediate model and be used to capture the path selection rule of high-frequency access regions.

[0027] As an implementation, in step S203, the node weight distribution of the initial path optimization network is determined by a cross-validation algorithm, which can specifically include:

[0028] Step S2031: Divide the initial training set into multiple training subsets and validation subsets, and perform forward propagation training on the initial path optimization network based on the training subsets.

[0029] The operation of dividing the training subsets and the validation subsets needs to follow the K-fold cross-validation rule. For example, the initial training set is equally divided into 5 subsets (subsets 1 to 5), subset 1 is selected as the validation set, and subsets 2-5 are selected as the training set, one round of training is completed, and then the validation set is changed to subset 2, and the process is repeated until all subsets are used as the validation set. During the forward propagation training process, the input features are loaded into the network input layer in batches from the training subsets, and after the linear combination and nonlinear activation (such as the Sigmoid function) of the hidden layer nodes, the prediction values of the priority weight and the resource allocation ratio are generated. For example, when the training subset contains 1000 historical access records in the Eastern United States region, the forward propagation will calculate the path priority prediction value corresponding to each record one by one, and compare it with the true label to initialize the weight adjustment direction.

[0030] Step S2032: After each forward propagation training, the path prediction error of the initial path optimization network is calculated through the validation subset, and the node weight distribution is adjusted according to the path prediction error.

[0031] The path prediction error refers to the difference between the prediction value output by the model on the validation subset and the true configuration parameter, and the mean square error (MSE) or the mean absolute error (MAE) is usually used as an evaluation index. For example, if the model's priority weight prediction value for a record in the validation subset is 0.7, and the true value is 0.8, then the single-point error is 0.1; by taking the average of the errors of all validation samples, the overall path prediction error is obtained. When adjusting the node weight distribution, the stochastic gradient descent (SGD) algorithm is used to update the weight matrix according to the error gradient direction. For example, if the second node of the hidden layer has the largest contribution to the error (the gradient value is 0.15), then its weight will be significantly reduced to reduce the subsequent prediction bias.

[0032] Step S2033: When the path prediction error meets the preset convergence condition, the node weight distribution of the initial path optimization network is determined as the optimal weight distribution.

[0033] The preset convergence condition is usually set as an error change rate or an absolute error threshold, for example, requiring the overall path prediction error fluctuation amplitude of three consecutive iterations to be less than 0.5%, or the error absolute value to be reduced to 0.02 or less. When the condition is met, it indicates that the model has learned the distribution rule of the training data sufficiently, and the node weight distribution will no longer change significantly due to continuous training. At this time, the optimal weight distribution will be fixed, for example, the weight value of the first node of the hidden layer is fixed to 0.68, and the weight value of the third node is fixed to 0.73, forming a stable feature mapping relationship.

[0034] Step S2034: According to the optimal weight distribution and the region access frequency in the initial training set, a weight update function corresponding to the encryption level feature is generated.

[0035] The weight update function is a mathematical expression used to dynamically adjust the node weight according to the real-time region access frequency in the model inference stage, to adapt to the path selection requirements of different encryption level features. For example, if the optimal weight distribution shows that the high encryption level feature (such as [1, 0, 0]) is strongly related to the fifth node of the hidden layer (weight 0.85), the weight update function can be defined as: when the encryption level is high, the fifth node weight increases by 10%; when the encryption level is low, the fifth node weight decreases by 5%. This function is encoded into the model inference logic to ensure that the path optimization strategy can be adaptively adjusted when the encryption level changes.

[0036] Step S2035: The node weights of the initial path optimization network are iteratively updated through the weight update function until the path prediction error reaches the minimum threshold.

[0037] The iterative update process needs to repeatedly load the initial training set data in the training environment, adjust the node weights using the weight update function, and re-evaluate the path prediction error. For example, if the initial error is 0.1, after three iterations of updating, the error is reduced to 0.05, and the fifth node weight is adjusted from 0.85 to 0.89, indicating that the function effectively improves the prediction accuracy in the high encryption level scenario. When the error reaches the minimum threshold (such as 0.03), the iteration is terminated and the optimal weight distribution is imported into the intermediate path optimization model.

[0038] Step S204: Input the storage delay parameter and the concurrent connection peak value into the intermediate path optimization model to generate a path optimization parameter set corresponding to the cloud storage partition; the path optimization parameter set includes the priority weight of the dynamic storage path, the update period of the encryption shard index, and the storage resource allocation ratio.

[0039] The storage delay parameter refers to a measured index reflecting the response speed of each partition extracted from the historical storage log (for example, the average read delay of the Singapore partition is 120 ms), and the concurrent connection peak value refers to the statistical value of the concurrent connection number of the partition at the peak load period in history (for example, the highest concurrent connection number of the Frankfurt partition is 5000). When the above parameters are input into the intermediate path optimization model, the model performs nonlinear transformation on the delay and concurrent features through the hidden layer nodes to generate a set of path optimization parameters. For example, when the input is "storage delay parameter 150 ms, concurrent connection peak value 4500", the model may output the dynamic storage path priority weight 0.75 (indicating the recommended priority of the path under the current load), the encryption shard index update period 24 hours (indicating that the binding relationship of the shard index needs to be re-evaluated every 24 hours), and the storage resource allocation proportion 60% (i.e. 60% of the storage resources are allocated to the partition corresponding to the path). This process needs to rely on the generalization ability of the model to ensure that reasonable parameter configurations can be generated under unseen delay and concurrent combinations.

[0040] Step S205: performing back propagation optimization on the intermediate path optimization model according to the set of path optimization parameters to generate a storage path optimization model; wherein the output of the storage path optimization model includes the priority weight and the storage resource allocation proportion of the dynamic storage path corresponding to the target encryption shard index.

[0041] Back propagation optimization refers to a process of taking the target output (such as priority weight 0.8, resource allocation proportion 65%) in the set of path optimization parameters as a supervision signal, calculating the loss function (such as cross-entropy loss) between the model prediction output and the target value, and adjusting the hidden layer weight parameters along the network in the reverse direction. In this stage, the storage delay parameter is used to calculate the delay loss value (such as the difference between the predicted delay and the measured delay), and the concurrent connection peak value is used to constrain the calculation range of the resource allocation proportion (such as when the concurrent peak value exceeds 4000, the resource allocation proportion cannot be lower than 50%). When the delay loss value reaches the preset optimization threshold (such as the loss value is reduced to below 0.05), the hidden layer weight parameters are fixed as the optimal weight, and the output layer parameters of the storage path optimization model are generated based on the optimal weight and the output layer activation function (such as Softmax). For example, the model may strongly associate the priority weight calculation logic of the Western Pacific partition with the fourth node of the hidden layer (weight 0.92) after optimization, to ensure that the priority of the path is automatically increased in a high-concurrent scenario.

[0042] As an implementation manner, in step S205, the back propagation optimization is performed on the intermediate path optimization model according to the set of path optimization parameters, which can specifically include:

[0043] Step S2051: inputting the priority weight and the storage resource allocation proportion in the set of path optimization parameters into the hidden layer of the intermediate path optimization model to generate hidden layer output features.

[0044] The hidden layer output feature refers to an intermediate representation generated by non-linear transformation of the input parameters through the hidden layer nodes, and the dimension is consistent with the number of hidden layer nodes. For example, when the input priority weight is 0.8 and the storage resource allocation ratio is 65%, the hidden layer can generate a 128-dimensional feature vector, where each dimension corresponds to an abstract feature of different path optimization modes (for example, the 12th dimension represents "high priority-low delay path preference", and the 45th dimension represents "balanced resource allocation tendency").

[0045] Step S2052: Calculate the delay loss value of the intermediate path optimization model according to the hidden layer output feature and the storage delay parameter.

[0046] The delay loss value is used to quantify the deviation of the predicted storage path in the actual delay performance, and the calculation formula can be the mean square error of the predicted delay (based on the hidden layer output feature) and the actual storage delay parameter. For example, if the hidden layer output feature indicates that the predicted delay of a certain path is 130 ms, and the actual delay is 145 ms, the single-point loss value is (145-130)^2=225; by averaging the loss values of all samples, the overall delay loss value is obtained.

[0047] Step S2053: Adjust the hidden layer weight parameters of the intermediate path optimization model by backpropagation of the delay loss value through the gradient descent algorithm.

[0048] The gradient descent algorithm determines the weight adjustment direction and amplitude by calculating the partial derivative (gradient) of the delay loss value with respect to the hidden layer weight. For example, if the gradient calculation shows that increasing the weight of the third node of the hidden layer by 0.1 can reduce the loss value by 5%, the optimizer will gradually increase the weight by the learning rate (such as 0.001). This process needs to be iterated multiple times until the improvement of the weight adjustment on the loss value tends to be stable.

[0049] Step S2054: When the delay loss value reaches the preset optimization threshold, the hidden layer weight parameters of the intermediate path optimization model are fixed as the optimal weight parameters.

[0050] The preset optimization threshold is usually set as an acceptable performance deviation range, for example, the delay loss value needs to be less than 10 ms of mean absolute error. When the loss value decreases below the threshold (such as 8 ms), the hidden layer weight parameters are no longer updated and are marked as the optimal weight parameters. For example, the weight of the seventh node of the hidden layer is fixed at 0.72, indicating that its contribution to low-delay path selection has reached the optimal state.

[0051] Step S2055: Generate the output layer parameters of the storage path optimization model according to the optimal weight parameters and the peak value of concurrent connections, so that the output of the storage path optimization model matches the priority weight and the storage resource allocation ratio of the dynamic storage path.

[0052] The generation of output layer parameters needs to be combined with the constraints of the maximum weight parameter and the concurrent connection peak. For example, if the concurrent connection peak is 5000, the output layer activation function needs to map the hidden layer features to the priority weight range [0, 1] and the resource allocation proportion range [0%, 100%], while ensuring that the resource allocation proportion is not lower than the preset lower limit (such as 30%) when the concurrency is high. Through linear regression or Softmax function, the model finally outputs parameters that completely match the dynamic storage path configuration, such as priority weight 0.78 and resource allocation proportion 58%.

[0053] Step S300: Based on the dynamic storage path corresponding to the target encrypted fragment index, the structured data block and the unstructured data block are segmented into encrypted data units distributed across regions, and the encrypted data units are mapped to the corresponding cloud storage partition.

[0054] The encrypted data unit distributed across regions refers to the data entity that the original data block is cut into multiple sub-blocks according to the preset fragmentation rule, and is dispersedly stored in different geographical or logical location cloud storage partitions through encryption algorithm and region binding strategy. The process of segmenting structured data blocks and unstructured data blocks needs to follow the dynamic storage path rules defined in the target encrypted fragment index, for example: for structured data blocks, if the dynamic storage path requires that the fragment size does not exceed 1MB and needs to be stored across at least three regions, the data needs to be cut according to the fixed block size (such as 1MB), and a region identifier needs to be attached to each sub-block; for unstructured data blocks, if the dynamic storage path requires dynamic adjustment of fragment size to adapt to network bandwidth fluctuations, the fragment boundary needs to be dynamically calculated according to the real-time transmission rate. The generation of encrypted data units needs to be realized through a fragmentation encryption protocol, which includes data compression algorithm (such as ZIP or Snappy), encryption algorithm type (such as AES or RSA), and fragmentation metadata packaging rules (such as combining fragment serial number, region identifier and check code into fragment header).

[0055] Specifically, the operation of mapping the encrypted data units to the cloud storage partitions includes the following stages: first, according to the partition priority weight defined in the dynamic storage path (for example, the partition weight in the United States and Western Europe is 0.8, and the partition weight in Singapore is 0.6), each encrypted data unit is assigned an initial target partition; second, the encrypted data unit is transmitted to the target partition through a cross-region transmission protocol (such as HTTP / 3 or a dedicated tunnel protocol), and the partition availability state is monitored in real time during transmission; finally, the shard index mapping relationship of the encrypted data unit is recorded in the storage service of the target partition to ensure that subsequent access requests can quickly locate the shard position through the index. For example, an unstructured data block (such as a video file) can be divided into 100 encrypted data units, of which the first 30 units are mapped to the Tokyo cloud storage partition, the middle 40 units are mapped to the Sydney cloud storage partition, and the remaining 30 units are mapped to the Mumbai cloud storage partition, and the encrypted shard index of each unit contains the access key and geographic location code of the corresponding partition.

[0056] As an implementation, in step S300, the structured data block and the unstructured data block are divided into encrypted data units distributed across regions based on the dynamic storage path corresponding to the target encrypted shard index, which can specifically include:

[0057] In step S310, the number of encrypted shards of each cloud storage partition is determined according to the regional distribution characteristics of the dynamic storage path.

[0058] The regional distribution characteristics of the dynamic storage path refer to the attribute set parsed from the target encrypted shard index to describe the geographic location or logical grouping of the cloud storage partition, including but not limited to the physical location code of the partition (such as latitude and longitude coordinates), the logical level label (such as core storage area or edge cache area), and the upper limit of storage capacity (such as the maximum storage capacity of 10PB in the United States and Western Europe). The process of determining the number of encrypted shards needs to combine the storage capacity in the regional distribution characteristics and the load balancing strategy: first, calculate the theoretical maximum number of shards according to the upper limit of the partition storage capacity (such as a partition with a capacity of 10TB can accommodate 10000 shards at 1GB per shard); second, dynamically adjust the actual number of shards based on the shard redundancy coefficient defined in the load balancing strategy (such as each data unit needs at least 3 copies in each partition). For example, if the dynamic storage path requires the United States and Western Europe partition to store the primary copy, the Tokyo partition to store the first backup copy, and the Frankfurt partition to store the second backup copy, the number of encrypted shards needs to be allocated by multiplying the total number of data units by the copy coefficient 3 to ensure that the number of shards in each partition matches the proportion of its capacity.

[0059] In step S320, a shard encryption protocol corresponding to each cloud storage partition is generated based on the number of encrypted shards and the data type of the structured data block.

[0060] The sharding encryption protocol is a set of data sharding and encryption rules customized for a specific cloud storage partition. Its core elements include the sharding size range (e.g., unstructured data block sharding size limit of 64MB to 256MB), encryption algorithm type (e.g., AES-256 or ChaCha20), compression mode (e.g., ZIP compression or Snappy compression), and metadata packaging format (e.g., sharding header containing sharding sequence number, region identifier, and check code). When generating the sharding encryption protocol, first determine the data compression mode based on the data type of the structured data block: for highly structured database transaction logs (e.g., MySQL binlog), use the low-latency LZ4 compression algorithm; for semi-structured JSON data streams, use the Zstandard algorithm that balances compression rate and speed. Then, based on the number of encrypted shards, divide the sharding size range of unstructured data blocks, for example, when the number of encrypted shards is 1000, the initial sharding size of unstructured video files can be set to 100MB, and the size floating factor (e.g., ±20%) in the sharding encryption protocol is used to allow dynamic adjustment.

[0061] As an implementation, in step S320, based on the number of encrypted shards and the data type of the structured data block, a sharding encryption protocol corresponding to each cloud storage partition is generated, which can specifically include:

[0062] In step S321, determine the data compression mode according to the data type of the structured data block, and select an encryption algorithm type from the pre-set encryption algorithm library that matches the data compression mode.

[0063] The data type of the structured data block is defined by the format identifier in its metadata (e.g., "type=sql" indicates a relational database log, and "type=json" indicates semi-structured JSON data). When determining the data compression mode, the system queries the pre-set compression strategy table according to the data type: for example, "sql" type data uses the LZ4 compression algorithm by default to achieve low-latency processing, "json" type data uses the Zstandard algorithm to balance compression rate and speed, and "csv" type data uses gzip compression to maximize storage efficiency. The selection of the encryption algorithm type needs to be compatible with the compression mode, for example, LZ4 compressed data is small in size and can be adapted to high-strength AES-256 encryption; while gzip compressed data is time-consuming to process, and may choose the low-computational ChaCha20 algorithm. When selecting an algorithm from the encryption algorithm library, the hardware acceleration support of the target partition (e.g., whether Intel QAT supports the AES-NI instruction set) also needs to be considered.

[0064] In step S322, based on the number of encrypted shards, divide the sharding size range of unstructured data blocks, and generate initial sharding rules according to the sharding size range.

[0065] When dividing the size range of the fragments, the system calculates the theoretically optimal fragment size according to the number of encrypted fragments and the average network bandwidth of the target partition. For example, if the number of encrypted fragments is 1000 and the bandwidth of the target partition is 1 Gbps, the size range of the fragments can be initialized to 1 MB to 10 MB to ensure that the transmission time of the fragments (1 MB / 125 MBps = 8 ms) is much lower than the TCP timeout threshold. When generating the initial fragment rule, the fragment boundary determination logic (such as cutting according to a fixed size or content-sensitive segmentation), the allowed range of fragment overlap (such as adjacent fragments can overlap by 5% to prevent data loss), and the fragment metadata format (such as the fragment header containing the version number and the fragment check code) need to be defined. For example, the initial fragment rule for an unstructured data block may stipulate that each fragment size is 5 MB ± 10%, the overlapping area between fragments does not exceed 512 KB, and the fragment header contains a SHA-256 check code.

[0066] Step S323: Perform encryption strength adaptation on the size range of the fragments in the initial fragment rule by the encryption algorithm type to generate an intermediate fragment rule.

[0067] Encryption strength adaptation refers to adjusting the size range of the fragments according to the computational overhead and security requirements of the selected encryption algorithm. For example, the AES-256 encryption algorithm requires the fragment size to be an integer multiple of 16 bytes due to its block encryption characteristics, so the 5 MB ± 10% in the initial fragment rule needs to be adjusted to 4.992 MB (i.e. 4992 KB, which can be divided by 16); while the ChaCha20 algorithm does not have strict alignment requirements for the size of the fragments due to its stream encryption characteristics, so the initial fragment rule can be kept unchanged. When generating the intermediate fragment rule, the system encodes the encryption algorithm constraints (such as the fragment padding method, the initialization vector length) into the fragment metadata and updates the legal value range of the fragment size (such as the adjusted size range is 4.992 MB to 10.016 MB).

[0068] Step S324: Divide the unstructured data block into a plurality of initial encrypted fragments according to the encryption strength adaptation result in the intermediate fragment rule.

[0069] When performing the fragment division, the system cuts the unstructured data block according to the size range and encryption constraints in the intermediate fragment rule. For example, a 10 GB video file is divided into 2048 initial encrypted fragments (10*1024 / 4.992 ≈ 2048) with a fixed size of 4.992 MB, each fragment is padded to a 16-byte boundary at the end to meet the AES encryption requirements, and a fragment header containing an initialization vector (IV) and a timestamp is attached. In this process, the fragment tool needs to calculate the fragment hash value (such as SHA-256) in real time and write it into the metadata to verify the integrity of the fragments later.

[0070] Step S325: Perform a compression mode check on the initial encrypted shard, and adjust the matching relationship between the data compression mode and the shard size range according to the check result.

[0071] The compression mode check aims to verify the adaptability of the compression algorithm to the shard size. For example, if the shard compressed by Snappy compression is too small (e.g., 1 MB) and the compression rate is less than 20%, the shard size range needs to be increased to 2 MB or more to improve compression efficiency; conversely, if the shard compressed by Zstandard compression is too large (e.g., 10 MB) and the single-shard compression time exceeds 50 ms, the size range needs to be reduced to 5 MB to reduce the delay. The check result is evaluated by three indicators: compression rate, compression time, and shard volume fluctuation rate. If the indicators do not meet the preset threshold (e.g., compression rate ≥ 30%), the system will automatically trigger the re-matching process of the compression mode and the shard size. For example, when the compression rate of LZ4 compression is only 25% at 2 MB shard, the system may switch to Zstandard algorithm or increase the shard size to 4 MB.

[0072] Step S326: Generate a final shard rule based on the adjusted matching relationship, and bind the final shard rule with the encryption algorithm type to generate a shard encryption protocol.

[0073] The final shard rule integrates the complete configuration of the compression mode, shard size range, encryption algorithm parameters, and shard metadata format. For example, the adjusted rule may specify that the shard size is fixed at 4 MB, Zstandard compression (compression level 5) and AES-256-GCM encryption (IV length 12 bytes) are used, and the shard header contains the shard sequence number, region identifier, pre- and post-compression volume, and SHA-256 checksum. When generating the shard encryption protocol, the system binds the final shard rule with the encryption algorithm type (e.g., "AES-256-GCM") and writes it into a protocol configuration file (e.g., XML or JSON format) for shard tools and storage services to call. For example, the protocol file may contain the following entries:

[0074] {

[0075] "Shard Size Range": "4MB ± 5%",

[0076] "Compression Algorithm": "Zstandard-level5",

[0077] "Encryption Algorithm": "AES-256-GCM",

[0078] "Metadata Fields": ["Shard ID", "Region Code", "Pre-compression Volume", "SHA-256"]

[0079] }

[0080] As an implementation, the generation process of the sharding encryption protocol further includes dynamically adjusting the final sharding rule through a sharding optimization model, which can specifically include:

[0081] Step S3261: Extract sharding performance features associated with the encryption algorithm type, data compression mode, and sharding size range from the historical sharding log.

[0082] The historical sharding log records detailed information about past sharding and encryption processing of data, including the use of encryption algorithm types, the effects of data compression modes, and the rationality of sharding size ranges. The encryption algorithm type is selected from the preset encryption algorithm library in step S321, the data compression mode is determined according to the data type of the structured data block, and the sharding size range is obtained after multiple adjustments in the previous steps. The sharding performance features are a series of indicators related to the encryption algorithm type, data compression mode, and sharding size range, such as encryption time, compression ratio, data integrity, etc.

[0083] Extracting sharding performance features associated with the encryption algorithm type, data compression mode, and sharding size range from the historical sharding log is to obtain experience and data in actual operation. Through analysis of the historical sharding log, the sharding performance under different combinations of encryption algorithm types, data compression modes, and sharding size ranges can be understood. For example, certain encryption algorithms may have higher encryption efficiency within a specific sharding size range, and certain data compression modes may achieve better compression rates when processing specific types of data. Extracting these sharding performance features provides data support for subsequent dynamic adjustment using the sharding optimization model.

[0084] For example, the historical sharding log records key performance indicators of past sharding operations, including sharding compression time consumption (such as Zstandard compression of 4MB sharding with an average time consumption of 12ms), encryption throughput (such as AES-256-GCM encryption rate of 550MB / s), sharding transmission success rate (such as 99.98%), and storage partition response delay (such as an average write delay of 45ms in the US West partition). When extracting sharding performance features, the system aggregates log data according to the three dimensions of encryption algorithm type, data compression mode, and sharding size range to generate a feature vector. For example, for the combination of “AES-256-GCM + Zstandard + 4MB”, the feature vector may contain [compression ratio 35%, encryption rate 550MB / s, sharding retransmission rate 0.02%].

[0085] Step S3262: Input the sharding performance features into the sharding optimization model to generate sharding adjustment parameters corresponding to the final sharding rule.

[0086] The slice optimization model is a machine learning-based regression model that takes slice performance feature vectors as input and outputs slice adjustment parameters (such as recommended slice size float range ±8%, compression level adjustment to 4). The model analyzes performance bottlenecks in historical data (such as high compression level causing CPU overload) to predict the impact of adjustment parameters on slice efficiency. For example, when the input features show that a 4MB slice in the Tokyo partition has a transmission success rate of 97%, the model may output the slice size adjustment parameter "reduce to 3.5MB" to adapt to the network fluctuation characteristics of the partition.

[0087] Step S3263: According to the slice overlap threshold in the slice adjustment parameter, the boundary of the initial encrypted slice is detected for the overlap area.

[0088] The slice overlap threshold is used to control the proportion of data duplication between adjacent slices (such as allowing a maximum overlap of 5%) to prevent data loss or verification failure. When detecting the overlap area, the system compares the start and end offsets of adjacent slices and calculates the proportion of overlapping bytes. For example, if the end offset of slice N is 1000000 bytes and the start offset of slice N+1 is 999500 bytes, the overlap area is 500 bytes, accounting for 0.49% of the slice size (1MB), which is below the 5% threshold and is considered compliant.

[0089] Step S3264: When the boundary overlap area of the initial encrypted slice is detected to exceed the slice overlap threshold, the slice size range is recalculated and the final slice rule is updated.

[0090] If the overlap area of a batch of slices reaches 6%, the system will recalculate the slice size according to the slice adjustment parameter. For example, if the original slice size is 4MB, the overlap area exceeding the standard may be caused by inaccurate slicing due to network packet loss, in which case the size needs to be reduced to 3.8MB and the number of slices needs to be increased, and the size range in the final slice rule is updated to "3.8MB ±3%". The updated rule will be synchronized to the slice encryption protocol to ensure that subsequent slice operations meet the new constraints.

[0091] Step S3265: Based on the updated final slice rule, the unstructured data block is re-sliced to generate an optimized encrypted slice set.

[0092] The re-slicing process uses the updated slice rule to re-cut the unstructured data block. For example, a 10GB video file is divided into 2632 slices (10*1024 / 3.8≈2632) with a size of 3.8MB, and each slice is encrypted after Zstandard compression to generate an optimized encrypted slice. In this process, the system records the actual size, compression rate, and encryption time of each slice and writes it into the historical slice log for subsequent model training.

[0093] Step S3266: According to the cross combination result of the optimized encrypted fragment set and the structured data block, the binding relationship between the fragment size range and the encryption algorithm type in the fragment encryption protocol is corrected.

[0094] When correcting the fragment encryption protocol, the parameter binding relationship needs to be adjusted according to the optimized fragment performance data. For example, if the compression rate of Zstandard compression level 4 is improved to 38% at 3.8MB fragments after secondary fragmentation, the compression level parameter in the protocol is adjusted from "level5" to "level4"; at the same time, the fragment size range is updated to "3.8MB±3%", and the encryption algorithm type remains "AES-256-GCM". The corrected protocol will be applied to all subsequent fragment operations as the new baseline configuration until the optimization process is triggered again.

[0095] Step S330: Perform fragment compression on the unstructured data block through the fragment encryption protocol to generate multiple compressed and encrypted sub-blocks.

[0096] When performing fragment compression, the compression mode and encryption algorithm in the fragment encryption protocol are sequentially applied to the unstructured data block: first, the original data is cut into multiple sub-blocks according to the fragment size range (such as cutting a 10GB video file into 100 100MB sub-blocks); second, each sub-block is compressed using a specified compression algorithm (such as using Snappy compression to reduce the sub-block volume to 70MB); finally, the compressed sub-block is encrypted by an encryption algorithm (such as using AES-256-GCM mode encryption and attaching an identity verification label). In this process, the fragment encryption protocol requires that each compressed and encrypted sub-block be attached with fragment metadata, including fragment sequence number (for data recombination), region identifier (such as "us-west-1"), and volume check code before and after compression (such as CRC32 hash value), to ensure fragment integrity and traceability. For example, a 100MB video sub-block that has been compressed by Snappy and encrypted by AES may be labeled as "Fragment 023-us-west-1-CRC32-0x8a3e1d", where "023" represents its sequential position in the original file.

[0097] Step S340: Cross combine the compressed and encrypted sub-blocks with the structured data block to generate cross-region distributed encrypted data units.

[0098] Cross-combination refers to the mixing and packaging of structured data blocks (such as database records) and unstructured compressed encrypted sub-blocks (such as video slices) into encrypted data units of a unified format according to preset rules. The specific operation includes: according to the priority weight of the dynamic storage path (such as the weight of the US West partition 0.8 and the weight of the Tokyo partition 0.6), a main storage partition is allocated for each structured data block, and it is bound with unstructured sub-blocks belonging to the same logical group; then, a cross-index table is inserted during packaging (such as record "structured data block ID-001" associated with "video sub-block 023" and "video sub-block 045"), which ensures that the associated data can be quickly located during subsequent access. For example, a structured data block (ID-001) containing sales order records may be combined with three video feedback sub-blocks (023, 045, 067) into an encrypted data unit, which is marked as "unit ID-7a3b" and stored in the US West, Tokyo and Frankfurt partitions.

[0099] Step S350: According to the priority weight of the dynamic storage path, the encrypted data unit is sequentially mapped to the corresponding cloud storage partition.

[0100] The process of mapping the encrypted data unit to the cloud storage partition needs to strictly follow the priority weight and area binding strategy defined in the dynamic storage path. The priority weight is used to indicate the priority level of different partitions in the competition for storage resources (such as the partition with weight 0.8 can preferentially occupy high-bandwidth links), and the area binding strategy specifies the mandatory storage location of a specific type of data unit (such as financial transaction data must be stored in a partition that meets the requirements of GDPR). In specific operation, the system queries the dynamic storage path configuration table according to the type label of the encrypted data unit (such as "high security level-video data"), matches the target partition list that meets the conditions, and arranges them in descending order of priority weight (such as Frankfurt partition 0.9, Singapore partition 0.7); then, the encrypted data unit is attempted to be written into the target partition in the list order, and if the target partition is insufficient in resources (such as the storage space is below the threshold), it is automatically downgraded to the sub-optimal partition. For example, an encrypted data unit marked as "high security level" may be preferentially mapped to the Zurich partition (weight 0.95), and if the partition is full, it is transferred to the Dublin partition (weight 0.85).

[0101] Step S400: Dynamically allocate the storage resources of the cloud storage partition through the load balancing algorithm, and adjust the priority weight of the dynamic storage path according to the real-time traffic characteristics.

[0102] The load balancing algorithm is a decision mechanism for optimizing resource allocation, and its core function is to dynamically adjust the storage location and connection number allocation ratio of data shards according to the real-time load state of cloud storage partitions (such as CPU utilization, storage I / O throughput, network bandwidth occupancy). The process of dynamically allocating storage resources includes: periodically collecting performance indicators of each partition (such as current connection number, data throughput fluctuation range), generating partition load scores through a weighted calculation formula (such as load balancing coefficient = 0.6 × connection number stability factor + 0.4 × throughput fluctuation factor), and migrating part of the storage tasks of high-load partitions to low-load partitions according to the score results. In this process, real-time traffic characteristics refer to the instantaneous data flow attributes (such as request rate, data packet size distribution, transmission protocol type) captured from the network transmission layer, which are used to dynamically adjust the priority weight of the storage path. For example, when a sudden surge in HTTP request rate is detected in a certain partition, the weight of its corresponding dynamic storage path may be increased from 0.7 to 0.9 to prioritize high-traffic requests.

[0103] The operation of adjusting the priority weight needs to perform the following steps: first, identify key patterns in real-time traffic (such as burst access peaks, persistent low-bandwidth transmission) through a traffic analysis engine; second, input the pattern characteristics into the storage path optimization model to recalculate the weight values of each path; finally, update the dynamic storage path configuration table to reflect the new weight allocation. For example, if the access request of the US West cloud storage partition increases due to a promotion activity, the load balancing algorithm may lower its path weight from 0.8 to 0.5, and divert part of the request to the Singapore cloud storage partition whose weight is raised to 0.7, while delaying the transmission of non-urgent data shards to low-peak periods.

[0104] As an implementation, step S400, dynamically allocating storage resources of cloud storage partitions through a load balancing algorithm, can specifically include:

[0105] Step S410: Real-time monitoring of the current connection number and data throughput of each cloud storage partition.

[0106] The current connection number refers to the number of data access sessions simultaneously processed by a specific cloud storage partition within a real-time monitoring window (e.g., 1 second granularity), including active read-write request connections, data transmission connections, and metadata query connections; the data throughput refers to the volume of data traffic passing through the partition per unit time (e.g., MB / s or GB / min), including the sum of uplink (write) and downlink (read) traffic. Real-time monitoring is achieved through monitoring agents deployed in each partition, which periodically collect connection number counters (e.g., ESTABLISHED count in the TCP connection state table) and throughput meters (e.g., network interface layer byte transmission statistics), and aggregate data to a central load balancing controller. For example, the monitoring agent of the Mexico partition collects the current connection number (e.g., 2450) and data throughput (e.g., 1.2 GB / s) every 500 ms, and uploads the indicators to the controller database via HTTPS protocol.

[0107] Step S420: Calculate the load balancing coefficient of each cloud storage partition based on the current connection number and data throughput.

[0108] The load balancing coefficient is a normalized value used to quantify the load state of the partition, which integrates connection number stability and throughput fluctuation in the calculation process. Specifically, the system first obtains the historical connection number sequence of the partition (e.g., concurrent connection number records every minute in the past 24 hours) and the current data throughput fluctuation range (e.g., the difference between the maximum and minimum values of throughput in the last 5 minutes), generates a first load factor reflecting connection number stability (e.g., coefficient of variation 0.15) and a second load factor reflecting throughput variation amplitude (e.g., range 0.4) through statistical analysis. Subsequently, the two factors are input into a weight fusion model, which determines the fusion proportion (e.g., solid state disk weight proportion 0.6, mechanical hard disk proportion 0.4) based on the storage resource type of the partition (e.g., solid state disk storage resource type or mechanical hard disk storage resource type) and the priority weight of the dynamic storage path (e.g., Mexico partition weight 0.8), and outputs the initial load coefficient (e.g., 0.72). Finally, the initial load coefficient is mapped to a normalized load coefficient (e.g., 0.68) within the range of 0-1 through a dynamic range compression function (e.g., Sigmoid function), serving as the final evaluation indicator of the partition load state.

[0109] As an implementation, step S420, calculating the load balancing coefficient of each cloud storage partition, can specifically include:

[0110] Step S421: Obtain the historical connection number sequence and the current data throughput fluctuation range of the cloud storage partition; the historical connection number sequence contains concurrent connection number statistics in multiple time windows.

[0111] The historical connection number sequence is a time series dataset extracted from the partition monitoring log, and its time window granularity can be configured to be minute-level (such as recording the number of concurrent connections every minute) or hour-level (such as recording the peak connection number every hour). For example, the historical connection number sequence of the US West partition may contain 1440 records (24 hours x 60 minutes), each record containing a timestamp (such as 2023-10-05 14:30:00) and a corresponding number of concurrent connections (such as 2350). The current data throughput fluctuation range is calculated by a sliding time window (such as the last 5 minutes), taking the maximum value (such as 1.5 GB / s) and the minimum value (such as 0.9 GB / s) of the throughput in the window, and the difference between them is the fluctuation range (0.6 GB / s).

[0112] Step S422: According to the distribution characteristics of each statistical value in the historical connection number sequence, a first load factor reflecting the stability of the connection number is generated.

[0113] The quantification of the stability of the connection number is realized by analyzing the statistical distribution of the historical connection number sequence, specifically including calculating the variance, standard deviation or coefficient of variation (standard deviation / mean) of the sequence. For example, if the historical connection number of the US West partition has a mean of 2000 and a standard deviation of 150, then the coefficient of variation is 150 / 2000 = 0.075, and the smaller the coefficient of variation, the more stable the connection number fluctuation. The first load factor can be designed as an inverse proportional function of the coefficient of variation (such as 1-coefficient of variation), when the coefficient of variation is 0.075, the first load factor is 0.925, indicating high stability.

[0114] Step S423: Based on the difference between the upper and lower limits of the current data throughput fluctuation range, a second load factor reflecting the change amplitude of the throughput is generated.

[0115] The change amplitude of the throughput is calculated by the ratio of the difference between the upper and lower limits of the current data throughput fluctuation range (such as 0.6 GB / s) and the maximum theoretical throughput of the partition (such as 10 GB / s). For example, if the difference is 0.6 GB / s and the maximum theoretical throughput is 10 GB / s, then the change amplitude is 0.6 / 10 = 0.06. The second load factor can be designed as an inverse proportional function of the amplitude value (such as 1-amplitude value), when the amplitude value is 0.06, the second load factor is 0.94, indicating that the throughput fluctuation is small and the system load is stable.

[0116] Step S424: The first load factor and the second load factor are input into a pre-set weight fusion model to generate an initial load coefficient; the weight fusion model determines the fusion proportion based on the priority weight of the storage resource type and the dynamic storage path of the cloud storage partition.

[0117] The weight fusion model dynamically adjusts the weight proportion of the first load factor and the second load factor according to the storage resource type of the partition. For example, for the partition of the solid state disk storage resource type, the model allocates the first load factor weight as 0.7 and the second load factor weight as 0.3 because the low latency characteristic is more sensitive to the stability of the number of connections; and for the partition of the mechanical hard disk storage resource type, the model allocates the first load factor weight as 0.4 and the second load factor weight as 0.6 because the throughput limitation is more significant. If the first load factor of the Mexico partition (solid state disk) is 0.925 and the second load factor is 0.94, then the initial load coefficient = 0.7*0.925+0.3*0.94=0.9295.

[0118] Step S425: dynamically range compressing the initial load coefficient to generate a normalized load coefficient; the strength of the dynamic range compression is dynamically adjusted according to the regional security level of the cloud storage partition.

[0119] The dynamic range compression adopts a nonlinear function (such as a logarithmic function or a piecewise linear function) to map the initial load coefficient to the range of 0-1. For example, when the regional security level is “high”, a strong compression strategy (such as the initial load coefficient 0.9295 is mapped to 0.72 by the Sigmoid function) is adopted to avoid frequent migration triggered by slight fluctuations; when the security level is “medium”, a linear mapping (such as 0.9295 is directly used as the normalized value) is adopted. This design ensures that the load evaluation of the high security level partition is more conservative, reducing the risk of misjudgment.

[0120] Step S426: determining the division boundary of the saturated load partition and the non-saturated load partition according to the comparison result of the normalized load coefficient and the preset load threshold; wherein the division boundary is used to control the migration proportion of the number of concurrent connections and the update amplitude of the priority weight.

[0121] The preset load threshold is set according to the hardware performance and business demand of the partition, for example, the threshold of the solid state disk storage resource type partition is set as 0.8, and the threshold of the mechanical hard disk storage resource type partition is set as 0.7. If the normalized load coefficient of the Mexico partition is 0.72 and the threshold is set as 0.8, the partition is marked as a non-saturated load partition; if the normalized load coefficient of the Frankfurt partition is 0.85 (threshold 0.8), the partition is marked as a saturated load partition. The division boundary further affects the calculation logic of the migration proportion: when the coefficient exceeds the threshold, the exceeding part is determined by a linear proportion (such as (0.85-0.8) / 0.85≈5.88%) or a nonlinear proportion (such as a square relationship) to determine the migration amount, and the update amplitude of the priority weight is proportional to the exceeding value (such as reducing the weight by 0.5% for every 0.01 exceeding).

[0122] Step S430: sorting the load balancing coefficients by the load balancing algorithm to determine the saturated load partition and the non-saturated load partition.

[0123] The load balancing algorithm ranks all partitions in descending order according to the normalized load coefficient, and divides the saturated load partitions (coefficient ≥ 0.75) and the non-saturated load partitions (coefficient < 0.75) based on a preset load threshold (such as 0.75). For example, if the normalized load coefficient of the Frankfurt partition is 0.82 and the Tokyo partition is 0.68, the former is marked as a saturated load partition and the latter is a non-saturated load partition. The hardware performance difference of the partition needs to be considered in the division process: for the partition configured with high-performance solid-state disk storage resource type, the load threshold can be appropriately floated (such as 0.8); while the threshold of the partition with mechanical hard disk storage resource type is adjusted downward (such as 0.7), in order to avoid performance degradation caused by hardware bottlenecks.

[0124] Step S440: The number of concurrent connections in the saturated load partition is proportionally migrated to the non-saturated load partition, and the priority weight of the dynamic storage path is updated.

[0125] The proportion of concurrent connection migration is determined by the amplitude of the load coefficient of the saturated partition exceeding the threshold. For example, if the normalized load coefficient of the Mexico partition is 0.85 (threshold 0.75), the exceeding amplitude is 0.1, and the migration proportion is the ratio of the exceeding amplitude to the total load capacity (such as 0.1 / 0.85≈11.76%). The migration operation is realized by the load balancing controller sending a redirection instruction to the client: for newly accessed connection requests, directly assign to the non-saturated partition; for established connections, gradually guide their session transfer to the target partition. At the same time, the priority weight of the dynamic storage path is dynamically adjusted according to the migration proportion: if the original weight of the Mexico partition is 0.8, after migrating 11.76% of the connections, its weight decreases to 0.8×(1-0.1176) = 0.706, while the weight of the Singapore partition increases from 0.6 to 0.6+0.1176×0.8 = 0.694, to reflect the rebalancing of resource allocation.

[0126] Step S450: According to the updated priority weight, the storage path of the encrypted data unit is re-allocated to realize the elastic expansion of the cloud storage partition.

[0127] The re-allocation of the storage path needs to be combined with the updated priority weight and the real-time available capacity of the partition. For example, if the weight of the Singapore partition increases to 0.694 and its remaining storage capacity is 5TB, while the volume of the newly generated encrypted data unit is 200GB, the system preferentially allocates it to the Singapore partition, and updates the region binding relationship in the shard index. The elastic expansion mechanism is automatically triggered in this process: when the resource utilization of the non-saturated partition exceeds the preset expansion threshold (such as 80%), the system dynamically applies for additional storage resources (such as expanding 10% of the storage cluster nodes) through the cloud service API, to ensure that the migrated connections and data load can be effectively carried.

[0128] Step S500: When detecting an access request of an encrypted data unit, re-allocate the concurrent connection number of the cloud storage partition according to the updated priority weight and the storage path optimization model, to realize elastic expansion.

[0129] The access request refers to the read or write operation instruction initiated by the client or application to a specific encrypted data unit, which contains the target data identifier (such as the shard index hash value), the access area identifier (such as the user geographic location code), and the operation type label (such as the read / write permission level). The process of re-allocating the concurrent connection number needs to combine the updated priority weight (reflecting the current load status of each partition) and the prediction result of the storage path optimization model (such as the connection number growth trend in the next 5 minutes), to dynamically adjust the maximum connection number threshold and resource quota of the partition through the elastic expansion controller. For example, when the storage path optimization model predicts that the Frankfurt partition will receive more than 5000 concurrent connections in the next 10 minutes, the elastic expansion controller can temporarily expand the number of virtual server instances of the partition, and increase the maximum connection number from 3000 to 8000 to cope with the peak pressure.

[0130] The specific process of realizing elastic expansion includes: first, parsing the target data identifier in the access request, locating the corresponding cloud storage partition through the encrypted shard index; second, calculating the additional concurrent connection quota according to the current connection number, historical connection number peak of the partition, and the expansion suggestion generated by the storage path optimization model; finally, dynamically adjusting the partition resources (such as increasing load balancer nodes, expanding storage cluster size) through the API interface of the cloud service provider. In this process, if the overflow risk level of the partition exceeds the preset threshold (such as the connection number occupancy rate exceeding 95%), the system will automatically create temporary storage nodes to handle excess requests, and gradually release redundant resources after the traffic falls back. For example, when the Tokyo cloud storage partition encounters a DDoS attack, it may trigger the elastic expansion mechanism to expand the concurrent connection number from 2000 to 10000, and restore the connection number to the normal level through reverse merging operation after the attack ends.

[0131] As an implementation, step S500, when detecting an access request of an encrypted data unit, re-allocating the concurrent connection number of the cloud storage partition according to the updated priority weight and the storage path optimization model, can specifically include:

[0132] Step S510: Parsing the target data identifier and access area information in the access request.

[0133] An access request is a read or write operation instruction initiated by a client or application targeting a specific encrypted data unit, whose packet header contains the target data identifier and access region information. The target data identifier is composed of the globally unique identifier (e.g., UUID) of the encrypted data unit and the shard index hash value (e.g., SHA-256 hash), which uniquely identifies the target data unit and its belonging shard set; the access region information is determined by the geographical location resolution of the client IP address or the user-specified region code (e.g., "EU-Central-1"), which indicates the logical or physical region attribution of the request source. The resolution process needs to be implemented through a request resolution engine: first, extract the target data identifier field (e.g., "DataID = 7a3b-c41d") from the packet header, and split it into UUID (7a3b) and shard index hash (c41d) according to the preset format; second, convert the client IP (e.g., 203.0.113.45) to the standard region identifier (e.g., "AP-Southeast-1") through the IP address library or region code mapping table. For example, a video access request from a Singapore user generates the target data identifier "7a3b-c41d" and the access region information "AP-Southeast-1" after resolution, which are used for subsequent path decision.

[0134] Step S520: Locate the corresponding encrypted data unit from the encrypted shard index according to the target data identifier, and obtain the storage path history record of the encrypted data unit.

[0135] The encrypted shard index is a distributed key-value storage structure, with the shard index hash value as the key and the metadata set (e.g., shard storage location, access key, history path record) of the encrypted data unit as the value. When locating the encrypted data unit, the system queries the encrypted shard index through the shard index hash value (c41d) in the target data identifier, and obtains the metadata of the data unit belonging to the shard (e.g., the main storage partition is "AP-Southeast-1", and the backup shard is located in "EU-Central-1"). The storage path history record is a log data set extracted from the metadata, describing the historical storage location changes of the data unit, containing the timestamp (e.g., 2023-10-05 14:30:00), the source partition identifier (e.g., "AP-Southeast-1"), the target partition identifier (e.g., "EU-Central-1"), the migration reason (e.g., load balancing), and the performance indicators after migration (e.g., the response delay decreases from 120 ms to 80 ms). For example, the history record of the encrypted data unit "7a3b" shows that it has been migrated to the European partition three times in the past week due to high access peak in the Asia-Pacific region, and the delay decreases by about 30% after each migration.

[0136] Step S530: Based on the storage path history record and the access region information, a new priority weight is generated through the storage path optimization model.

[0137] When generating the new priority weight, the storage path optimization model performs multi-dimensional correlation analysis on the performance indicators (such as average delay, throughput fluctuation) in the storage path history record and the geographical location features (such as inter-regional network delay, compliance constraints) in the access region information. For example, if the access region is "AP-Southeast-1" and the history record shows that the cross-continental transmission delay from this region to "EU-Central-1" is high (200 ms), the model may reduce the priority weight of the European sub-region (from 0.7 to 0.5) and increase the weight of the local sub-region in the Asia-Pacific region (from 0.8 to 0.9). In the specific calculation process, the model quantifies the influence coefficient of inter-regional network quality on the weight through a logistic regression algorithm, and dynamically adjusts the output value by combining real-time load data (such as the current connection number of the European sub-region is 3800), ensuring that the weight allocation not only conforms to historical laws but also adapts to real-time state.

[0138] Step S540: Adjust the storage path distribution of the encrypted data unit according to the new priority weight, and re-allocate the concurrent connection number of the cloud storage sub-region.

[0139] Adjusting the storage path distribution requires updating the priority weight field in the dynamic storage path configuration table and triggering the sharding migration task. For example, when the Asia-Pacific sub-region weight of the encrypted data unit "7a3b" increases to 0.9, the system migrates its primary storage copy from the European sub-region back to the Singapore sub-region and updates the region binding relationship in the shard index. When re-allocating the concurrent connection number, the load balancing controller calculates the connection quota for each sub-region according to the new weight: if the Singapore sub-region weight 0.9 corresponds to a 70% connection number quota (such as 3500 out of a total concurrent connection number of 5000), and the European sub-region weight 0.5 corresponds to a 30% quota (1500), the system will migrate the excess connections (such as the current connection number 2000) in the European sub-region by the proportion (500 connections are migrated to the Singapore sub-region). The migration operation is implemented through a session redirection protocol (such as HTTP 307 temporary redirection), ensuring seamless switching of the client to the new sub-region.

[0140] Step S550: When the concurrent connection number exceeds the elastic expansion threshold, create a temporary storage node to handle excess connection requests.

[0141] The elastic expansion threshold is the upper limit of the number of connections preset according to the hardware specifications and business needs of the partition (for example, the threshold for a solid state disk storage resource type partition is 5000, and the threshold for a mechanical hard disk storage resource type is 3000). When it is detected that the number of concurrent connections of a certain partition exceeds the threshold (for example, the number of connections of the Singapore partition reaches 5100), the system automatically calls the API interface of the cloud service provider to create a temporary storage node. The temporary storage node uses an on-demand billing elastic instance (such as an AWS EC2 Spot instance), and its configuration (such as the number of vCPUs and the size of memory) is dynamically determined according to the excess number of connections (such as 2-core 4-GB instances are deployed for 100 excess connections). After the node is created, the system routes the excess connection requests (100 connections) to the temporary node and marks the life cycle of the node in the node metadata (such as it is automatically released after 12 hours). For example, during the “Double 11” promotion, the number of connections of the Singapore partition increases to 6000, and the system creates 3 temporary nodes (each of which handles 300 connections), ensuring that the number of connections of the main partition is stable below 5100, and gradually releasing the temporary resources after the traffic falls.

[0142] In an optional derivative embodiment, after reallocating the number of concurrent connections of the cloud storage partition in step S500, the method can further include:

[0143] Step S600: Monitor the real-time response delay and throughput fluctuation characteristics of the cloud storage partition after elastic expansion, and generate an expansion performance index set.

[0144] The real-time response delay refers to the time interval consumed from receiving a request to returning a response when the cloud storage partition after elastic expansion processes a data access request (for example, the average delay of the US West partition increases from 50 ms to 70 ms), and its monitoring process is achieved by periodically sending simulated requests (such as HTTP GET / POST) to the target partition through distributed probe nodes and recording the round-trip time; the throughput fluctuation characteristics refer to the variation amplitude of the partition data flow per unit time (for example, the throughput of the Singapore partition increases from 1.2 GB / s to 2.5 GB / s and then falls to 1.8 GB / s within 5 minutes), which is calculated by statistical maximum, minimum and standard deviation of the sliding time window (such as 10 seconds granularity). When generating the expansion performance index set, the system combines the time series data of the real-time response delay and the throughput fluctuation range into a structured data set, and adds a timestamp, a partition identifier, and an expansion operation type label (such as “temporary node expansion”). For example, the expansion performance index set of the Frankfurt partition after elastic expansion can include the fields:

[0145] {

[0146] "partition ID": "eu-central-1",

[0147] "average delay": "75ms",

[0148] "Throughput fluctuation range": "1.5-2.3GB / s",

[0149] "Expansion type": "Temporary node expansion",

[0150] "Time range": "2023-10-05 15:00:00 to 15:05:00"

[0151] }

[0152] Step S700: Adjust the priority weight update frequency in the storage path optimization model according to the response delay threshold and throughput fluctuation range in the expansion performance indicator set.

[0153] The response delay threshold is the maximum allowed delay value (e.g., 100ms) preset according to the service level agreement (SLA), and the throughput fluctuation range threshold is defined as the allowed throughput range percentage (e.g., no more than 30% of the maximum theoretical throughput of the partition). When adjusting the priority weight update frequency, the system compares the expansion performance indicators with the thresholds: if the real-time response delay continuously exceeds the threshold (e.g., delay ≥ 110ms for three consecutive monitoring periods), the priority weight update frequency is increased (e.g., from once every hour to once every 10 minutes) to accelerate path optimization response; if the throughput fluctuation range is below the threshold (e.g., range is 20%), the update frequency is reduced (e.g., from once every 15 minutes to once every 30 minutes) to reduce model calculation overhead. For example, after temporary node expansion, the throughput fluctuation range of the Tokyo partition decreases to 15%, and the system adjusts its weight update frequency from once every 10 minutes to once every 20 minutes while maintaining the response delay threshold at 90ms.

[0154] Step S800: Input the adjusted priority weight update frequency into the storage path optimization model to recalculate the encryption shard index mapping relationship of the dynamic storage path.

[0155] The storage path optimization model dynamically adjusts its internal time step control logic according to the new update frequency parameter. For example, when the update frequency is increased to once every 10 minutes, the model shortens the time window of historical data (e.g., from 24 hours to 6 hours) and increases the weight proportion of real-time features (e.g., current connection number) to 70% to respond faster to recent load changes. When recalculating the encryption shard index mapping relationship, the model combines the latest priority weights (e.g., Singapore partition weight increases from 0.7 to 0.85) and partition available capacity (e.g., remaining storage space 5TB) to generate a new shard index binding relationship. For example, the encryption data unit "7a3b" originally mapped to the Frankfurt partition is remapped to the Singapore partition due to its weight decreasing to 0.6, and the region code field in the shard index is updated to "ap-southeast-1".

[0156] Step S900: Based on the updated encrypted shard index mapping relationship, the encrypted data units of the cloud storage partition are redistributed across regions, and the shard location metadata of the virtualized storage node is synchronously updated.

[0157] Cross-region redistribution refers to migrating encrypted data units from the original storage partition to the target partition according to the new encrypted shard index mapping relationship. The migration operation is performed through the shard transmission service: first, read the shard content of the encrypted data unit (such as the encrypted data block of shard 023) from the original partition; second, verify the shard integrity (such as checking the SHA-256 hash value); finally, write the shard to the target partition and update the metadata of the virtualized storage node (such as changing the location field of shard 023 from "eu-central-1" to "ap-southeast-1"). During synchronization, the system uses a two-phase commit protocol to ensure data consistency: lock the original shard to prevent writing in the preparation phase, and release the original shard resources after confirming that the target partition has been successfully written in the commit phase. For example, after the 200 shards of encrypted data unit "7a3b" are migrated from the Frankfurt partition to the Singapore partition, the "storage path" field in its metadata is batch updated, and the migration timestamp and operation ID are recorded for auditing.

[0158] As an implementation, step S900, the encrypted data units of the cloud storage partition are redistributed across regions, which can specifically include:

[0159] Step S910: Extract the access frequency features and regional security level trend of the encrypted data unit.

[0160] Access frequency features refer to the number distribution pattern of encrypted data units accessed within a specified time window (such as 24 hours) (such as shard 023's access volume in the Asia-Pacific region working hours (09:00-18:00) accounts for 80%), which is generated by statistical request logs of each shard; the regional security level trend refers to the dynamic adjustment record of the security policy of the storage partition where the shard is located (such as the data encryption standard of the Singapore partition is upgraded from AES-128 to AES-256), which is extracted from security policy update logs and compliance audit reports. For example, the access frequency features of encrypted data unit "7a3b" show that its access volume in the early morning hours (00:00-06:00) in the European region drops by 60%, and the regional security level trend shows that the security level of the Frankfurt partition is reduced from "high" to "medium" due to GDPR audit.

[0161] Step S920: Generate hot region labels based on access frequency features, and perform matching degree calculation between regional security level trend and encrypted shard index of dynamic storage path.

[0162] Hotspot region marking is used to identify logical or physical regions with access frequency exceeding a preset threshold (e.g., daily access volume exceeding 10,000 times). The matching degree calculation is achieved by comparing the region security level trend with the security requirements in the encryption shard index: if the shard index requires a security level of "high" but the current level of the region is "medium", the matching degree score is reduced (e.g., from 0.9 to 0.6). The calculation can use cosine similarity or Euclidean distance algorithm, for example, when the security level of the Singapore partition changes from "high" to "medium", the matching degree with the shard index requirements decreases from 0.95 to 0.75.

[0163] Step S930: When the matching degree calculation result is lower than the preset security threshold, filter out the encryption shard index with low security level from the virtualized storage node, and trigger the shard encryption protocol to perform key rotation on the encryption shard index.

[0164] The preset security threshold is usually set to 0.7, when the matching degree is lower than this value, the system scans the virtualized storage node to filter out all encryption shard indexes bound to low security level partitions (e.g., shard 023, 045 of Singapore partition). The key rotation operation is performed by the key management service in the shard encryption protocol: first, generate a new key pair (e.g., RSA-2048 public / private key); second, re-encrypt the shard data using the new key; finally, update the key identifier field in the shard index. For example, the original key "KeyID-001" of shard 023 is replaced by "KeyID-002", and the rotation timestamp and operator ID are recorded in the shard header.

[0165] Step S940: Based on the encryption shard index after key rotation, perform dynamic masking processing on the hotspot region marking of the encryption data unit to generate a temporary storage path.

[0166] Dynamic masking processing refers to temporarily removing or replacing the hotspot region marking from the shard metadata with a blurred identifier (e.g., changing "ap-southeast-1" to "region-xx") to prevent malicious attackers from inferring the location of critical data through access patterns. When generating a temporary storage path, the system binds the masked region marking with a backup partition (e.g., Mumbai partition) and sets the path validity period (e.g., 24 hours). For example, the temporary storage path of encryption data unit "7a3b" points to the Mumbai partition, and the region field in its metadata is masked as "region-5a", and only the real partition mapping is retained in the internal routing table.

[0167] Step S950: Superimpose the temporary storage path with the conflict avoidance path for verification, and update the priority weight allocation strategy of the cloud storage partition according to the verification result.

[0168] The superposition verification refers to simultaneously applying the temporary storage path and the conflict avoidance path, and testing the performance of the system in the mixed routing state. The verification process includes: 1) distributing test requests to the temporary path and the conflict avoidance path (such as allocating 50% traffic to each); 2) monitoring response delay, throughput and error rate; 3) if the performance of the temporary path meets the standard (such as delay ≤ 120 ms), the weight of the temporary path is increased to 0.7, and the weight of the conflict avoidance path is reduced to 0.3; otherwise, it is rolled back to the original path configuration. For example, the temporary path of the Mumbai partition performs well in the test (delay 105 ms), and the system increases the weight of the temporary path from 0.5 to 0.7 and updates the priority weight allocation strategy document.

[0169] Step S1000: When detecting the access path conflict of the redistributed encrypted data unit, generating a conflict avoidance path according to the storage path optimization model, and triggering a load balancing algorithm to perform secondary allocation of the number of concurrent connections of the conflict path.

[0170] The access path conflict refers to that multiple clients simultaneously request different copy shards of the same encrypted data unit, causing the number of connections of the target partition to surge or data transmission congestion (such as shard 023 of the Singapore partition being accessed by 1000 concurrent connections at the same time). Conflict detection is achieved by real-time monitoring of the access request source of the shard and the load of the target partition: when the same shard receives more than a preset conflict threshold (such as 500 requests) within 1 second and the load coefficient of the target partition exceeds 0.8, it is marked as a conflict path. When generating a conflict avoidance path, the storage path optimization model analyzes the performance indicators of the candidate partitions (such as the delay of the Mumbai partition being 110 ms and the load coefficient being 0.65), selects the optimal standby partition, and generates temporary routing rules (such as diverting 50% of the requests to the Mumbai partition).

[0171] As an implementation, in step S1000, triggering the load balancing algorithm to perform secondary allocation of the number of concurrent connections of the conflict path, which can specifically include:

[0172] Step S1001: Identifying the target storage partition with the number of concurrent connections exceeding the limit in the conflict path, and obtaining the historical peak number of connections of the target storage partition and the current load proportion of the encrypted shard.

[0173] The identification of the target storage partition is based on real-time connection number monitoring data and elastic expansion threshold (such as the threshold of 5000 for the Singapore partition). When the number of concurrent connections reaches 5100, it is marked as an over-limit partition. The historical peak number of connections is extracted from the performance log (such as the highest number of connections in the past 7 days being 5200), and the current load proportion of the encrypted shard is calculated as the percentage of the number of occupied connections to the total connection quota (such as 5100 / 5000 = 102%). For example, the historical peak of the Singapore partition is 5200, the current load proportion is 102%, and the secondary allocation condition is triggered.

[0174] Step S1002: Generate a connection number prediction curve according to historical connection number peaks, and calculate the overflow risk level of the target storage partition in combination with the current encrypted shard load ratio.

[0175] The connection number prediction curve is generated by a time series prediction algorithm (such as ARIMA or LSTM), with historical connection number peaks and time characteristics (such as weekdays / holidays) as input, and the estimated connection number in the next hour (such as 5300 in 30 minutes) as output. The overflow risk level is calculated based on the predicted value and the current load ratio: if the predicted value exceeds 120% of the elastic expansion threshold (such as 5000 x 120% = 6000), the risk level is "high"; if it is between 110% and 120%, the risk level is "medium". For example, the predicted value of the Singapore partition is 5300 (106% threshold), and the risk level is marked as "medium".

[0176] Step S1003: When the overflow risk level exceeds the preset risk threshold, dynamically allocate standby storage partitions from the virtualized storage nodes, and perform hierarchical migration of the encrypted data units of the target storage partition according to the overflow risk level.

[0177] The preset risk threshold is usually set to "medium" level, and when the risk level is ≥ "medium", the system selects the standby partition with the lowest load coefficient from the virtualized storage node pool (such as the Mumbai partition with a load coefficient of 0.4). Hierarchical migration assigns priority according to the overflow risk level of the shard: high-risk shards (such as the top 10% of access frequency) are migrated first, followed by medium-risk shards. For example, the 200 high-risk shards of the Singapore partition are migrated to the Mumbai partition first, with a migration rate of 100 shards per minute.

[0178] Step S1004: During the hierarchical migration process, real-time adjust the encrypted shard index binding relationship of the standby storage partition according to the availability state of the temporary storage path.

[0179] The availability state of the temporary storage path is evaluated in real time by heartbeat detection and performance probes (such as the delay of the Mumbai partition decreasing from 130ms to 115ms). If the standby partition state is stable (such as continuous 5 times of detection delay ≤ 150ms), gradually increase its shard index binding number (such as from 50 to 150); if the state is abnormal (such as packet loss rate ≥ 5%), suspend migration and switch to the secondary standby partition (such as the Sydney partition). For example, after migrating 50 shards, the delay of the Mumbai partition stabilizes at 120ms, and the system increases the number of bound shards to 100 and updates the region code in the shard index.

[0180] Step S1005: When the overflow risk level of the target storage partition falls within the safe range, trigger the storage path optimization model to perform reverse merging on the encrypted data units of the standby storage partition, and release the redundant storage resources generated during the migration process.

[0181] Reverse merging refers to gradually migrating the encrypted data units temporarily migrated to the standby partition back to the target storage partition. When the overflow risk level of the Singapore partition falls to "low" (predicted connection number 4800), the system starts the migration task according to the shard priority (low risk before high risk), and releases the temporary storage instances of the standby partition (such as closing 3 EC2 Spot instances). For example, 150 shards of the Mumbai partition are batched back to the Singapore partition within 24 hours, and 5TB of storage space occupied by the temporary nodes is released after the migration is completed.

[0182] The embodiment of the present application also provides a data storage system, as shown in the figure, the data storage system 100 includes: a processor 101 and a memory 103. Wherein, the processor 101 and the memory 103 are connected, such as through the bus 102. Optionally, the data storage system 100 can also include a transceiver 104. It should be noted that in actual application, the transceiver 104 is not limited to one, and the structure of the data storage system 100 does not constitute a limitation on the embodiments of the present application. Figure 2

[0183] The embodiment of the present application provides a data storage system, and the data storage system in the embodiment of the present application comprises: one or more processors; a memory; one or more computer programs, wherein the one or more computer programs are stored in the memory and are configured to be executed by the one or more processors, and when the one or more programs are executed by the processor, the method provided in the above embodiment is realized.​

Claims

1. A cloud computing-based digital enterprise management data storage method, characterized in that, The method comprises: acquiring real-time management data sets in enterprise data flow, and generating a virtual storage node according to structured data blocks and unstructured data blocks in the real-time management data sets; the virtual storage node contains encrypted shard indexes associated with different regions, and each encrypted shard index corresponds to a dynamic storage path; determining a target encrypted shard index matched with the virtual storage node according to a preset storage path optimization model; the storage path optimization model is generated by training based on region access frequency and encryption level features in historical storage logs; based on the dynamic storage path corresponding to the target encrypted shard index, segmenting the structured data blocks and unstructured data blocks into cross-region distributed encrypted data units, and mapping the encrypted data units to corresponding cloud storage partitions, comprising: determining the number of encrypted shards of each cloud storage partition according to the region distribution characteristics of the dynamic storage path; generating a shard encryption protocol corresponding to each cloud storage partition based on the number of encrypted shards and the data type of the structured data blocks; generating a plurality of compressed encrypted sub-blocks by performing shard compression on the unstructured data blocks through the shard encryption protocol; cross-combining the compressed encrypted sub-blocks with the structured data blocks to generate the cross-region distributed encrypted data units; and sequentially mapping the encrypted data units to corresponding cloud storage partitions according to the priority weight of the dynamic storage path; dynamically allocating storage resources of the cloud storage partitions through a load balancing algorithm, and adjusting the priority weight of the dynamic storage path according to real-time traffic characteristics; when detecting an access request of the encrypted data units, reallocating the number of concurrent connections of the cloud storage partitions according to the updated priority weight and the storage path optimization model, to realize elastic expansion.

2. The method of claim 1, wherein, The training process of the storage path optimization model comprises the following steps: extracting a plurality of region association features from the historical storage logs; the region association features include region access frequency, encryption level features, storage delay parameters, and concurrent connection peak values; generating an initial training set according to the region access frequency and encryption level features, and inputting the initial training set into an initial path optimization network for first round training; in the first round training process, determining the node weight distribution of the initial path optimization network through a cross-validation algorithm, and constructing an intermediate path optimization model according to the node weight distribution; inputting the storage delay parameters and concurrent connection peak values into the intermediate path optimization model to generate a path optimization parameter set corresponding to the cloud storage partitions; the path optimization parameter set includes the priority weight of the dynamic storage path, the update period of the encrypted shard index, and the storage resource allocation ratio; performing back propagation optimization on the intermediate path optimization model according to the path optimization parameter set to generate the storage path optimization model; wherein the output of the storage path optimization model includes the priority weight of the dynamic storage path corresponding to the target encrypted shard index and the storage resource allocation ratio.

3. The method of claim 2, wherein, The determining the node weight distribution of the initial path optimization network through a cross-validation algorithm comprises: dividing the initial training set into multiple training subsets and validation subsets, and performing forward propagation training on the initial path optimization network based on the training subsets; after each forward propagation training, calculating the path prediction error of the initial path optimization network through the validation subsets, and adjusting the node weight distribution according to the path prediction error; when the path prediction error meets a preset convergence condition, determining the node weight distribution of the initial path optimization network as an optimal weight distribution; generating a weight update function corresponding to the encryption level feature according to the optimal weight distribution and the region access frequency in the initial training set; iteratively updating the node weights of the initial path optimization network through the weight update function until the path prediction error reaches a minimum threshold value; The back propagation optimization of the intermediate path optimization model according to the path optimization parameter set comprises: inputting the priority weight and storage resource allocation ratio in the path optimization parameter set to the hidden layer of the intermediate path optimization model to generate hidden layer output features; calculating the delay loss value of the intermediate path optimization model according to the hidden layer output features and the storage delay parameter; adjusting the hidden layer weight parameter of the intermediate path optimization model through a gradient descent algorithm for back propagation of the delay loss value; when the delay loss value reaches a preset optimization threshold value, fixing the hidden layer weight parameter of the intermediate path optimization model as an optimal weight parameter; generating the output layer parameter of the storage path optimization model according to the optimal weight parameter and the concurrent connection peak value, so that the output of the storage path optimization model matches the priority weight and storage resource allocation ratio of the dynamic storage path.

4. The method of claim 1, wherein, The generation of the fragment encryption protocol corresponding to each cloud storage partition based on the number of encryption fragments and the data type of the structured data block comprises: determining a data compression mode according to the data type of the structured data block, and selecting an encryption algorithm type matching the data compression mode from a preset encryption algorithm library; dividing the fragment size range of the unstructured data block based on the number of encryption fragments, and generating an initial fragment rule according to the fragment size range; performing encryption strength adaptation on the fragment size range in the initial fragment rule through the encryption algorithm type to generate an intermediate fragment rule; segmenting the unstructured data block into multiple initial encryption fragments according to the encryption strength adaptation result in the intermediate fragment rule; performing compression mode verification on the initial encryption fragments, and adjusting the matching relationship between the data compression mode and the fragment size range according to the verification result; generating a final fragment rule based on the adjusted matching relationship, and binding the final fragment rule with the encryption algorithm type to generate the fragment encryption protocol.

5. The method of claim 4, wherein, The generation process of the fragment encryption protocol further comprises dynamically adjusting the final fragment rule through a fragment optimization model, comprising: extracting, from the historical shard logs, shard performance features associated with the encryption algorithm type, the data compression mode, and the shard size range; inputting the shard performance features into the shard optimization model to generate shard adjustment parameters corresponding to the final shard rule; performing overlap region detection on the boundaries of the initial encrypted shards according to a shard overlap threshold in the shard adjustment parameters; when it is detected that the boundary overlap region of the initial encrypted shards exceeds the shard overlap threshold, recalculating the shard size range and updating the final shard rule; performing secondary shredding on the unstructured data blocks based on the updated final shard rule to generate an optimized encrypted shard set; based on the cross combination result of the optimized encrypted shard set and the structured data blocks, correcting the shard size range and the encryption algorithm type binding relationship in the shard encryption protocol.

6. The method of claim 1, wherein, The dynamic allocation of the storage resources of the cloud storage partitions through the load balancing algorithm comprises: real-time monitoring of the current connection number and data throughput of each cloud storage partition; calculating the load balancing coefficient of each cloud storage partition according to the current connection number and data throughput; determining the saturated load partition and the non-saturated load partition by sorting the load balancing coefficients through the load balancing algorithm; proportionally migrating the concurrent connection number of the saturated load partition to the non-saturated load partition, and updating the priority weight of the dynamic storage path; redistributing the storage path of the encrypted data unit according to the updated priority weight to realize the elastic expansion of the cloud storage partition.

7. The method of claim 6, wherein, The calculation of the load balancing coefficient of each cloud storage partition comprises: obtaining a historical connection number sequence and a current data throughput fluctuation range of the cloud storage partition; the historical connection number sequence contains concurrent connection number statistics values in multiple time windows; generating a first load factor reflecting the stability of the connection number according to the distribution characteristics of the statistics values in the historical connection number sequence; generating a second load factor reflecting the change amplitude of the throughput based on the upper and lower limit difference value of the current data throughput fluctuation range; inputting the first load factor and the second load factor into a preset weight fusion model to generate an initial load coefficient; the weight fusion model determines the fusion proportion based on the storage resource type of the cloud storage partition and the priority weight of the dynamic storage path; performing dynamic range compression on the initial load coefficient to generate a normalized load coefficient; the strength of the dynamic range compression is dynamically adjusted according to the regional security level of the cloud storage partition; determining the division boundary of the saturated load partition and the non-saturated load partition according to the comparison result of the normalized load coefficient and a preset load threshold; wherein the division boundary is used to control the migration proportion of the concurrent connection number and the update amplitude of the priority weight.

8. The method of claim 1, wherein, When the access request of the encrypted data unit is detected, the concurrent connection number of the cloud storage partition is redistributed according to the updated priority weight and the storage path optimization model, comprising: parsing the target data identifier and access region information in the access request; According to the target data, an encrypted data unit corresponding to the encrypted fragment index is located, and a storage path history record of the encrypted data unit is obtained; Based on the storage path history record and access region information, a new priority weight is generated by the storage path optimization model; According to the new priority weight, the storage path distribution of the encrypted data unit is adjusted, and the concurrent connection number of the cloud storage partition is re-allocated; When the concurrent connection number exceeds an elastic expansion threshold, a temporary storage node is created to accommodate excess connection requests.

9. A data storage system, characterized by Comprise: One or more processors; Memory; One or more computer programs; wherein the one or more computer programs are stored in the memory and are configured to be executed by the one or more processors, when the one or more computer programs are executed by the processor, the method as claimed in any one of claims 1~8 is realized.

Images