A data private storage and access method, medium and system of a public cloud service

Abstract

The application provides a data private storage and access method, medium and system of public cloud services, and belongs to the technical field of electric digital data processing. The application collects data access and modification frequency information, calculates a resource consumption vector, performs feature analysis by using a data classification layer and a feature extraction layer, obtains a high-low frequency division matrix through a matrix decomposition layer, calculates a data redundant storage index in combination with a resource evaluation layer, establishes a comprehensive judgment function to optimize a storage strategy, and finally realizes multi-level storage allocation and access control of data. The application can more accurately identify data access features, more reasonably allocate storage resources, and more efficiently manage data access processes, and solves the technical problem in the prior art that public cloud service data cannot be dynamically optimized and allocated according to access features and resource consumption in a multi-level storage system.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of digital data processing, and particularly relates to a data private storage and access method, medium and system of a public cloud service. BACKGROUND

[0002] In the field of public cloud computing, data storage and access control is one of the core technologies. Traditional public cloud storage systems mainly adopt a hierarchical storage architecture, including multiple levels such as cache, main storage and auxiliary storage. This hierarchical storage architecture allocates data through preset storage strategies. Common allocation methods include hot data hierarchical storage based on access frequency, hierarchical storage based on data importance, and hierarchical storage based on data life cycle, etc. These methods have formed a relatively perfect technical system in practical application and are widely used in public cloud platforms. For example, mainstream public cloud platforms such as Ali Cloud and Tencent Cloud adopt multi-level storage architecture to meet the data storage needs of different application scenarios.

[0003] However, with the rapid development of cloud computing technology, traditional data storage allocation methods face many challenges. First, existing storage allocation strategies are often static and cannot be adjusted in time according to the dynamic changes of data access patterns. Second, traditional methods mainly consider a single dimension of features (such as access frequency) when allocating storage, ignoring resource consumption characteristics during data access, resulting in low storage resource utilization efficiency. Third, existing technologies lack deep analysis capabilities for data access and modification behavior, and cannot accurately identify data access patterns and resource consumption patterns. In addition, in terms of data redundancy storage, existing technologies often use fixed redundancy strategies and fail to dynamically optimize according to data characteristics and system resource conditions.

[0004] These problems make it difficult for public cloud platforms to achieve optimal utilization of resources when allocating data storage. Especially in large-scale data processing scenarios, how to dynamically optimize allocation according to data access characteristics and resource consumption has become a technical problem that needs to be solved. Existing technologies cannot effectively solve the contradiction between data storage allocation and resource utilization, and cannot meet the dual needs of data access performance and resource utilization efficiency. That is, there is a technical problem in existing technologies that public cloud service data cannot be dynamically optimized and allocated in a multi-level storage system according to access characteristics and resource consumption. SUMMARY

[0005] Therefore, the application provides a data private storage and access method, medium and system of a public cloud service, which can solve the technical problem that public cloud service data cannot be dynamically optimized and allocated in a multi-level storage system according to access characteristics and resource consumption in existing technologies.

[0006] The application is implemented in the following manner: a first aspect of the application provides a data private storage and access method for public cloud services, including the following steps: establishing and training a pyramid structure neural network model, the pyramid structure neural network model including a data classification layer, a feature extraction layer, a matrix decomposition layer, and a resource evaluation layer; collecting data access frequency information and data modification frequency information in the public cloud service; calculating unit data storage resource consumption vectors, unit data reading resource consumption vectors, and unit data modification resource consumption vectors; using the pyramid structure neural network model to classify and process the data access frequency information and the data modification frequency information, generating a data access frequency matrix and a data modification frequency matrix; extracting features from the data access frequency matrix and the data modification frequency matrix to obtain an access frequency feature vector and a modification frequency feature vector; performing matrix decomposition operations using the matrix decomposition layer to obtain high-frequency access sub-matrices, low-frequency access sub-matrices, high-frequency modification sub-matrices, and low-frequency modification sub-matrices; calculating a data redundancy storage index and establishing a decision function, and performing multi-level storage allocation according to the calculation results of the decision function; establishing a data storage index table and implementing private storage and access control of data.

[0007] The specific structure of the pyramid structure neural network model includes: the data classification layer uses a one-hot encoding method to process input data, and converts the data access frequency information and the data modification frequency information into standard numerical features; the feature extraction layer includes parallel long short-term memory network units and convolutional neural network units, the long short-term memory network units are used to extract time sequence features, and the convolutional neural network units are used to extract spatial features; the matrix decomposition layer includes a first layer pyramid structure and a second layer pyramid structure, the first layer pyramid structure is used to preliminarily decompose the access frequency feature vector and the modification frequency feature vector, and the second layer pyramid structure is used to deeply decompose the preliminarily decomposed matrix; the resource evaluation layer is used to evaluate resource consumption in the data storage and access process.

[0008] The data access frequency information includes data access times, access time intervals, access concurrency, and data distribution amounts, and the data modification frequency information includes data modification times, modification time intervals, modification concurrency, and data version numbers.

[0009] The data access frequency matrix includes time dimensions, user dimensions, and operation dimensions, and the data modification frequency matrix includes time dimensions, user dimensions, and modification dimensions.

[0010] The determination function includes a threshold determination unit, a weight calculation unit and an allocation decision unit, and the input parameters of the determination function include the access high-frequency sub-matrix, the access low-frequency sub-matrix, the modified high-frequency sub-matrix, the modified low-frequency sub-matrix, the unit data storage resource consumption vector, the unit data reading resource consumption vector, the unit data modification resource consumption vector and the data redundancy storage index.

[0011] The multi-level storage allocation includes a cache layer, a main storage layer and an auxiliary storage layer, wherein the cache layer is used for storing data with the highest access frequency, the main storage layer is used for storing data with medium access frequency, and the auxiliary storage layer is used for storing data with low access frequency.

[0012] The data storage index table includes a data identifier, a storage location identifier, an access permission identifier, a timestamp identifier and a resource consumption identifier, wherein the access permission identifier includes read permission, write permission and execution permission, the timestamp identifier includes a creation timestamp, an access timestamp and a modification timestamp, and the resource consumption identifier includes a storage resource consumption value, a reading resource consumption value and a modification resource consumption value.

[0013] The unit data storage resource consumption vector represents the consumption of computing resources, storage resources and network resources required for storing a unit of data, the unit data reading resource consumption vector represents the consumption of computing resources, storage resources and network resources required for reading a unit of data, the unit data modification resource consumption vector represents the consumption of computing resources, storage resources and network resources required for modifying a unit of data, and the data redundancy storage index represents the degree of redundancy backup performed in the data storage process to ensure data reliability.

[0014] The second aspect of the present application provides a computer readable storage medium, the computer readable storage medium stores program instructions, the program instructions are used to execute the above-mentioned data private storage and access method of public cloud service when running in the computer.

[0015] The third aspect of the present application provides a data private storage and access system of public cloud service, which includes the above-mentioned computer readable storage medium, the system is any one of computer, server and single-chip microcomputer, the computer readable storage medium is arranged in the system, and the system is provided with a microprocessor for executing the program instructions stored in the computer readable storage medium.

[0016] Compared with the prior art, the data private storage and access method, medium and system of a public cloud service provided by the application, the application proposes a data storage and access method based on a pyramid structure neural network, which realizes the intelligentization and dynamic optimization of data storage allocation by establishing a multi-level data analysis and processing mechanism. The method introduces innovative indicators such as unit data resource consumption vector and data redundancy storage index, and constructs a complete data feature analysis system, effectively solving the problems existing in the traditional technology.

[0017] The scheme of the application realizes technical innovation in the following aspects: first, the pyramid structure neural network model is used for data feature extraction and analysis, realizing deep learning of data access mode, which can accurately identify the dynamic characteristics of data access and modification behavior. Second, the resource evaluation layer is introduced, and the unit data storage, reading and modification resource consumption vector is calculated to comprehensively evaluate the resource consumption in the data operation process. Third, the multidimensional analysis of data access features is realized through matrix decomposition technology, and the data redundancy storage index is combined to realize dynamic optimization of storage strategy. Finally, a complete data storage index system is established to realize the unified management of data storage location, access permission and resource consumption.

[0018] Through the above technical innovation, the application successfully solves the technical problem that the public cloud service data in the existing technology cannot be dynamically optimized and allocated according to the access characteristics and resource consumption in the multi-level storage system. Compared with the prior art, the application can more accurately identify data access characteristics, more reasonably allocate storage resources, and more efficiently manage data access. This not only improves the resource utilization efficiency of the storage system, but also significantly improves the data access performance, providing more advanced technical support for data management of public cloud platforms. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 The flowchart of the method of the application.

[0020] Figure 2 The comparison chart of system access volume and concurrency volume in different time periods in Example 2.

[0021] Figure 3 The resource consumption chart of different operation types in Example 2.

[0022] Figure 4 The heat map of data access frequency distribution in Example 2.

[0023] Figure 5 The performance index chart of three storage levels in Example 2. DETAILED DESCRIPTION

[0024] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application.

[0025] As Figure 1 shown in a flow chart of a data private storage and access method of a public cloud service provided by a first aspect of the present application, the method comprises the following steps:

[0026] S01, a pyramid structure neural network model is established and trained, the pyramid structure neural network model comprising a data classification layer, a feature extraction layer, a matrix decomposition layer and a resource evaluation layer;

[0027] S02, data access frequency information and data modification frequency information in the public cloud service are collected, the data access frequency information comprising data access times, access time intervals, access concurrency and data distribution, and the data modification frequency information comprising data modification times, modification time intervals, modification concurrency and data version numbers;

[0028] S03, a unit data storage resource consumption vector, a unit data reading resource consumption vector and a unit data modification resource consumption vector are calculated;

[0029] S04, the data classification layer of the pyramid structure neural network model is used to classify the collected data access frequency information, so as to generate a data access frequency matrix, the data access frequency matrix comprising a time dimension, a user dimension and an operation dimension;

[0030] S05, the data classification layer of the pyramid structure neural network model is used to classify the collected data modification frequency information, so as to generate a data modification frequency matrix, the data modification frequency matrix comprising a time dimension, a user dimension and a modification dimension;

[0031] S06, the feature extraction layer of the pyramid structure neural network model is used to extract features from the data access frequency matrix, so as to obtain an access frequency feature vector;

[0032] S07, the feature extraction layer of the pyramid structure neural network model is used to extract features from the data modification frequency matrix, so as to obtain a modification frequency feature vector;

[0033] S08, the matrix decomposition layer of the pyramid structure neural network model is used to perform a matrix decomposition operation on the access frequency feature vector, so as to obtain an access high-frequency sub-matrix and an access low-frequency sub-matrix;

[0034] S09, performing matrix decomposition operation on the modified frequency feature vector by using the matrix decomposition layer of the pyramid structure neural network model to obtain a modified high-frequency matrix and a modified low-frequency matrix;

[0035] S10, calculating a data redundancy storage index by using a resource evaluation layer of the pyramid structure neural network model;

[0036] S11, establishing a decision function, the decision function comprising a threshold determination unit, a weight calculation unit, and an allocation decision unit, and the input parameters of the decision function comprising the access high-frequency matrix, the access low-frequency matrix, the modified high-frequency matrix, the modified low-frequency matrix, the unit data storage resource consumption vector, the unit data reading resource consumption vector, the unit data modification resource consumption vector, and the data redundancy storage index;

[0037] S12, inputting the calculation result of the decision function into a storage allocation module, the storage allocation module comprising a cache layer, a main storage layer, and an auxiliary storage layer, and performing multi-level storage allocation on data according to the calculation result;

[0038] S13, establishing a data storage index table according to the multi-level storage allocation result, the data storage index table comprising a data identifier, a storage location identifier, an access permission identifier, a timestamp identifier, and a resource consumption identifier;

[0039] S14, realizing private storage and access control of data according to the data storage index table;

[0040] Wherein, the specific structure of the pyramid structure neural network model is:

[0041] The data classification layer adopts a one-hot encoding method to process input data, and converts the data access frequency information and the data modification frequency information into standard numerical features;

[0042] The feature extraction layer comprises parallel long short-term memory network units and convolutional neural network units, the long short-term memory network units being used for extracting time sequence features, and the convolutional neural network units being used for extracting spatial features;

[0043] The matrix decomposition layer comprises a first layer pyramid structure and a second layer pyramid structure, the first layer pyramid structure being used for preliminarily decomposing the access frequency feature vector and the modification frequency feature vector to output a preliminary decomposition matrix, and the second layer pyramid structure being used for deeply decomposing the preliminary decomposition matrix to output the high-frequency matrix and the low-frequency matrix;

[0044] The resource evaluation layer is used for evaluating resource consumption in data storage and access processes, and outputs a resource consumption evaluation result;

[0045] The specific structure of the determination function is:

[0046] The threshold determination unit is configured to set a threshold range of the data access frequency information and the data modification frequency information;

[0047] The weight calculation unit is configured to calculate weight coefficients of the high-frequency sub-matrix and the low-frequency sub-matrix;

[0048] The allocation decision unit generates a storage allocation strategy according to the weight coefficients;

[0049] The specific structure of the storage allocation module is:

[0050] The cache layer is configured to store data with the highest access frequency;

[0051] The main storage layer is configured to store data with a medium access frequency;

[0052] The auxiliary storage layer is configured to store data with a low access frequency;

[0053] The specific structure of the data storage index table is:

[0054] The access permission identifier includes read permission, write permission, and execution permission;

[0055] The timestamp identifier includes a creation timestamp, an access timestamp, and a modification timestamp;

[0056] The resource consumption identifier includes a storage resource consumption value, a read resource consumption value, and a modification resource consumption value;

[0057] The unit data storage resource consumption vector represents the consumption of computing resources, storage resources, and network resources required for storing a unit of data;

[0058] The unit data read resource consumption vector represents the consumption of computing resources, storage resources, and network resources required for reading a unit of data;

[0059] The unit data modification resource consumption vector represents the consumption of computing resources, storage resources, and network resources required for modifying a unit of data;

[0060] The data redundancy storage index represents the degree of redundancy backup performed to ensure data reliability during data storage.

[0061] The specific implementation of step S01 is to establish a four-layer neural network model with a pyramid-shaped hierarchical architecture, which includes a data classification layer, a feature extraction layer, a matrix decomposition layer, and a resource evaluation layer from bottom to top. In the data classification layer, the data access frequency information and data modification frequency information are first converted into standard numerical features using one-hot encoding technology. This encoding method can effectively avoid the partial order relationship between data and improve the accuracy of feature representation. In the feature extraction layer, parallel long short-term memory network units and convolutional neural network units are set up. The long short-term memory network uses a gating mechanism to capture the timing patterns of data access and modification, while the convolutional neural network extracts spatial features of data distribution through multi-layer convolution operations. In the matrix decomposition layer, a two-level pyramid structure is used. The first level pyramid structure is responsible for the preliminary decomposition of the feature vector, and the second level pyramid structure is used for deep decomposition to obtain high and low frequency components. In the resource evaluation layer, a multi-layer perceptron structure is used to evaluate the resource consumption of the storage and access process. The design purpose of this pyramid structure is to extract and refine data features layer by layer, and to realize accurate decision-making of data storage strategy.

[0062] The specific implementation of step S02 is to collect data access and modification frequency information in a public cloud service environment. First, the number of data access times is collected by deploying distributed monitoring probes, the timestamp information of each access is recorded to calculate the access time interval, the number of concurrent access requests in unit time is counted to obtain the access concurrency, and the data volume distributed to different nodes is calculated. At the same time, the number of data modification operations is recorded, the time difference between adjacent modification operations is calculated to obtain the modification time interval, the number of concurrent modification requests in unit time is counted to obtain the modification concurrency, and the data version number is recorded by recording the version change history of the data. During data collection, the sampling period of access frequency is recommended to be set to 1 minute, the sampling period of modification frequency is recommended to be set to 5 minutes, and the data retention period is recommended to be set to 30 days. The purpose of this step is to obtain the basic feature information of data access and modification patterns.

[0063] The specific implementation of step S03 is to calculate three types of resource consumption vectors. When calculating the unit data storage resource consumption vector, first, the CPU usage, memory occupancy, storage space size, network bandwidth occupancy, etc. required for storing 1MB of data are counted, and then these indicators are normalized to form a vector. When calculating the unit data reading resource consumption vector, the CPU usage, memory occupancy, disk read rate, network transmission rate, etc. during the reading of 1MB of data are counted, and similarly normalized. When calculating the unit data modification resource consumption vector, the CPU usage, memory occupancy, disk write rate, network transmission rate, etc. during the modification of 1MB of data are counted, and normalized to obtain a vector. During the calculation, the exponential moving average method is used to smooth the indicators, and the smoothing factor is recommended to be set to 0.8. The role of this step is to quantitatively evaluate the resource consumption of data operations.

[0064] The specific implementation of step S04 is to process data access frequency information using the data classification layer. First, a three-dimensional tensor structure is constructed to represent the data access frequency matrix, with the time dimension recording the time information of the access, the user dimension recording the user identifier initiating the access, and the operation dimension recording the specific access operation type. For the time dimension, the time slice is divided into hours; for the user dimension, the user role and permission level are classified; for the operation dimension, the access operation is divided into reading, previewing, downloading, etc. Then, the one-hot encoding method is used to convert the category features of each dimension into numerical features to generate a standardized data access frequency matrix. During matrix generation, data with an access frequency of less than 1 per hour is marked as low-frequency access, data with an access frequency of 1 to 10 per hour is marked as medium-frequency access, and data with an access frequency of more than 10 per hour is marked as high-frequency access. The purpose of this step is to construct a multi-dimensional data access pattern representation.

[0065] The specific implementation of step S05 is to process data modification frequency information using the data classification layer. First, a three-dimensional tensor structure is constructed to represent the data modification frequency matrix, with the time dimension recording the time information of the modification operation, the user dimension recording the user identifier initiating the modification, and the modification dimension recording the specific modification operation type. For the time dimension, the time slice is divided into days; for the user dimension, the user role and modification permission are classified; for the modification dimension, the modification operation is divided into updating, deleting, version changing, etc. Then, the one-hot encoding method is used to convert the category features of each dimension into numerical features to generate a standardized data modification frequency matrix. During matrix generation, data with a modification frequency of less than 1 per day is marked as low-frequency modification, data with a modification frequency of 1 to 5 per day is marked as medium-frequency modification, and data with a modification frequency of more than 5 per day is marked as high-frequency modification. The purpose of this step is to construct a multi-dimensional data modification pattern representation.

[0066] The specific implementation of step S06 is to process the data access frequency matrix using a feature extraction layer. First, a long short-term memory network is used to extract time sequence features, the network includes an input layer, two hidden layers and an output layer, the number of neurons in the hidden layers is 128 and 64 respectively, and the activation function uses the hyperbolic tangent function. At the same time, a convolutional neural network is used to extract spatial features, the network includes two convolutional layers and a fully connected layer, the convolution kernel size is 3x3, the step is 1, the padding method is the same padding, and the activation function uses the linear rectifier function. Finally, the time sequence features and the spatial features are fused to obtain the access frequency feature vector. During feature extraction, batch normalization technology is used to process feature data, and the batch size is set to 32. The role of this step is to extract deep feature representation of data access pattern.

[0067] The specific implementation of step S07 is to process the data modification frequency matrix using a feature extraction layer. First, a long short-term memory network is used to extract time sequence features, the network structure is the same as that of the access frequency feature extraction, but the input data dimension needs to be adjusted according to the actual size of the modification frequency matrix. At the same time, a convolutional neural network is used to extract spatial features, the network structure is also the same as that of the access frequency feature extraction, but the convolution kernel parameters need to be adjusted to adapt to the characteristics of the modification frequency data. Finally, the time sequence features and the spatial features are fused to obtain the modification frequency feature vector. During feature extraction, batch normalization technology is also used, but the batch size can be adjusted according to the scale of the modification data. The role of this step is to extract deep feature representation of data modification pattern.

[0068] The specific implementation of step S08 is to process the access frequency feature vector using a matrix decomposition layer. First, in the first layer pyramid structure, a non-negative matrix decomposition algorithm is used to preliminarily decompose the feature vector, the decomposition rank is set to 80% of the rank of the original matrix, and the alternating least squares method is used to optimize the decomposition result. Then in the second layer pyramid structure, a singular value decomposition algorithm is used to deeply decompose the preliminary decomposition matrix, and the decomposition result is divided into access high frequency component and low frequency component according to the singular value size. During the division process, the part with a singular value cumulative energy proportion exceeding 90% is taken as the high frequency component, and the rest is taken as the low frequency component. The purpose of this step is to realize the frequency stratification of the access pattern.

[0069] The specific implementation of step S09 is to modify the frequency feature vector using matrix decomposition layer processing. First, in the first layer of the pyramid structure, the same non-negative matrix decomposition algorithm as the access frequency processing is used for preliminary decomposition, but the decomposition rank can be set to 70% of the original matrix rank, because the modification operation usually has strong local correlation. Then in the second layer of the pyramid structure, the same singular value decomposition algorithm is used for deep decomposition, but when dividing high and low frequency components, the threshold of singular value cumulative energy ratio can be set to 85%, considering that the frequency distribution of the modification operation is usually more concentrated than the access operation. The purpose of this step is to realize the frequency stratification of the modification mode.

[0070] The specific implementation of step S10 is to calculate the data redundancy storage index using the resource evaluation layer. First, the number of backup copies of the data is counted, the geographical distance between different storage locations is calculated, and the reliability level of the storage nodes is evaluated. Then a weighted summation model is constructed according to these factors, the copy number weight is set to 0.4, the geographical distribution weight is set to 0.3, and the node reliability weight is set to 0.3. Finally, the weighted result is normalized to the range of 0 to 1 to obtain the data redundancy storage index. The redundancy storage index less than 0.3 indicates insufficient redundancy, between 0.3 and 0.7 indicates moderate redundancy, and greater than 0.7 indicates higher redundancy. The role of this step is to evaluate the reliability level of data storage.

[0071] The specific implementation of step S11 is to establish a decision function for storage allocation. In the threshold judgment unit, the high frequency threshold of access frequency is set to 10 times per hour, and the low frequency threshold is set to 1 time per hour; the high frequency threshold of modification frequency is set to 5 times per day, and the low frequency threshold is set to 1 time per day. In the weight calculation unit, the soft maximum value function is used to calculate the weight coefficients of high and low frequency components, considering the influence of resource consumption vector and redundancy storage index. In the allocation decision unit, the decision tree algorithm is used to generate storage allocation strategy according to the weight coefficients, the depth of the decision tree is set to 3 layers, and the leaf nodes correspond to different storage levels. The purpose of this step is to realize the automatic decision of data storage strategy.

[0072] The specific implementation of step S12 is to perform storage allocation according to the output result of the decision function. In the cache layer, the memory cache technology is used to store the data with the highest access frequency, and the cache capacity is set to 10% of the total data amount; in the main storage layer, the solid state disk is used to store the data with medium access frequency, and the storage capacity is set to 60% of the total data amount; in the auxiliary storage layer, the mechanical hard disk is used to store the data with low access frequency, and the storage capacity is set to 30% of the total data amount. In the storage allocation process, the correlation between data needs to be considered, and the data with strong correlation should be allocated to the same storage level as much as possible. The role of this step is to realize the hierarchical storage management of data.

[0073] The specific implementation of step S13 is to construct a data storage index table. First, a unique identifier with a length of 32 bytes is generated for each data object using the uniform resource identifier rule, which is composed of a timestamp, a random number, a data type identifier, and other information. In the storage location identification, the storage level number and the physical address information in that level are recorded, the storage level number is represented by 2 bytes, and the physical address is represented by 8 bytes. In the access permission identification, a 9-bit binary bitmap is used to represent the permission combination, where the first 3 bits represent the owner's permission, the middle 3 bits represent the user group's permission, and the last 3 bits represent the other user's permission, and each group of 3 bits corresponds to read, write, and execute permissions, respectively. In the timestamp identification, a 64-bit integer is used to record the Unix timestamp, accurate to the millisecond level, to record the creation time, the last access time, and the last modification time of the data. In the resource consumption identification, the normalized resource consumption value is recorded, and the storage resource consumption value ranges from 0 to 100, representing the storage space occupancy percentage, the read resource consumption value ranges from 0 to 100, representing the read bandwidth occupancy percentage, and the modification resource consumption value ranges from 0 to 100, representing the relative overhead of the modification operation. The index table is organized using a B+ tree structure, the leaf nodes are sorted according to the data identifier, each node can accommodate 128 index items, and the intermediate nodes store the minimum key values of the index items. A multi-level cache mechanism is set up in the B+ tree, the first-level cache uses memory to store recently accessed index items, with a capacity of 5% of the total number of index items, and the second-level cache uses a solid-state drive to store less frequently used index items, with a capacity of 20% of the total number of index items. At the same time, auxiliary indexes are established in the index table to support fast queries according to the access time, modification time, resource consumption, and other dimensions. In order to ensure the reliability of the index table, regular backups of the index table are performed, with a backup period of 1 hour and a retention of the last 24 backup versions. The purpose of this step is to establish an efficient and reliable data indexing mechanism to support fast positioning and access control.

[0074] The specific implementation of step S14 is to realize data access control according to the index table. First, an access verification process is established. When receiving a data access request, the user identity token in the request is extracted. The token is in the JSON Web Token format and contains fields such as user identification, role information, and permission level. Then, the identity authentication service verifies the validity of the token, checks whether the token signature is correct and whether it is within the valid period. Next, according to the user role and permission level, a 9-bit permission bitmap is used for permission matching. Only when the requested operation permission is a subset allowed by the permission bitmap can the verification pass. After verification, the corresponding index item is found in the B+ tree index according to the data identifier. First, it is found in the first-level cache, if it is not hit, it is found in the second-level cache, if it is still not hit, it is found in the disk. After obtaining the storage location information, the storage level of the data is determined. If the data is in the cache layer, the data is returned directly; if it is in the main storage layer, it is necessary to check whether the data needs to be loaded to the cache layer; if it is in the auxiliary storage layer, it is necessary to evaluate whether the level migration is needed. Before returning the data, the access timestamp is updated. For read operation, the recent access time is updated, and for modification operation, the recent modification time is updated. Then, the resource consumption of this access is counted, including the actual storage space occupation, read bandwidth usage and processor time consumption. The sliding window method is used to update the value of the resource consumption identifier, and the window size is set to 100 accesses. For the case that needs to be migrated, the cost and benefit of migration are evaluated first. The network bandwidth and storage space required for migration are calculated, as well as the possible access performance improvement after migration. When the benefit is more than 1.5 times the cost, migration is performed. The migration process is performed in an asynchronous manner to avoid affecting normal data access. When data migration occurs, a migration lock is needed to ensure data consistency. The granularity of the lock can be at the data block level, and the default data block size is 4MB. At the same time, a double-write mechanism is maintained during the migration process, that is, data is written to the new and old storage locations at the same time, and the data in the old location is deleted only after the migration is completed. The access control strategy is evaluated and optimized regularly, and the evaluation period is 24 hours. According to the statistical access mode, the cache strategy and migration strategy are automatically adjusted. The role of this step is to realize a safe and efficient data access mechanism, and to ensure the controllability and performance of data access.

[0075] Specifically, the pyramid structure neural network model adopts a four-layer architecture from bottom to top, including data classification layer, feature extraction layer, matrix decomposition layer and resource evaluation layer. Among them, the data classification layer as the input layer of the model, adopts one-hot encoding method to process input data, and converts data access frequency information and data modification frequency information into standard numerical feature vector. The structure of the data classification layer includes three parts of input nodes, encoding nodes and output nodes, the input nodes are responsible for receiving original data, the encoding nodes perform one-hot encoding conversion, and the output nodes generate standardized feature vectors. In the data classification layer, different encoding schemes are used for different types of input information, one-hot encoding is used for discrete features, interval encoding is used for continuous features, and periodic encoding is used for time features. The feature extraction layer as the second layer of the model, contains parallel long short-term memory network unit and convolutional neural network unit, the long short-term memory network unit is used to extract the time sequence features of data access and modification mode, and the convolutional neural network unit is used to extract the spatial features of data distribution. The long short-term memory network unit adopts a bidirectional LSTM structure, containing 128 memory units, the activation functions of input gate, forget gate and output gate are all Sigmoid functions, and the state update adopts hyperbolic tangent function. The convolutional neural network unit adopts a deep separable convolution structure, containing 3 convolution layers, the convolution kernel size of each layer is 3×3, the step is 1, the padding method is the same padding, and the activation function adopts linear rectifier function. The matrix decomposition layer as the third layer of the model, contains two levels of pyramid structure, the first level of pyramid structure is responsible for the preliminary decomposition of feature vectors, and the second level of pyramid structure is responsible for the deep decomposition. The first level of pyramid structure adopts non-negative matrix factorization algorithm, optimizes the decomposition result by alternating least squares method, and the decomposition rank is set to 80% of the rank of the original matrix. The second level of pyramid structure adopts singular value decomposition algorithm, and the decomposition result is divided into high frequency component and low frequency component according to the energy distribution of singular value. The resource evaluation layer as the top layer of the model, adopts a multilayer perceptron structure to evaluate the resource consumption in the data storage and access process, contains 3 fully connected layers, the number of neurons is 256, 128 and 64 respectively, the activation function adopts parameterized linear rectifier function, and the output layer adopts Softmax function.

[0076] The establishment process of the training data set includes the following steps: first, collect basic data through system monitoring tools, including CPU usage, memory occupancy, storage space usage, network bandwidth usage and other system-level indicators, the sampling period is set to 1 minute, and the data is continuously collected for not less than 30 days. Then collect application layer data through the log analysis system, including data access records, modification records, user behavior logs and other information, the statistical period is set to 1 hour, and the latest 90 days of historical data is retained. Then the collected raw data is preprocessed, including data cleaning, outlier processing, missing value filling and other operations, the moving median method is used for data cleaning to remove noise, the 3σ principle is used for outlier judgment, and multiple interpolation methods are used for missing value filling. Then a feature matrix is constructed according to the preprocessed data, including time features, user features, operation features and other dimensions, and the recursive feature elimination algorithm is used for feature selection, and the top 80% of important features are retained. Finally, the processed data set is divided into training set, validation set and test set according to the ratio of 8:1:1.

[0077] The training process of the model adopts the strategy of hierarchical pre-training and end-to-end fine-tuning. First, the data classification layer is pre-trained, the cross-entropy loss function is used, the Adam algorithm is selected as the optimizer, the learning rate is set to 0.001, the batch size is 64, and the training rounds are 100. Then the feature extraction layer is pre-trained, the mean square error loss function is used for long short-term memory network and convolutional neural network respectively, the Adam algorithm is also selected as the optimizer, the learning rate is set to 0.0005, the batch size is 32, and the training rounds are 200. Then the matrix decomposition layer is pre-trained, the Frobenius norm is used as the loss function, the stochastic gradient descent method is selected as the optimizer, the learning rate is set to 0.0001, the momentum factor is 0.9, and the training rounds are 300. Then the resource evaluation layer is pre-trained, the weighted cross-entropy loss function is used, the AdamW algorithm is selected as the optimizer, the learning rate is set to 0.0002, the weight decay coefficient is 0.01, and the training rounds are 150. Finally, end-to-end fine-tuning training is performed, a multi-task learning framework is used, the loss function includes classification loss, reconstruction loss and prediction loss, the Adam algorithm with cosine annealing is selected as the optimizer, the initial learning rate is 0.0001, the minimum learning rate is 0.000001, and the training rounds are 500. During the training process, the early stopping strategy is used to prevent overfitting, and the gradient clipping technique is used to ensure the stability of the training, and the clipping threshold is set to 5.0. In order to improve the generalization ability of the model, data enhancement techniques are also used in the training process, including random noise injection, time window sliding, feature replacement and other methods. In addition, the regularization term is introduced to control the complexity of the model, the L1 regularization coefficient is set to 0.0001, and the L2 regularization coefficient is set to 0.00001.

[0078] The main calculation process and matrix of the present application are described in detail as follows:

[0079] The unit data storage resource consumption vector is specifically represented as follows:

[0080] V s = [α c C s , α m M s , α d D s , α n N s ]+ ∈ s ;

[0081] In the formula, V s is the unit data storage resource consumption vector; C s is the CPU usage rate; M s is the memory occupancy rate; D s is the storage space size; N s is the network bandwidth occupancy rate; α c , α m , α d , α n are the weight coefficients of the respective resources; and ∈ s is a random error term.

[0082] The unit data reading resource consumption vector is specifically represented as follows:

[0083] V r = [β c C r , β m M r， β d R d , β n Nr ] + ∈ r ;

[0084] In the formula, V r is the unit data reading resource consumption vector; C r is the CPU usage rate; M r is the memory occupancy rate; R d is the disk reading rate; N r is the network transmission rate; β c , β m , β d , β n are the weight coefficients of the respective resources; and ∈ r is a random error term.

[0085] The unit data modification resource consumption vector is specifically represented as follows:

[0086] V m =[γ c C m γ c M m γ d W d γ n N m ]+∈ m ;

[0087] In the formula, V m Modify the resource consumption vector for the cell data; C m CPU utilization; M m Memory usage; W d N represents the disk write speed. m For network transmission rate; γ c γ m γ d γ n These are the weighting coefficients for each resource; ∈ m This is the random error term.

[0088] The data access frequency matrix is ​​represented as follows:

[0089]

[0090] In the formula, A is the data access frequency matrix; a ij This represents the access frequency of the j-th user within the i-th time segment; m is the number of time segments; and n is the number of users.

[0091] The data modification frequency matrix is ​​represented as follows:

[0092]

[0093] In the formula, M is the data modification frequency matrix; m ij This represents the modification frequency of the j-th user within the i-th time segment; k is the number of time segments; and n is the number of users.

[0094] The access frequency feature vector is specifically represented as follows:

[0095]

[0096] In the formula, F a The access frequency feature vector; LSTM(·) represents the Long Short-Term Memory network operation; CNN(·) represents the Convolutional Neural Network operation; Indicates feature fusion operation; δ a This is the feature extraction error term.

[0097] The specific representation of modifying the frequency eigenvector is as follows:

[0098]

[0099] In the formula, F m is the modified frequency feature vector; LSTM(·) represents a long short-term memory network operation; CNN(·) represents a convolutional neural network operation; represents a feature fusion operation; δ m is a feature extraction error term.

[0100] The matrix decomposition operation is specifically represented as follows:

[0101]

[0102] In the formula, U a , U m is a left singular matrix; S a , S m is a singular value matrix; V a , V m is a right singular matrix; E a , E m is a decomposition error term.

[0103] The data redundancy storage index is specifically represented as follows:

[0104] R = ω1N c + ω2D g + ω3R n + ∈ r ;

[0105] In the formula, R is the data redundancy storage index; N c is the number of copies; D g is the geographical distance; R n is the node reliability; ω1, ω2, and ω3 are weight coefficients; and ∈ r is an evaluation error term.

[0106] The parameter acquisition method is described as follows: resource consumption related parameters are collected through a system monitoring tool, with a sampling period of 1 minute and a sampling duration of not less than 7 days; access frequency and modification frequency related parameters are obtained through a log analysis system, with a statistical period of 24 hours; network parameters are measured through a network testing tool, including bandwidth, delay, packet loss rate and other indicators; weight coefficients are obtained through machine learning method training, with a training data set size of not less than 10,000; and the error term is determined through experimental verification to have a reasonable range, generally controlled within ±5%.

[0107] The principle of constructing the above equation shows that the resource consumption vector adopts a linear weighting model, considers the contribution of different resources to the overall consumption, introduces a random error term to adapt to the fluctuations of the actual environment; the frequency matrix adopts a two-dimensional structure, which can express the feature distribution of time dimension and user dimension at the same time; the feature extraction adopts a deep learning method, LSTM can capture the time sequence pattern, and CNN can extract spatial features; the matrix decomposition adopts the singular value decomposition method, which can effectively separate the high-frequency and low-frequency components; the redundant storage index considers multiple influencing factors, and adopts the form of weighted summation for comprehensive evaluation;

[0108] The decision function is composed of a threshold decision unit, a weight calculation unit and an allocation decision unit, and is specifically represented as follows:

[0109] The decision function of the threshold decision unit is:

[0110]

[0111] In the formula, T a is the access frequency threshold decision result; T m is the modification frequency threshold decision result; f a is the actual access frequency; f m is the actual modification frequency; θ a is the access frequency threshold; θ m is the modification frequency threshold; ∈ t is the decision error term.

[0112] The calculation function of the weight calculation unit is:

[0113]

[0114] In the formula, W h is the high-frequency component weight; W l is the low-frequency component weight; H is the norm of the high-frequency component matrix; L is the norm of the low-frequency component matrix; P i is the component matrix set; α h , α l , α i are weight coefficients; ∈ w is the weight calculation error term.

[0115] The decision function of the allocation decision unit is:

[0116]

[0117] In the formula, D is the final decision result; W h is the high-frequency component weight; W l is the low-frequency component weight; R is the redundant storage index; V is the resource consumption vector; λ1, λ2, λ3, λ4 are decision coefficients; is the time derivative of resource consumption; ∈ d is the decision error term.

[0118] where the access high-frequency matrix and the access low-frequency matrix are specifically represented as follows:

[0119]

[0120] where h ij and l ij respectively represent the high-frequency access feature and the low-frequency access feature of the jth user in the ith time slice; m is the number of time slices; and n is the number of users.

[0121] where the modified high-frequency matrix and the modified low-frequency matrix are specifically represented as follows:

[0122]

[0123] where h′ ij and l′ ij respectively represent the high-frequency modification feature and the low-frequency modification feature of the jth user in the ith time slice; k is the number of time slices; and n is the number of users.

[0124] where the derivation process of the threshold determination unit is as follows:

[0125] 1. First, consider a simple threshold determination function: g(x) = sign(x - θ);

[0126] 2. In order to avoid the discontinuity of the function, a Sigmoid function is introduced for smoothing:

[0127] 3. Considering the influence of noise in the actual environment, an error term is added:

[0128] 4. Respectively applied to access frequency and modification frequency determination, the final determination function is obtained:

[0129]

[0130] where the derivation process of the weight calculation unit is as follows:

[0131] 1. The initial weight calculation adopts an exponential form:

[0132] 2. In order to ensure that the weight sum is 1, normalization processing is introduced:

[0133] 3. Respectively consider the weight calculation of the high-frequency component and the low-frequency component:

[0134]

[0135] Wherein, the derivation process of the allocation decision unit is:

[0136] 1. The base decision function considers the logarithmic form of the weight: d1 = λ1ln(W h ) + λ2ln(W l );

[0137] 2. The nonlinear influence of the redundant storage is introduced: d2 = λ3R 2 ;

[0138] 3. The dynamic change of the resource consumption is considered:

[0139] 4. The final decision function is obtained by combination:

[0140]

[0141] The acquisition method of the related parameters is: θ a , θ m : obtained by statistical data, generally taking 80% quantile value; α h , α l , α i : obtained by stochastic gradient descent method training; λ1, λ2, λ3, λ4: obtained by multi-objective optimization algorithm; ∈ t , ∈ w , ∈ d : a reasonable range is determined by experiment verification, generally not more than 0.1.

[0142] The equation construction principle is explained as follows: the threshold determination adopts a Sigmoid function, a smooth threshold conversion process is realized; the weight calculation adopts a Softmax function, it is ensured that the sum of the weights is 1 and non-negative; the logarithmic term is introduced in the decision function, so that the weight change is more sensitive; the square term is introduced to reflect the nonlinear influence of the redundant storage; the time partial derivative term reflects the dynamic change trend of the resource consumption. Smooth threshold determination is realized, discontinuity caused by the hard threshold is avoided; the weight calculation considers the mutual influence of multiple components; the decision function comprehensively considers the static characteristics and dynamic change; the nonlinear transformation is introduced to improve the expression ability of the model.

[0143] The second aspect of the present application provides a computer readable storage medium, the computer readable storage medium has program instructions stored therein, the program instructions are used to execute the above-mentioned data private storage and access method of public cloud service when running in the computer.

[0144] The third aspect of the present application provides a data private storage and access system of public cloud service, comprising the computer readable storage medium, the system is any one of computer, server, single chip microcomputer, the computer readable storage medium is arranged in the system, the system is provided with microprocessor for executing the program instruction stored in the computer readable storage medium.

[0145] Specifically, the principle of the present application is that the core technical principle of the present application is based on deep learning and resource optimization theory. The pyramid structure neural network model can effectively capture the time sequence features and spatial features in the data access mode through multi-level feature extraction and analysis. Among them, the data classification layer uses one-hot encoding method to process input data, which ensures the integrity and accuracy of feature representation; the feature extraction layer extracts time sequence features and spatial features through parallel long short-term memory network units and convolutional neural network units, respectively, and realizes comprehensive analysis of the data access mode.

[0146] In terms of resource evaluation, the unit data resource consumption vector introduced by the present application innovatively quantifies the consumption of computing resources, storage resources and network resources in the data operation process. This representation method enables the system to accurately evaluate the resource consumption of different types of data operations, providing a reliable basis for storage allocation decisions. At the same time, the introduction of data redundancy storage index enables the system to dynamically adjust the redundancy storage strategy of data according to the importance of data and the resource status of the system.

[0147] The design of the decision function adopts a multi-level decision mechanism, which realizes the intelligent formulation of the storage allocation strategy through the collaborative work of the threshold determination unit, the weight calculation unit and the allocation decision unit. This decision mechanism not only considers traditional features such as data access frequency and modification frequency, but also considers innovative indicators such as resource consumption and redundancy storage, making the storage allocation decision more comprehensive and reasonable.

[0148] A specific embodiment 1 of the present application is provided below, and the specific implementation of each step in embodiment 1 is described in detail as follows.

[0149] The specific implementation of step S01 is to establish a pyramid-structured neural network model. This model adopts a bottom-up four-layer architecture, including a data classification layer, a feature extraction layer, a matrix factorization layer, and a resource evaluation layer. The data classification layer, as the model's input layer, processes the input data using one-hot encoding, converting data access frequency information and data modification frequency information into standard numerical feature vectors. Specifically, it includes three parts: an input node, an encoding node, and an output node. The input node receives the raw data, the encoding node performs the one-hot encoding transformation, and the output node generates standardized feature vectors. Different encoding schemes are used for different types of input information: one-hot encoding is used for discrete features, interval encoding is used for continuous features, and periodic encoding is used for time-related features. The mathematical expression of one-hot encoding is:

[0150] X onehot =E·X raw +∈ e ;

[0151] In the formula, X onehot X is the encoded feature vector; raw The original input data is E; E is the encoding matrix; ∈ e This represents the encoding error term. The feature extraction layer, as the second layer of the model, contains parallel Long Short-Term Memory (LSTM) network units and convolutional neural network units. The LSM network unit adopts a bidirectional structure, containing 128 memory units. The activation functions for the input gate, forget gate, and output gate are all Sigmoid functions, and the state update uses the hyperbolic tangent function. The mathematical expression of the LSM unit is:

[0152] f t =σ(W f ·[h t-1 x t ]+b f );

[0153] i t =σ(W i ·[h t-1 x t ]+b i );

[0154] o t =σ(W o ·[h t-1 x t ]+b o );

[0155] c t =f t ·c t-1 +i t ·tanh(W c ·[h t-1, x t ]+b c );

[0156] h t =o t ·tanh(c t );

[0157] In the formula, f t , i t , o t are the states of the forget gate, the input gate and the output gate respectively; c t is the cell state; h t is the hidden state; W f , W i , W o , W c are weight matrices; b f , b i , b o , b c are bias vectors; sigma is a Sigmoid function; and tanh is a hyperbolic tangent function. The convolutional neural network unit adopts a depth separable convolution structure, contains three convolution layers, the convolution kernel size of each layer is 3*3, the step is 1, the padding mode is the same padding, and the activation function adopts a linear rectifier function. The mathematical expression of the convolution operation is:

[0158] Y = Conv (X, W) + b;

[0159] In the formula, Y is the convolution output; X is the input feature map; W is the convolution kernel weight; and b is the bias term. The matrix decomposition layer is the third layer of the model, contains a two-level pyramid structure, the first level of the pyramid structure adopts a non-negative matrix decomposition algorithm, and the decomposition result is optimized by an alternating least squares method. The second level of the pyramid structure adopts a singular value decomposition algorithm, and the decomposition result is divided into a high-frequency component and a low-frequency component according to the energy distribution of the singular values. The resource evaluation layer is the top layer of the model, adopts a multilayer perceptron structure to evaluate the resource consumption in the data storage and access process, contains three fully connected layers, and the number of neurons is 256, 128 and 64 respectively. The training of the model adopts a hierarchical pre-training and end-to-end fine-tuning strategy, and the loss function includes three parts of a classification loss, a reconstruction loss and a prediction loss.

[0160] The specific implementation of step S02 is to collect data access frequency information and data modification frequency information in the public cloud service. The collection of the data access frequency information is realized by deploying distributed monitoring probes, the probes are arranged at key nodes of the system, including a load balancer, an application server and a storage node. First, the number of data access times is recorded, the number of access operations in a unit of time is counted in a counter mode, and the update of the counter adopts an atomic operation to ensure concurrent safety. The statistical function of the number of access times is expressed as:

[0161]

[0162] where C a (t) is the number of visits at time t; a i (t) is the count value of the i-th visit operation; n is the total number of operations in the sampling period; ∈ c is the counting error term. Then the visit time interval is calculated, which records the time difference between adjacent visit operations, and the calculation function of the time interval is:

[0163] T i (t) = t i+1 -t i + ∈ t ;

[0164] where T i (t) is the time interval of the i-th visit; t i is the time stamp of the visit operation; ∈ t is the time error term. Next, the visit concurrency is calculated, which records the number of concurrent visit requests in a unit of time, and the calculation function of the concurrency is:

[0165]

[0166] where P a (t) is the concurrency in the time window; Δt is the size of the statistical window; ∈ p is the concurrency statistical error term. Finally, the data distribution volume is calculated, which is the amount of data distributed from the source node to the target node, and the calculation function of the distribution volume is:

[0167]

[0168] where D a (t) is the data distribution volume at time t; d j (t) is the distribution data volume of the j-th node; m is the number of nodes; ∈ d is the distribution statistical error term.

[0169] The collection of data modification frequency information is also achieved through distributed monitoring probes. First, the number of data modifications is recorded, which counts the number of modification operations in a unit of time, and the calculation function of the modification number is:

[0170]

[0171] where C m (t) is the number of modifications at time t; m k (t) is the count value of the k-th modification operation; l is the total number of operations in the sampling period; ∈ mTo modify the count error term. Then calculate the modification time interval, record the time difference between adjacent modification operations, and calculate the time interval in a similar way to the access time interval. Then count the modification concurrency, record the number of modification requests occurring simultaneously in a unit of time, and count the modification concurrency in a similar way to the access concurrency. Finally, record the data version number, count the version change history of the data object, and the version number counting function is:

[0172]

[0173] wherein V n (t) is the version number at time t; v r (r) is the count value of the rth version; s is the total number of versions; ∈ v is the version count error term.

[0174] The specific implementation of step S03 is to calculate the unit data storage resource consumption vector, the unit data read resource consumption vector, and the unit data modification resource consumption vector. The unit data storage resource consumption vector is specifically represented as:

[0175] V s = [α c C s , α m M s , α d D s , α n N s ] + ∈ s ;

[0176] wherein V s is the unit data storage resource consumption vector; C s is the CPU usage rate; M s is the memory occupancy rate; D s is the storage space size; N s is the network bandwidth occupancy rate; α c , α m , α d , α n are the weight coefficients of each resource; and ∈ s is the random error term. The unit data read resource consumption vector is specifically represented as:

[0177] V r = [β c C r , β m M r , β d R d , β n N r ] + ∈ r ;

[0178] In the formula, V r Read the resource consumption vector for the cell data; C r CPU utilization; M r Memory usage; R d N represents the disk read speed; r For network transmission rate; β c ,β m ,β d ,β n For each resource, the weight coefficient is ∈ r This represents the random error term. The specific representation of the resource consumption vector for modifying unit data is as follows:

[0179] V m =[γ c C m γ m M m γ d W d γ n N m ]+∈ m ;

[0180] In the formula, V m Modify the resource consumption vector for the cell data; C m CPU utilization; M m Memory usage; W d N represents the disk write speed. m For network transmission rate; γ c γ m γ d γ n For each resource, the weight coefficient is ∈ m This is the random error term.

[0181] The specific implementation of step S04 involves using a data classification layer to classify the data access frequency information and generate a data access frequency matrix. First, a three-dimensional tensor structure is constructed to represent the data access frequency matrix, as shown below:

[0182]

[0183] In the formula, A is the data access frequency matrix; a ij Let represent the access frequency of the j-th user within the i-th time segment; m is the number of time segments; and n is the number of users. The access frequency matrix is ​​processed using one-hot encoding, and the encoding function is expressed as:

[0184] E a =OneHot(A)+δ a ;

[0185] In the formula, Ea is the encoded access frequency matrix; OneHot(·) is the one-hot encoding function; δ a is the encoding error term. The encoding in the time dimension uses a periodic function:

[0186] T e = sin(2πf t ·t) + cos(2πf t ·t) + ∈ t ;

[0187] In the formula, T e is the time encoding value; f t is the time period parameter; t is the time stamp; ∈ t is the time encoding error term.

[0188] The specific implementation of step S05 is to use the data classification layer to classify the data modification frequency information and generate a data modification frequency matrix. The representation form of the modification frequency matrix is:

[0189]

[0190] In the formula, M is the data modification frequency matrix; m ij represents the modification frequency of the jth user in the ith time slice; k is the number of time slices; n is the number of users. The encoding process of the modification frequency matrix is similar to that of the access frequency matrix, but the encoding parameters need to be adjusted according to the characteristics of the modification operation. The encoding function is represented as:

[0191] E m = OneHot(M) + δ m ;

[0192] In the formula, E m is the encoded modification frequency matrix; OneHot(·) is the one-hot encoding function; δ m is the encoding error term.

[0193] The specific implementation of step S06 is to use the feature extraction layer to process the data access frequency matrix. The feature extraction uses a parallel processing method of long short-term memory network and convolutional neural network. The processing process of the long short-term memory network is represented as:

[0194] F lstm = LSTM(E a ) + ∈ lstm ;

[0195] In the formula, F lstm is the time sequence feature; LSTM(·) is the long short-term memory network function; ∈ lstm is the feature extraction error term. The processing process of the convolutional neural network is represented as:

[0196] F cnn =CNN(E a )+∈ cnn ;

[0197] In the formula, F cnn Spatial features; CNN(·) is a convolutional neural network function; ∈ cnn This represents the feature extraction error term. The feature fusion process is represented as:

[0198]

[0199] In the formula, F a This is a feature vector representing the access frequency. Indicates feature fusion operation; δ a This is the feature fusion error term.

[0200] The specific implementation of step S07 involves processing the data modification frequency matrix using a feature extraction layer. The extraction process for modification frequency features is similar to that for access frequency features, but the network parameters need to be adjusted to suit the characteristics of modification operations. The time-series feature extraction is represented as follows:

[0201] F lstm =LSTM(E) m )+∈ lstm ′;

[0202] In the formula, F lstm ′ represents the modification of temporal features; LSTM(·) is the Long Short-Term Memory network function; ∈ lstm ' represents the feature extraction error term. Spatial feature extraction is represented as:

[0203] F cnn =CNN(E) m )+∈ cnn ′;

[0204] In the formula, F cnn ′ represents modifying spatial features; CNN(·) is a convolutional neural network function; ∈ cnn ' represents the feature extraction error term. The feature fusion process is represented as:

[0205]

[0206] In the formula, F m To modify the frequency eigenvector; Indicates feature fusion operation; δ m This represents the feature fusion error term. During feature extraction, batch normalization is used to process the feature data; the normalization function is expressed as:

[0207]

[0208] In the formula, B n The result is the normalized value; x is the input feature; μ is the batch mean; σ is the normalized value. 2 denoted as batch variance; ∈ represents the numerical stability coefficient.

[0209] The specific implementation of step S08 involves performing matrix factorization on the access frequency feature vector using a matrix factorization layer. First, a preliminary decomposition is performed in the first-layer pyramid structure using a non-negative matrix factorization algorithm, with the following optimization objective function:

[0210]

[0211] In the formula, F a U is the access frequency feature vector; V is the decomposition matrix; ||·|| F Let be the Frobenius norm; α and β are regularization coefficients. The update rule for the decomposition results is:

[0212]

[0213] In the formula, ⊙ represents the Hadamard product; t is the number of iterations. Then, depth decomposition is performed in the second-level pyramid structure using the singular value decomposition algorithm:

[0214]

[0215] In the formula, U a S is a left singular matrix; a V is a singular value matrix; a E is a right singular matrix; a This is to decompose the error term.

[0216] The specific implementation of step S09 involves using a matrix factorization layer to perform matrix factorization operations on the modified frequency eigenvectors. The optimization objective of the nonnegative matrix factorization of the first-layer pyramid structure is:

[0217]

[0218] In the formula, F m To modify the frequency eigenvectors; P and Q are decomposition matrices; γ and δ are regularization coefficients. The update rule for the decomposition results is:

[0219]

[0220] In the formula, t represents the number of iterations. The singular value decomposition of the second-level pyramid structure is expressed as:

[0221]

[0222] In the formula, U mis a left singular matrix; S m is a singular value matrix; V m is a right singular matrix; E m is a decomposition error term.

[0223] The specific implementation of step S10 is to calculate the data redundancy storage index by using the resource evaluation layer. The calculation function of the redundancy storage index is:

[0224] R = ω1N c + ω2D g + ω3R n + ∈ r ;

[0225] In the formula, R is the data redundancy storage index; N c is the number of copies; D g is the geographical distance; R n is the node reliability; ω1, ω2, and ω3 are weight coefficients; and ∈ r is an evaluation error term. The calculation of the node reliability adopts an exponential decay model:

[0226] R n = e -λt + ∈ n ;

[0227] In the formula, λ is the reliability decay coefficient; t is the running time; and ∈ n is a reliability error term.

[0228] The specific implementation of step S11 is to establish a decision function. The decision function of the threshold decision unit is:

[0229]

[0230] In the formula, T a is the access frequency threshold decision result; T m is the modification frequency threshold decision result; f a is the actual access frequency; f m is the actual modification frequency; θ a is the access frequency threshold; θ m is the modification frequency threshold; and ∈ t is a decision error term. The calculation function of the weight calculation unit is:

[0231]

[0232] In the formula, W h is the high-frequency component weight; W l is the low-frequency component weight; H is the norm of the high-frequency component matrix; L is the norm of the low-frequency component matrix; P i is the component matrix set; and α h , αl α i For weighting coefficients; ∈ w The error term is calculated for the weights. The decision function for allocating the decision units is:

[0233]

[0234] In the formula, D is the final decision result; λ1, λ2, λ3, and λ4 are decision coefficients; The time partial derivative of resource consumption; ∈ d This represents the decision error term.

[0235] The specific implementation of step S12 involves multi-level storage allocation based on the calculation result of the decision function. First, the capacity of each storage level is initialized; the cache layer capacity calculation function is as follows:

[0236] C h =α h ·C total +∈ c ;

[0237] In the formula, C h C is the cache layer capacity. total Total data volume; α h is the capacity coefficient, with a value of 0.1; ∈ c This is the capacity error term. The main storage layer capacity calculation function is:

[0238] C m =α m ·C total +∈ m ;

[0239] In the formula, C m Main storage layer capacity; α m ∈ ... m This represents the capacity error term. The auxiliary storage layer capacity calculation function is:

[0240] C l =α l ·C total +∈ l ;

[0241] In the formula, C l To supplement the storage layer capacity; α l is the capacity factor, with a value of 0.3; ∈ l This represents the capacity error term. The data correlation evaluation function is:

[0242]

[0243] In the formula, R c The correlation coefficient; xi , y j is a data item; μ x , μ y is a mean value; ∈ r is an evaluation error term.

[0244] The specific implementation of step S13 is to construct a data storage index table. First, a globally unique identifier is assigned to each data object, and the physical location of the data in different storage levels is recorded. Then, three basic permissions of reading, writing and executing are set, and the permission combination is represented in bitmap form. Then, the creation time, the latest access time and the latest modification time of the data are recorded, and the time stamp is in a unified format. Finally, the resource consumption value of data operation is recorded, including storage space occupation, read bandwidth consumption and modification operation overhead. The index table is organized in B+ tree structure to improve query efficiency. The purpose of this step is to establish an efficient data indexing mechanism.

[0245] The specific implementation of step S14 is to realize data access control according to the index table. First, verify the user identity and operation permission in the access request to ensure that it meets the requirements of the permission identifier. Then, locate the physical location of the data according to the storage location identifier, and select the appropriate storage level for access. Then, update the timestamp information to record the time of access or modification operation. Finally, count the resource consumption and update the related values of the resource consumption identifier. During the access control process, the cost of data migration needs to be considered for cross-level data access. The role of this step is to ensure the security and efficiency of data access.

[0246] In order to better understand and implement the present application, the following provides an embodiment 2 of a specific application scenario of the present application: In order to improve the storage efficiency of the data center, a cloud computing service provider implements the data private storage and access method of the present application. The data center contains 10,000 servers, and the daily data processing volume reaches 100TB, with more than 1 million users. The data storage scheme before implementation adopts a unified three-copy storage strategy, resulting in low storage space utilization and uneven data access performance.

[0247] First, a pyramid structure neural network model is constructed, implemented using the PyTorch deep learning framework. The network structure parameters are shown in Table 1:

[0248] Table 1 Neural network model structure parameters

[0249] Network hierarchy Parameter configuration Activation function Output dimension Data classification layer Input nodes: 256 ReLU 512 Feature extraction layer LSTM Hidden units: 128 Tanh 256 Feature extraction layer CNN Convolution kernel: 3x3 ReLU 256 Matrix factorization layer 1 Factorization rank: 80% - 128 Matrix factorization layer 2 Number of singular values: 64 - 64 Resource evaluation layer Fully connected layers: 256 / 128 / 64 PReLU 32

[0250] 30 days of system operation data are collected, including access frequency and modification frequency information, and the data sampling results are shown in Table 2:

[0251] Table 2 Data access and modification frequency statistics

[0252]

[0253] Figure 2 The system access volume and concurrency volume of different time periods are shown, and the resource consumption vector is calculated based on the collected data, wherein the weight coefficients of CPU usage, memory occupancy, storage space and network bandwidth are set to 0.3, 0.2, 0.3 and 0.2 respectively. The resource consumption statistics of unit data volume (1MB) are shown in Table 3:

[0254] Table 3 Resource consumption statistics of unit data volume

[0255]

[0256] Figure 3 The resource consumption of different operation types is shown, including CPU usage, memory occupancy, storage space and network bandwidth, etc. The pyramid structure neural network model is used to process the data, and first, the access frequency matrix and the modification frequency matrix are generated, with the matrix size being 24x1000000, representing the access and modification of 100 million users in 24 time slices. The access frequency feature vector and the modification frequency feature vector are obtained through feature extraction, with the vector dimension being 256. After matrix decomposition, the high frequency component and the low frequency component are obtained, wherein the energy distribution of the access frequency is shown in Table 4:

[0257] Table 4 Energy distribution of access frequency

[0258] Component type Proportion of singular value energy Proportion of data Access frequency range (times / second) High-frequency component 92.5％ 15.8％ >10 Low-frequency component 7.5％ 84.2％ ≤10

[0259] Figure 4 The data access frequency distribution of different time periods within a week is shown through the heat map, which reflects the periodic characteristics of the system access mode. According to the decomposition result, the data redundancy storage index is calculated, considering the number of replicas, geographical distance and node reliability, with the weights being 0.4, 0.3 and 0.3 respectively. The redundancy storage index calculation formula is: R = 0.4N c + 0.3D g + 0.3R n + ∈ r .

[0260] Based on the calculation result, a decision function is established, and the specific form of the function is: Wherein the decision coefficients λ1, λ2, λ3, λ4 are solved by a multi-objective optimization algorithm, and the final values are 0.3, 0.2, 0.3 and 0.2 respectively. The output result of the decision function is used to determine the storage strategy of the data, and the storage level division is shown in Table 5:

[0261] Table 5 Storage level division

[0262] Storage hierarchy Capacity proportion Storage medium Access latency Data type Cache layer 10％ Memory <1ms High-frequency access data Main storage layer 60％ SSD <10 ms Medium-frequency access data Auxiliary storage layer 30％ HDD <100 ms Low-frequency access data

[0263] Figure 5 The performance indicators of the three storage levels are compared, including response time, throughput and resource utilization, and a data storage index table is established, containing information such as data identifier, storage location, access permission, timestamp and resource consumption. The index table is organized in a B+ tree structure, with the order of the tree set to 128 and the leaf node fill factor set to 0.8. An example of the structure of the index table is shown in Table 6:

[0264] Table 6: Data storage index table example

[0265] Data identifier Storage location Access permission Creation time Recent access time Resource consumption value 0x1A2B3C Cache / 0x001 rwx--- 2024-01-01 2024-01-21 85.6 0x4D5E6F Main storage / 0x102 rw-rw-- 2024-01-05 2024-01-20 62.3 0x7G8H9I Auxiliary storage / 0x203 rw-r—– 2024-01-10 2024-01-15 28.9

[0266] Compared with the traditional unified three-copy storage scheme, the method adopted by the present application has the following advantages: first, the traditional scheme adopts the same storage strategy for all data, while the present application performs hierarchical storage according to the data access characteristics, improving the storage space utilization, and the experimental results show that the storage space is saved by 35%. Second, the data access performance of the traditional scheme is not balanced, while the present application optimizes the access performance through a multi-layer storage structure, and the average response time of high-frequency access data is reduced by 80%. Third, the traditional scheme lacks a fine-grained access control mechanism, while the present application realizes precise access control through the establishment of a detailed index table, improving data security. Fourth, the traditional scheme cannot adapt to the dynamic changes of data access mode, while the present application can adjust the storage strategy in real time through a neural network model, and the overall performance of the system is improved by 45%. After applying the storage scheme of the present application, the operating cost of the data center is reduced by 25%, and the user satisfaction is improved by 40%, fully verifying the practical value of the method.

[0267] It should be noted that the variables involved in the present application are explained in detail as shown in Table 7 and Table 8.

[0268] Table 7: Variable explanation table (first part)

[0269]

[0270]

[0271] Table 8: Variable explanation table (second part)

[0272]

[0273]

[0274] The above merely illustrates the specific embodiments of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application.

Claims

1. A method for private data storage and access in public cloud services, characterized in that, Includes the following steps: A pyramid structure neural network model is established and trained, which includes a data classification layer, a feature extraction layer, a matrix factorization layer, and a resource evaluation layer; data access frequency information and data modification frequency information in public cloud services are collected. The system calculates the resource consumption vectors for data storage, data reading, and data modification of each unit. It then uses the pyramid-structured neural network model to classify and process the data access frequency and data modification frequency information, generating a data access frequency matrix and a data modification frequency matrix. Finally, it extracts features from the data access frequency matrix and the data modification frequency matrix to obtain access frequency feature vectors and modification frequency feature vectors. Matrix decomposition operations are performed using the matrix decomposition layer to obtain access to high-frequency sub-matrices, access to low-frequency sub-matrices, modification of high-frequency sub-matrices, and modification of low-frequency sub-matrices. Calculate the data redundancy storage index and establish a decision function, and perform multi-level storage allocation based on the calculation result of the decision function; A data storage index table is established to implement private data storage and access control. The judgment function includes a threshold judgment unit, a weight calculation unit, and an allocation decision unit. The input parameters of the judgment function include the access high-frequency sub-matrix, access low-frequency sub-matrix, modification of high-frequency sub-matrix, modification of low-frequency sub-matrix, unit data storage resource consumption vector, unit data reading resource consumption vector, unit data modification resource consumption vector, and data redundancy storage index. The unit data storage resource consumption vector represents the consumption of computing resources, storage resources, and network resources required for storing a unit amount of data. The unit data reading resource consumption vector represents the consumption of computing resources, storage resources, and network resources required for reading a unit amount of data. The unit data modification resource consumption vector represents the consumption of computing resources, storage resources, and network resources required for modifying a unit amount of data. The data redundancy storage index represents the degree of redundancy backup performed during data storage to ensure data reliability.

2. The method for private data storage and access in public cloud services according to claim 1, characterized in that, The specific structure of the pyramid structure neural network model includes: the data classification layer processes the input data using one-hot encoding, converting the data access frequency information and the data modification frequency information into standard numerical features; the feature extraction layer includes parallel long short-term memory network units and convolutional neural network units, where the long short-term memory network units are used to extract temporal features and the convolutional neural network units are used to extract spatial features; the matrix decomposition layer includes a first pyramid structure and a second pyramid structure, where the first pyramid structure is used to perform preliminary decomposition of the access frequency feature vector and the modification frequency feature vector to obtain a preliminary decomposition matrix, and the second pyramid structure is used to perform deep decomposition of the preliminary decomposition matrix; and the resource evaluation layer is used to evaluate the resource consumption during data storage and access.

3. The method for private data storage and access in public cloud services according to claim 1, characterized in that, The data access frequency information includes the number of data accesses, the access time interval, the access concurrency, and the data distribution volume. The data modification frequency information includes the number of data modifications, the modification time interval, the modification concurrency, and the number of data versions.

4. The method for private data storage and access in public cloud services according to claim 1, characterized in that, The data access frequency matrix includes time dimension, user dimension, and operation dimension, and the data modification frequency matrix includes time dimension, user dimension, and modification dimension.

5. The method for private data storage and access in public cloud services according to claim 1, characterized in that, The multi-level storage allocation includes a cache layer, a main storage layer, and an auxiliary storage layer, wherein the cache layer is used to store the most frequently accessed data, the main storage layer is used to store data with medium access frequency, and the auxiliary storage layer is used to store data with low access frequency.

6. The method for private data storage and access in public cloud services according to claim 1, characterized in that, The data storage index table includes a data identifier, a storage location identifier, an access permission identifier, a timestamp identifier, and a resource consumption identifier. The access permission identifier includes read permission, write permission, and execute permission. The timestamp identifier includes a creation timestamp, an access timestamp, and a modification timestamp. The resource consumption identifier includes storage resource consumption value, read resource consumption value, and modification resource consumption value.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions, which, when executed in a computer, are used to perform a method for private data storage and access in a public cloud service as described in any one of claims 1-6.

8. A private data storage and access system for public cloud services, characterized in that, The system includes the computer-readable storage medium of claim 7, wherein the system is any one of a computer, a server, or a microcontroller, the computer-readable storage medium is disposed within the system, and the system is provided with a microprocessor that executes the program instructions stored in the computer-readable storage medium.

Images