Data private storage and access method of public cloud service, medium and system

Through the pyramid structure neural network model, data access characteristics and resource consumption are analyzed, and dynamic optimization allocation of public cloud storage systems is realized, which solves the problems of low resource utilization efficiency and unbalanced data access performance in the existing technology, and improves the overall performance of the storage system.

CN120122878AActive Publication Date: 2025-06-10BEIJING NANCAL RUIYUAN DIGITAL TECH CO LTD

Patent Information

Application Number
CN202510147559.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-02-11
Publication Date
2025-06-10
Estimated Expiration
2045-02-11

AI Technical Summary

Technical Problem

Existing public cloud storage systems cannot achieve dynamic optimization allocation based on data access characteristics and resource consumption, resulting in low resource utilization efficiency and unbalanced data access performance.

Method used

The pyramid structure neural network model is adopted to generate data access frequency matrix and modify frequency matrix through data classification, feature extraction, matrix decomposition and resource evaluation layers, calculate resource consumption vectors, and perform multi-level storage allocation through judgment functions to realize the private storage and access control of data.

Benefits of technology

It realizes intelligent and dynamic optimization of data storage allocation, improves resource utilization efficiency and data access performance of the storage system, and meets the dual needs of users for data access performance and resource utilization efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120122878A_ABST
    Figure CN120122878A_ABST
Patent Text Reader

Abstract

The invention provides a data private storage and access method, a medium and a system for public cloud services, and belongs to the technical field of electric digital data process.The method includes the steps that a pyramid structure neural network model is established, data access and frequency information modification are collected, and resource consumption vectors are calculated; the data classification layer and the feature extraction layer are used for carrying out feature analysis, a high and low frequency division matrix is obtained through the matrix decomposition layer, a data redundancy storage index is calculated in combination with the resource evaluation layer, a comprehensive judgment function is established for storage strategy optimization, and finally multi-level storage distribution and access control of data are achieved. According to the method, the data access characteristics can be identified more accurately, the storage resources can be allocated more reasonably, the data access process can be managed more efficiently, and the technical problem in the prior art that the public cloud service data cannot be dynamically and optimally allocated in a multi-level storage system according to the access characteristics and resource consumption is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of electronic digital data processing, and in particular, relates to a method, medium and system for private storage and access of data of a public cloud service. Background Art

[0002] In the field of public cloud computing, data storage and access control are one of the core technologies. Traditional public cloud storage systems mainly adopt a hierarchical storage architecture, including multiple levels such as cache, primary storage, and auxiliary storage. This hierarchical storage architecture allocates data through preset storage strategies. Common allocation methods include hierarchical storage of hot data based on access frequency, hierarchical storage based on data importance, and hierarchical storage based on data life cycle. These methods have formed a relatively complete technical system in actual applications and have been widely used in public cloud platforms. For example, mainstream public cloud platforms such as Alibaba Cloud and Tencent Cloud have adopted a multi-level storage architecture to meet the data storage needs in different application scenarios.

[0003] However, with the rapid development of cloud computing technology, traditional data storage allocation methods face many challenges. First, existing storage allocation strategies are often static and cannot be adjusted in a timely manner according to the dynamic changes in data access patterns. Secondly, when performing storage allocation, traditional methods mainly consider single-dimensional characteristics (such as access frequency) and ignore the resource consumption characteristics during data access, resulting in low efficiency in storage resource utilization. Thirdly, existing technologies lack the ability to deeply analyze data access and modification behaviors, and cannot accurately identify data access patterns and resource consumption patterns. In addition, in terms of data redundant storage, existing technologies often adopt fixed redundant strategies and fail to perform dynamic optimization based on data characteristics and system resource conditions.

[0004] These problems make it difficult for public cloud platforms to achieve optimal resource utilization when allocating data storage. Especially in large-scale data processing scenarios, how to dynamically optimize allocation based on data access characteristics and resource consumption has become a technical problem that needs to be solved urgently. The existing technology cannot effectively solve the contradiction between data storage allocation and resource utilization, nor can it meet users' dual needs for data access performance and resource utilization efficiency. In other words, there is a technical problem in the existing technology that public cloud business data cannot be dynamically optimized and allocated in a multi-level storage system based on access characteristics and resource consumption. Summary of the invention

[0005] In view of this, the present invention provides a method, medium and system for private storage and access of public cloud business data, which can solve the technical problem in the prior art that public cloud business data cannot be dynamically optimized and allocated in a multi-level storage system according to access characteristics and resource consumption.

[0006] The present invention is implemented as follows: In a first aspect, the present invention provides a method for private data storage and access of a public cloud service, comprising the following steps: establishing and training a pyramid structure neural network model, the pyramid structure neural network model comprising a data classification layer, a feature extraction layer, a matrix decomposition layer, and a resource evaluation layer; collecting data access frequency information and data modification frequency information in the public cloud service; calculating a unit data storage resource consumption vector, a unit data reading resource consumption vector, and a unit data modification resource consumption vector; using the pyramid structure neural network model to classify and process the data access frequency information and the data modification frequency information to generate a data access frequency matrix and a data modification frequency matrix; performing feature extraction on the data access frequency matrix and the data modification frequency matrix to obtain an access frequency feature vector and a modification frequency feature vector; using the matrix decomposition layer to perform matrix decomposition operations to obtain an access high-frequency matrix, an access low-frequency matrix, a modification high-frequency matrix, and a modification low-frequency matrix; calculating a data redundancy storage index and establishing a determination function, and performing multi-level storage allocation according to the calculation result of the determination function; establishing a data storage index table and realizing private storage and access control of data.

[0007] Among them, the specific structure of the pyramid structure neural network model includes: the data classification layer uses the unique hot encoding method to process the input data, and converts the data access frequency information and the data modification frequency information into standard numerical features; the feature extraction layer includes parallel long short-term memory network units and convolutional neural network units, the long short-term memory network units are used to extract temporal features, and the convolutional neural network units are used to extract spatial features; the matrix decomposition layer includes a first layer pyramid structure and a second layer pyramid structure, the first layer pyramid structure is used to perform preliminary decomposition of the access frequency feature vector and the modification frequency feature vector, and the second layer pyramid structure is used to perform deep decomposition of the preliminary decomposition matrix; the resource evaluation layer is used to evaluate the resource consumption during data storage and access.

[0008] The data access frequency information includes the number of data accesses, access time interval, access concurrency, and data distribution volume; the data modification frequency information includes the number of data modifications, modification time interval, modification concurrency, and data version number.

[0009] The data access frequency matrix includes a time dimension, a user dimension, and an operation dimension, and the data modification frequency matrix includes a time dimension, a user dimension, and a modification dimension.

[0010] Among them, the judgment function includes a threshold judgment unit, a weight calculation unit, and an allocation decision unit, and the input parameters of the judgment function include accessing the high-frequency sub-matrix, accessing the low-frequency sub-matrix, modifying the high-frequency sub-matrix, modifying the low-frequency sub-matrix, unit data storage resource consumption vector, unit data reading resource consumption vector, unit data modification resource consumption vector, and data redundancy storage index.

[0011] Among them, the multi-level storage allocation includes a cache layer, a main storage layer, and an auxiliary storage layer, wherein the cache layer is used to store data with the highest access frequency, the main storage layer is used to store data with a medium access frequency, and the auxiliary storage layer is used to store data with a low access frequency.

[0012] Among them, the data storage index table includes a data identifier, a storage location identifier, an access permission identifier, a timestamp identifier, and a resource consumption identifier, wherein the access permission identifier includes read permission, write permission, and execute permission, the timestamp identifier includes a creation timestamp, an access timestamp, and a modification timestamp, and the resource consumption identifier includes a storage resource consumption value, a read resource consumption value, and a modification resource consumption value.

[0013] Among them, the unit data storage resource consumption vector represents the consumption of computing resources, storage resources, and network resources required for storing unit data volume; the unit data reading resource consumption vector represents the consumption of computing resources, storage resources, and network resources required for unit data volume reading operations; the unit data modification resource consumption vector represents the consumption of computing resources, storage resources, and network resources required for unit data volume modification operations; and the data redundancy storage index represents the degree of redundant backup performed during the data storage process to ensure data reliability.

[0014] A second aspect of the present invention provides a computer-readable storage medium, in which program instructions are stored. When the program instructions are run in a computer, they are used to execute the above-mentioned method for private data storage and access of a public cloud business.

[0015] A third aspect of the present invention provides a private data storage and access system for a public cloud business, comprising the above-mentioned computer-readable storage medium, wherein the system is any one of a computer, a server, and a single-chip microcomputer, wherein the computer-readable storage medium is arranged in the system, and wherein the system is provided with a microprocessor for executing program instructions stored in the computer-readable storage medium.

[0016] Compared with the prior art, the present invention provides a method, medium and system for private data storage and access of public cloud services. The present invention proposes a data storage and access method based on a pyramid structure neural network, which realizes intelligent and dynamic optimization of data storage allocation by establishing a multi-level data analysis and processing mechanism. The method introduces innovative indicators such as unit data resource consumption vector and data redundant storage index, constructs a complete data feature analysis system, and effectively solves the problems existing in traditional technologies.

[0017] The solution of the present invention achieves technical innovation through the following aspects: First, a pyramid structure neural network model is used to extract and analyze data features, which realizes deep learning of data access patterns and can accurately identify the dynamic characteristics of data access and modification behaviors. Secondly, a resource evaluation layer is introduced to comprehensively evaluate the resource consumption during data operations by calculating the unit data storage, reading and modifying resource consumption vectors. Thirdly, multi-dimensional analysis of data access features is achieved through matrix decomposition technology, and dynamic optimization of storage strategies is performed in combination with data redundancy storage index. Finally, a complete data storage index system is established to realize unified management of data storage location, access rights and resource consumption.

[0018] Through the above technical innovation, the present invention successfully solves the technical problem in the prior art that public cloud business data cannot be dynamically optimized and allocated in a multi-level storage system according to access characteristics and resource consumption. Compared with the prior art, the present invention can more accurately identify data access characteristics, more reasonably allocate storage resources, and more efficiently manage the data access process. This not only improves the resource utilization efficiency of the storage system, but also significantly improves the data access performance, providing more advanced technical support for data management of the public cloud platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of the method of the present invention.

[0020] Figure 2 This is a comparison chart of system access volume and concurrent volume in different time periods in Example 2.

[0021] Figure 3 This is a diagram of resource consumption for different operation types in Example 2.

[0022] Figure 4 This is a heat map of the data access frequency distribution in Example 2.

[0023] Figure 5 This is a performance indicator diagram of the three storage levels in Example 2. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0025] like Figure 1 FIG. 1 is a flowchart of a method for privately storing and accessing data of a public cloud service provided by the first aspect of the present invention. The method comprises the following steps:

[0026] S01, establishing and training a pyramid structure neural network model, wherein the pyramid structure neural network model includes a data classification layer, a feature extraction layer, a matrix decomposition layer, and a resource evaluation layer;

[0027] S02. Collect data access frequency information and data modification frequency information in public cloud services. The data access frequency information includes the number of data accesses, access time interval, access concurrency, and data distribution volume. The data modification frequency information includes the number of data modifications, modification time interval, modification concurrency, and data version number.

[0028] S03, calculating the unit data storage resource consumption vector, the unit data reading resource consumption vector, and the unit data modification resource consumption vector;

[0029] S04, using the data classification layer of the pyramid structure neural network model to classify the collected data access frequency information to generate a data access frequency matrix, wherein the data access frequency matrix includes a time dimension, a user dimension, and an operation dimension;

[0030] S05, using the data classification layer of the pyramid structure neural network model to classify the collected data modification frequency information to generate a data modification frequency matrix, wherein the data modification frequency matrix includes a time dimension, a user dimension, and a modification dimension;

[0031] S06, using the feature extraction layer of the pyramid structure neural network model to perform feature extraction on the data access frequency matrix to obtain an access frequency feature vector;

[0032] S07, using the feature extraction layer of the pyramid structure neural network model to perform feature extraction on the data modification frequency matrix to obtain a modification frequency feature vector;

[0033] S08, using the matrix decomposition layer of the pyramid structure neural network model to perform matrix decomposition operation on the access frequency feature vector to obtain an access high frequency matrix and an access low frequency matrix;

[0034] S09, using the matrix decomposition layer of the pyramid structure neural network model to perform matrix decomposition operation on the modified frequency feature vector to obtain a modified high-frequency component matrix and a modified low-frequency component matrix;

[0035] S10, calculating a data redundancy storage index using the resource evaluation layer of the pyramid structure neural network model;

[0036] S11, establishing a determination function, the determination function includes a threshold determination unit, a weight calculation unit, and an allocation decision unit, and the input parameters of the determination function include accessing the high-frequency matrix, accessing the low-frequency matrix, modifying the high-frequency matrix, modifying the low-frequency matrix, unit data storage resource consumption vector, unit data reading resource consumption vector, unit data modification resource consumption vector, and data redundancy storage index;

[0037] S12, inputting the calculation result of the determination function into a storage allocation module, wherein the storage allocation module includes a cache layer, a primary storage layer, and an auxiliary storage layer, and performs multi-level storage allocation on the data according to the calculation result;

[0038] S13, establishing a data storage index table according to the multi-level storage allocation result, wherein the data storage index table includes a data identifier, a storage location identifier, an access permission identifier, a timestamp identifier, and a resource consumption identifier;

[0039] S14, implementing private storage and access control of data according to the data storage index table;

[0040] The specific structure of the pyramid structure neural network model is as follows:

[0041] The data classification layer processes the input data using a one-hot encoding method, converting the data access frequency information and the data modification frequency information into standard numerical features;

[0042] The feature extraction layer comprises a parallel long short-term memory network unit and a convolutional neural network unit, wherein the long short-term memory network unit is used to extract temporal features, and the convolutional neural network unit is used to extract spatial features;

[0043] The matrix decomposition layer comprises a first layer pyramid structure and a second layer pyramid structure, wherein the first layer pyramid structure is used to perform preliminary decomposition on the access frequency feature vector and the modification frequency feature vector and output a preliminary decomposition matrix, and the second layer pyramid structure is used to perform deep decomposition on the preliminary decomposition matrix and output the high frequency sub-matrix and the low frequency sub-matrix;

[0044] The resource evaluation layer is used to evaluate resource consumption during data storage and access, and output resource consumption evaluation results;

[0045] The specific structure of the determination function is:

[0046] The threshold determination unit is used to set the threshold range of the data access frequency information and the data modification frequency information;

[0047] The weight calculation unit is used to calculate the weight coefficients of the high-frequency matrix and the low-frequency matrix;

[0048] The allocation decision unit generates a storage allocation strategy according to the weight coefficient;

[0049] The specific structure of the storage allocation module is:

[0050] The cache layer is used to store the data with the highest access frequency;

[0051] The primary storage layer is used to store data with medium access frequency;

[0052] The auxiliary storage layer is used to store data with low access frequency;

[0053] The specific structure of the data storage index table is:

[0054] The access permission identifier includes read permission, write permission, and execute permission;

[0055] The timestamp identifier includes a creation timestamp, an access timestamp, and a modification timestamp;

[0056] The resource consumption identification includes storing resource consumption value, reading resource consumption value, and modifying resource consumption value;

[0057] Wherein: the unit data storage resource consumption vector represents the consumption of computing resources, storage resources, and network resources required for storing a unit amount of data;

[0058] The unit data reading resource consumption vector represents the consumption of computing resources, storage resources, and network resources required for a unit data volume reading operation;

[0059] The unit data modification resource consumption vector represents the consumption of computing resources, storage resources, and network resources required for the unit data volume modification operation;

[0060] The data redundancy storage index indicates the degree of redundant backup performed during data storage to ensure data reliability.

[0061] The specific implementation of the above steps is described in detail below. The specific implementation of step S01 is to establish a four-layer neural network model, which adopts a pyramid-shaped hierarchical architecture, which is a data classification layer, a feature extraction layer, a matrix decomposition layer and a resource evaluation layer from bottom to top. In the data classification layer, the unique hot encoding technology is first used to convert the input data access frequency information and data modification frequency information into standard numerical features. This encoding method can effectively avoid the partial order relationship between data and improve the accuracy of feature expression. In the feature extraction layer, parallel long short-term memory network units and convolutional neural network units are set, wherein the long short-term memory network uses a gating mechanism to capture the temporal pattern of data access and modification, and the convolutional neural network extracts the spatial characteristics of data distribution through multi-layer convolution operations. In the matrix decomposition layer, a two-level pyramid structure is adopted, the first-level pyramid structure is responsible for the preliminary decomposition of the feature vector, and the second-level pyramid structure performs deep decomposition to obtain high-frequency and low-frequency components. In the resource evaluation layer, a multi-layer perceptron structure is used to evaluate the resource consumption of the storage and access process. The design purpose of the pyramid structure is to extract and refine data features layer by layer to achieve accurate decision-making of data storage strategies.

[0062] The specific implementation method of step S02 is to collect data access and modification frequency information in the public cloud business environment. First, the number of data accesses is collected by deploying distributed monitoring probes, the timestamp information of each access is recorded to calculate the access time interval, the number of concurrent access requests per unit time is counted to obtain the access concurrency, and the amount of data distributed to different nodes is calculated. At the same time, the number of data modification operations is recorded, the time difference between adjacent modification operations is calculated to obtain the modification time interval, the number of concurrent modification requests per unit time is counted to obtain the modification concurrency, and the version change history of the data is recorded to obtain the number of data versions. During the data collection process, it is recommended to set the sampling period of the access frequency to 1 minute, the sampling period of the modification frequency to 5 minutes, and the data retention period to 30 days. The purpose of this step is to obtain the basic characteristic information of data access and modification patterns.

[0063] The specific implementation method of step S03 is to calculate three types of resource consumption vectors. When calculating the unit data storage resource consumption vector, firstly, the CPU usage rate, memory occupancy rate, storage space size, network bandwidth occupancy rate and other indicators required to store 1MB of data are counted, and then these indicators are normalized to form a vector. When calculating the unit data reading resource consumption vector, the CPU usage rate, memory occupancy rate, disk read rate, network transmission rate and other indicators in the process of reading 1MB of data are counted, and normalization is also performed. When calculating the unit data modification resource consumption vector, the CPU usage rate, memory occupancy rate, disk write rate, network transmission rate and other indicators in the process of modifying 1MB of data are counted, and normalization is performed to obtain a vector. During the calculation process, the exponential sliding average method is used to smooth each indicator, and the smoothing factor is recommended to be set to 0.8. The role of this step is to quantitatively evaluate the resource overhead of data operations.

[0064] The specific implementation method of step S04 is to use the data classification layer to process the data access frequency information. First, a three-dimensional tensor structure is constructed to represent the data access frequency matrix. The time dimension records the time information of the access, the user dimension records the user ID that initiates the access, and the operation dimension records the specific access operation type. For the time dimension, hourly time slices are used for division; for the user dimension, user roles and permission levels are used for classification; for the operation dimension, access operations are divided into types such as reading, previewing, and downloading. Then, the category features of each dimension are converted into numerical features using the unique hot encoding method to generate a standardized data access frequency matrix. During the matrix generation process, data with an access frequency lower than 1 time per hour is marked as low-frequency access, data with an access frequency between 1 and 10 times per hour is marked as medium-frequency access, and data with an access frequency higher than 10 times per hour is marked as high-frequency access. The purpose of this step is to construct a multi-dimensional representation of data access patterns.

[0065] The specific implementation method of step S05 is to use the data classification layer to process the data modification frequency information. First, a three-dimensional tensor structure is constructed to represent the data modification frequency matrix. The time dimension records the time information of the modification operation, the user dimension records the user ID that initiates the modification, and the modification dimension records the specific modification operation type. For the time dimension, time slices at the day level are used for division; for the user dimension, user roles and modification permissions are used for classification; for the modification dimension, the modification operation is divided into types such as update, deletion, and version change. Then, the category features of each dimension are converted into numerical features using the unique hot encoding method to generate a standardized data modification frequency matrix. During the matrix generation process, data with a modification frequency lower than once a day are marked as low-frequency modifications, data with a modification frequency between 1 and 5 times a day are marked as medium-frequency modifications, and data with a modification frequency higher than 5 times a day are marked as high-frequency modifications. The purpose of this step is to construct a multi-dimensional representation of the data modification pattern.

[0066] The specific implementation method of step S06 is to process the data access frequency matrix using the feature extraction layer. First, the long short-term memory network is used to extract the time series features. The network includes an input layer, two hidden layers and an output layer. The number of neurons in the hidden layer is 128 and 64 respectively, and the activation function uses the hyperbolic tangent function. At the same time, a convolutional neural network is used to extract spatial features. The network includes two convolutional layers and a fully connected layer. The convolution kernel size is 3×3, the step size is 1, the filling method is the same filling, and the activation function uses a linear rectification function. Finally, the time series features and spatial features are fused to obtain the access frequency feature vector. During the feature extraction process, batch normalization technology is used to process the feature data, and the batch size is set to 32. The purpose of this step is to extract the deep feature representation of the data access pattern.

[0067] The specific implementation method of step S07 is to process the data modification frequency matrix using the feature extraction layer. First, the long short-term memory network is used to extract the temporal features. The network structure is the same as the access frequency feature extraction, but the input data dimension needs to be adjusted according to the actual size of the modification frequency matrix. At the same time, a convolutional neural network is used to extract spatial features. The network structure is also the same as the access frequency feature extraction, but the convolution kernel parameters need to be adjusted to adapt to the characteristics of the modification frequency data. Finally, the temporal features and spatial features are fused to obtain the modification frequency feature vector. During the feature extraction process, batch normalization technology is also used, but the batch size can be appropriately adjusted according to the scale of the modified data. The role of this step is to extract the deep feature representation of the data modification pattern.

[0068] The specific implementation method of step S08 is to use the matrix decomposition layer to process the access frequency feature vector. First, in the first layer of the pyramid structure, the non-negative matrix decomposition algorithm is used to perform a preliminary decomposition of the feature vector, and the decomposition rank is set to 80% of the original matrix rank, and the alternating least squares method is used to optimize the decomposition result. Then in the second layer of the pyramid structure, the singular value decomposition algorithm is used to perform a deep decomposition of the preliminary decomposition matrix, and the decomposition result is divided into access high-frequency components and low-frequency components according to the size of the singular value. In the division process, the part with a cumulative energy of more than 90% of the singular value is used as the high-frequency component, and the rest is used as the low-frequency component. The purpose of this step is to achieve frequency stratification of access patterns.

[0069] The specific implementation method of step S09 is to use the matrix decomposition layer to process the modified frequency feature vector. First, in the first layer of the pyramid structure, the same non-negative matrix decomposition algorithm as the access frequency processing is used for preliminary decomposition, but the decomposition rank can be set to 70% of the original matrix rank, because the modification operation usually has a strong local correlation. Then in the second layer of the pyramid structure, the singular value decomposition algorithm is also used for deep decomposition, but when dividing the high-frequency and low-frequency components, the threshold of the cumulative energy proportion of the singular value can be set to 85%, which is considered that the frequency distribution of the modification operation is usually more concentrated than the access operation. The purpose of this step is to achieve frequency stratification of the modification mode.

[0070] The specific implementation method of step S10 is to calculate the data redundant storage index using the resource evaluation layer. First, the number of backup copies of the data is counted, the geographical distance between different storage locations is calculated, and the reliability level of the storage node is evaluated. Then, a weighted sum model is constructed based on these factors, with the weight of the number of copies set to 0.4, the weight of the geographical distribution set to 0.3, and the weight of the node reliability set to 0.3. Finally, the weighted result is normalized to a range of 0 to 1 to obtain the data redundant storage index. A redundant storage index less than 0.3 indicates insufficient redundancy, between 0.3 and 0.7 indicates moderate redundancy, and greater than 0.7 indicates high redundancy. The purpose of this step is to evaluate the reliability level of data storage.

[0071] The specific implementation method of step S11 is to establish a determination function for storage allocation. In the threshold determination unit, the high-frequency threshold of the access frequency is set to 10 times per hour, and the low-frequency threshold is set to 1 time per hour; the high-frequency threshold of the modification frequency is set to 5 times per day, and the low-frequency threshold is set to 1 time per day. In the weight calculation unit, the soft maximum function is used to calculate the weight coefficients of the high-frequency and low-frequency components, considering the influence of the resource consumption vector and the redundant storage index. In the allocation decision unit, a decision tree algorithm is used to generate a storage allocation strategy based on the weight coefficient. The depth of the decision tree is set to 3 layers, and the leaf nodes correspond to different storage levels. The purpose of this step is to realize automatic decision-making of data storage strategies.

[0072] The specific implementation method of step S12 is to perform storage allocation according to the output result of the judgment function. In the cache layer, memory cache technology is used to store the data with the highest access frequency, and the cache capacity is set to 10% of the total data volume; in the primary storage layer, solid-state hard drives are used to store data with medium access frequency, and the storage capacity is set to 60% of the total data volume; in the auxiliary storage layer, mechanical hard drives are used to store data with low access frequency, and the storage capacity is set to 30% of the total data volume. In the storage allocation process, it is necessary to consider the correlation between data, and allocate data with strong correlation to the same storage level as much as possible. The purpose of this step is to realize hierarchical storage management of data.

[0073] The specific implementation method of step S13 is to construct a data storage index table. First, a unique identifier with a length of 32 bytes is generated for each data object using the uniform resource identifier rule. The identifier is composed of information such as a timestamp, a random number, and a data type identifier. The storage level number where the data is located and the physical address information in the level are recorded in the storage location identifier. The storage level number is represented by 2 bytes, and the physical address is represented by 8 bytes. A 9-bit binary bitmap is used in the access permission identifier to represent the permission combination, wherein the first 3 bits represent the owner's permissions, the middle 3 bits represent the user group permissions, and the last 3 bits represent other user permissions. Each group of 3 bits corresponds to read, write, and execute permissions. A 64-bit integer is used to record the Unix timestamp in the timestamp identifier, accurate to milliseconds, and the creation time, the last access time, and the last modification time of the data are recorded respectively. The normalized resource consumption value is recorded in the resource consumption identifier. The range of the storage resource consumption value is 0 to 100, indicating the percentage of storage space occupied, the range of the read resource consumption value is 0 to 100, indicating the percentage of read bandwidth occupied, and the range of the modification resource consumption value is 0 to 100, indicating the relative cost of the modification operation. The index table is organized in a B+ tree structure. The leaf nodes are sorted according to the data identifier. Each node can accommodate 128 index items, and the intermediate node stores the minimum key value of the index item. A multi-level cache mechanism is set up in the B+ tree. The first-level cache uses memory to store the most recently accessed index items, with a capacity of 5% of the total number of index items. The second-level cache uses a solid-state drive to store the less frequently used index items, with a capacity of 20% of the total number of index items. At the same time, auxiliary indexes are established in the index table to support fast queries based on dimensions such as access time, modification time, and resource consumption. In order to ensure the reliability of the index table, the index table is backed up regularly. The backup cycle is set to 1 hour, and the most recent 24 backup versions are retained. The purpose of this step is to establish an efficient and reliable data indexing mechanism to support fast positioning and access control.

[0074] The specific implementation method of step S14 is to implement data access control according to the index table. First, establish an access verification process. When a data access request is received, extract the user identity token in the request. The token adopts the JSON Web Token format and contains fields such as user identification, role information, and permission level. Then verify the validity of the token through the identity authentication service to check whether the signature of the token is correct and whether it is within the validity period. Then, according to the user role and permission level, use the 9-bit permission bitmap to match permissions. Only when the requested operation permission is a subset allowed by the permission bitmap can it pass the verification. After the verification is passed, the corresponding index item is searched in the B+ tree index according to the data identifier. First, search in the first-level cache. If it does not hit, search in the second-level cache. If it still does not hit, search in the disk. After obtaining the storage location information, determine the storage level where the data is located. If the data is in the cache layer, return the data directly; if it is in the primary storage layer, it is necessary to check whether the data needs to be loaded into the cache layer; if it is in the auxiliary storage layer, it is necessary to evaluate whether the level migration is required. Before returning the data, update the access timestamp. For read operations, update the last access time. For modification operations, update the last modification time at the same time. Then count the resource consumption of this access, including the actual storage space occupied, read bandwidth usage, and processor time consumption. Use the sliding window method to update the value of the resource consumption identifier, and the window size is set to 100 accesses. For situations where hierarchical migration is required, first evaluate the cost and benefit of the migration, calculate the network bandwidth and storage space required for the migration, and the possible access performance improvement after the migration. Migration is performed only when the benefit is greater than 1.5 times the cost. The migration process is performed asynchronously to avoid affecting normal data access. When data migration occurs, it is necessary to obtain a migration lock to ensure data consistency. The granularity of the lock can be at the data block level, and the default data block size is 4MB. At the same time, the dual-write mechanism is maintained during the migration process, that is, data is written to the new and old storage locations at the same time, and the data in the old location is not deleted until the migration is completed. The access control policy is evaluated and optimized regularly, with an evaluation period of 24 hours, and the cache policy and migration policy are automatically adjusted according to the statistical access mode. The role of this step is to implement a safe and efficient data access mechanism and ensure the controllability and performance of data access.

[0075] Specifically, the pyramid structure neural network model adopts a bottom-up four-layer architecture design, from the bottom to the top, which are the data classification layer, feature extraction layer, matrix decomposition layer and resource evaluation layer. Among them, the data classification layer, as the input layer of the model, uses the one-hot encoding method to process the input data, and converts the data access frequency information and data modification frequency information into a standard numerical feature vector. The structure of the data classification layer includes three parts: input nodes, encoding nodes and output nodes. The input node is responsible for receiving the original data, the encoding node performs the one-hot encoding conversion, and the output node generates a standardized feature vector. In the data classification layer, different encoding schemes are used for different types of input information. One-hot encoding is used for discrete features, interval encoding is used for continuous features, and periodic encoding is used for time features. As the second layer of the model, the feature extraction layer contains parallel long short-term memory network units and convolutional neural network units. The long short-term memory network unit is used to extract the temporal features of data access and modification patterns, and the convolutional neural network unit is used to extract the spatial features of data distribution. The long short-term memory network unit adopts a bidirectional LSTM structure, which contains 128 memory units. The activation functions of the input gate, forget gate and output gate are all Sigmoid functions, and the state update uses the hyperbolic tangent function. The convolutional neural network unit adopts a depth-separable convolution structure, which contains 3 convolution layers. The convolution kernel size of each layer is 3×3, the step size is 1, the filling method is the same filling, and the activation function uses a linear rectifier function. The matrix decomposition layer, as the third layer of the model, contains a two-level pyramid structure. The first-level pyramid structure is responsible for the preliminary decomposition of the feature vector, and the second-level pyramid structure is responsible for deep decomposition. The first-level pyramid structure adopts a non-negative matrix decomposition algorithm, and optimizes the decomposition result by the alternating least squares method. The decomposition rank is set to 80% of the original matrix rank. The second-level pyramid structure adopts a singular value decomposition algorithm, and the decomposition result is divided into high-frequency components and low-frequency components according to the energy distribution of the singular value. As the top layer of the model, the resource evaluation layer uses a multi-layer perceptron structure to evaluate resource consumption during data storage and access. It includes three fully connected layers with 256, 128, and 64 neurons, respectively. The activation function uses a parameterized linear rectification function, and the output layer uses a Softmax function.

[0076] The process of establishing the training data set includes the following steps: First, basic data is collected through the system monitoring tool, including system-level indicators such as CPU usage, memory occupancy, storage space usage, and network bandwidth usage. The sampling period is set to 1 minute, and data is collected continuously for no less than 30 days. Then, application layer data is collected through the log analysis system, including data access records, modification records, user behavior logs and other information. The statistical period is set to 1 hour, and the historical data of the last 90 days is retained. Then, the collected raw data is preprocessed, including data cleaning, outlier processing, missing value filling and other operations. The moving median method is used to remove noise in data cleaning, the 3σ principle is used to determine outliers, and the multiple interpolation method is used to fill missing values. Then, a feature matrix is ​​constructed based on the preprocessed data, including multiple dimensions such as time features, user features, and operation features. The feature selection adopts a recursive feature elimination algorithm to retain the features ranked in the top 80% of importance. Finally, the processed data set is divided into a training set, a validation set, and a test set in a ratio of 8:1:1.

[0077] The training process of the model adopts the strategy of layered pre-training and end-to-end fine-tuning. First, the data classification layer is pre-trained, using the cross entropy loss function, the optimizer uses the Adam algorithm, the learning rate is set to 0.001, the batch size is 64, and the number of training rounds is 100. Then the feature extraction layer is pre-trained, the long short-term memory network and the convolutional neural network use the mean square error loss function respectively, the optimizer also uses the Adam algorithm, the learning rate is set to 0.0005, the batch size is 32, and the number of training rounds is 200. Then the matrix decomposition layer is pre-trained, using the Frobenius norm as the loss function, the optimizer uses the stochastic gradient descent method, the learning rate is set to 0.0001, the momentum factor is 0.9, and the number of training rounds is 300. Then the resource evaluation layer is pre-trained, using the weighted cross entropy loss function, the optimizer uses the AdamW algorithm, the learning rate is set to 0.0002, the weight decay coefficient is 0.01, and the number of training rounds is 150. Finally, end-to-end fine-tuning training was performed using a multi-task learning framework. The loss function included classification loss, reconstruction loss, and prediction loss. The optimizer used the Adam algorithm with cosine annealing, with an initial learning rate of 0.0001, a minimum learning rate of 0.000001, and 500 training rounds. During the training process, an early stopping strategy was used to prevent overfitting, and gradient clipping technology was used to ensure the stability of training. The clipping threshold was set to 5.0. In order to improve the generalization ability of the model, data enhancement techniques were also used during the training process, including random noise injection, time window sliding, feature permutation, and other methods. In addition, the model complexity was controlled by introducing regularization terms. The L1 regularization coefficient was set to 0.0001, and the L2 regularization coefficient was set to 0.00001.

[0078] The main calculation process and matrix involved in the present invention are described in detail as follows:

[0079] The unit data storage resource consumption vector is specifically expressed as follows:

[0080] V s =[α c C s , α m M s , α d D s , α n N s ]+∈ s ;

[0081] Where V s It is the resource consumption vector for storing unit data; C s is the CPU usage; M s is the memory usage; D s N is the storage space size; s is the network bandwidth occupancy rate; α c , α m , α d , α n is the weight coefficient of each resource; ∈ s is a random error term.

[0082] The unit data reading resource consumption vector is specifically expressed as follows:

[0083] V r =[β c C r , β m M r, β d R d , β n Nr ] +∈ r ;

[0084] Where V r Read resource consumption vector for cell data; C r is the CPU usage; M r is the memory usage; R d is the disk read rate; N r is the network transmission rate; β c , β m , β d , β n is the weight coefficient of each resource; ∈ r is a random error term.

[0085] The unit data modification resource consumption vector is specifically expressed as follows:

[0086] V m =[γ c C m , γ c M m , γ d W d , γ n N m ]+∈ m ;

[0087] Where V m Modify the resource consumption vector for the cell data; C m is the CPU usage; M m is the memory usage; W d is the disk write rate; N m is the network transmission rate; c , γ m , γ d , γ n is the weight coefficient of each resource; ∈ m is a random error term.

[0088] The data access frequency matrix is ​​specifically expressed as follows:

[0089]

[0090] Where A is the data access frequency matrix; a ij represents the access frequency of the jth user in the ith time segment; m is the number of time segments; n is the number of users.

[0091] The data modification frequency matrix is ​​specifically expressed as follows:

[0092]

[0093] Where M is the data modification frequency matrix; m ij represents the modification frequency of the jth user in the ith time segment; k is the number of time segments; n is the number of users.

[0094] The access frequency feature vector is specifically expressed as follows:

[0095]

[0096] In the formula, F a is the access frequency feature vector; LSTM(·) represents the long short-term memory network operation; CNN(·) represents the convolutional neural network operation; represents feature fusion operation; δ a The error term is extracted for the feature.

[0097] The modified frequency feature vector is specifically expressed as follows:

[0098]

[0099] In the formula, F m To modify the frequency feature vector; LSTM(·) represents the long short-term memory network operation; CNN(·) represents the convolutional neural network operation; represents feature fusion operation; δ m The error term is extracted for the feature.

[0100] The matrix decomposition operation is specifically expressed as follows:

[0101]

[0102] Where U a , U m is a left singular matrix; S a , S m is the singular value matrix; V a , V m is a right singular matrix; E a , E m is the decomposed error term.

[0103] The data redundancy storage index is specifically expressed as follows:

[0104] R=ω 1 N c +ω 2 D g +ω 3 R n +∈ r ;

[0105] Where R is the data redundancy storage index; N c is the number of copies; D g is the geographical distance; R n is the node reliability; 1 ,ω 2 ,ω 3 is the weight coefficient; ∈ r is the estimation error term.

[0106] Description of parameter acquisition method: Parameters related to resource consumption are collected through system monitoring tools, with a sampling period of 1 minute and a sampling period of no less than 7 days; parameters related to access frequency and modification frequency are obtained through the log analysis system, with a statistical period of 24 hours; network parameters are measured through network testing tools, including bandwidth, latency, packet loss rate and other indicators; weight coefficients are obtained through machine learning training, with a training data set size of no less than 10,000 items; the error term is verified through experiments to determine its reasonable range, which is generally controlled within ±5%.

[0107] The construction principle of the above equations is as follows: the resource consumption vector adopts a linear weighted model, taking into account the contribution of different resources to the overall consumption, and introducing random error terms to adapt to the fluctuations of the actual environment; the frequency matrix adopts a two-dimensional structure, which can simultaneously express the feature distribution of the time dimension and the user dimension; the feature extraction adopts a deep learning method, LSTM can capture the time series pattern, and CNN can extract spatial features; the matrix decomposition adopts the singular value decomposition method, which can effectively separate high-frequency and low-frequency components; the redundant storage index takes into account multiple influencing factors and uses the weighted summation form for comprehensive evaluation;

[0108] The decision function consists of a threshold decision unit, a weight calculation unit, and an allocation decision unit, which are specifically expressed as follows:

[0109] The decision function of the threshold decision unit:

[0110]

[0111] Where, T a is the access frequency threshold determination result; T m To modify the frequency threshold determination result; f a is the actual access frequency; f m is the actual modification frequency; θ a is the access frequency threshold; θ m To modify the frequency threshold; ∈ t is the judgment error term.

[0112] The calculation function of the weight calculation unit:

[0113]

[0114] Where W h is the high frequency component weight; W l is the weight of the low-frequency component; H is the norm of the high-frequency component matrix; L is the norm of the low-frequency component matrix; P i is a set of component matrices; α h , α l , α i is the weight coefficient; ∈ w Compute the error term for the weights.

[0115] The decision function of the allocation decision unit:

[0116]

[0117] Where D is the final decision result; W h is the high frequency component weight; W l is the low-frequency component weight; R is the redundant storage index; V is the resource consumption vector; λ 1 ,λ 2 ,λ3 ,λ 4 is the decision coefficient; is the time partial derivative of resource consumption; ∈ d is the decision error term.

[0118] Among them, accessing the high-frequency matrix and accessing the low-frequency matrix are specifically expressed as follows:

[0119]

[0120] In the formula, h ij and l ij They represent the high-frequency access features and low-frequency access features of the j-th user in the ith time segment, respectively; m is the number of time segments; and n is the number of users.

[0121] Among them, modifying the high-frequency matrix and modifying the low-frequency matrix are specifically expressed as follows:

[0122]

[0123] In the formula, h′ ij and l′ ij They respectively represent the high-frequency modification features and low-frequency modification features of the j-th user in the ith time segment; k is the number of time segments; and n is the number of users.

[0124] Among them, the derivation process of the threshold determination unit is:

[0125] 1. First consider a simple threshold decision function: g(x) = sign(x-θ);

[0126] 2. In order to avoid the discontinuity of the function, the Sigmoid function is introduced for smoothing:

[0127] 3. Consider the noise impact of the actual environment and add error terms:

[0128] 4. Apply to access frequency and modification frequency judgment respectively to obtain the final judgment function:

[0129]

[0130] Among them, the derivation process of the weight calculation unit is:

[0131] 1. The initial weight calculation adopts the exponential form:

[0132] 2. To ensure that the sum of weights is 1, normalization is introduced:

[0133] 3. Consider the weight calculation of high-frequency components and low-frequency components separately:

[0134]

[0135] Among them, the derivation process of the allocation decision unit is:

[0136] 1. The basic decision function considers the logarithmic form of the weight: d 1 =λ 1 ln(W h )+λ 2 ln(W l );

[0137] 2. Nonlinear impact of introducing redundant storage: d 2 =λ 3 R 2 ;

[0138] 3. Consider the dynamic changes in resource consumption:

[0139] 4. Combine to get the final decision function:

[0140]

[0141] How to obtain relevant parameters: θ a ,θ m : Obtained through historical data statistics, generally taking the 80% quantile value; α h , α l , α i : obtained by training with stochastic gradient descent method; λ 1 ,λ 2 ,λ 3 ,λ 4 : Solved by multi-objective optimization algorithm; ∈ t ,∈ w ,∈ d : The reasonable range is determined through experimental verification and generally does not exceed 0.1.

[0142] The principle of equation construction is as follows: the threshold judgment uses Sigmoid function to achieve a smooth threshold conversion process; the weight calculation uses Softmax function to ensure that the sum of weights is 1 and non-negative; the introduction of logarithmic terms in the decision function makes the weight change more sensitive; the introduction of square terms reflects the nonlinear impact of redundant storage; the time partial derivative reflects the dynamic trend of resource consumption. Smooth threshold judgment is achieved to avoid the discontinuity caused by hard thresholds; the weight calculation takes into account the mutual influence of multiple components; the decision function comprehensively considers static features and dynamic changes; the introduction of nonlinear transformation improves the expression ability of the model.

[0143] A second aspect of the present invention provides a computer-readable storage medium, in which program instructions are stored. When the program instructions are run in a computer, they are used to execute the above-mentioned method for private data storage and access of a public cloud business.

[0144] A third aspect of the present invention provides a private data storage and access system for a public cloud business, comprising the above-mentioned computer-readable storage medium, wherein the system is any one of a computer, a server, and a single-chip microcomputer, wherein the computer-readable storage medium is arranged in the system, and wherein the system is provided with a microprocessor for executing program instructions stored in the computer-readable storage medium.

[0145] Specifically, the principle of the present invention is: the core technical principle of the present invention is based on the theory of deep learning and resource optimization. The pyramid structure neural network model can effectively capture the temporal features and spatial features in the data access pattern through multi-level feature extraction and analysis. Among them, the data classification layer uses a unique hot encoding method to process the input data, ensuring the integrity and accuracy of the feature representation; the feature extraction layer uses parallel long short-term memory network units and convolutional neural network units to extract temporal features and spatial features respectively, realizing a comprehensive analysis of the data access pattern.

[0146] In terms of resource evaluation, the unit data resource consumption vector introduced in the present invention innovatively quantifies the consumption of computing resources, storage resources and network resources in the data operation process. This representation method enables the system to accurately evaluate the resource overhead of different types of data operations, providing a reliable basis for storage allocation decisions. At the same time, the introduction of the data redundancy storage index enables the system to dynamically adjust the data redundancy storage strategy according to the importance of the data and the system resource status.

[0147] The design of the decision function adopts a multi-level decision mechanism. Through the coordinated work of the threshold decision unit, the weight calculation unit and the allocation decision unit, the intelligent formulation of the storage allocation strategy is realized. This decision mechanism not only considers traditional features such as data access frequency and modification frequency, but also comprehensively considers innovative indicators such as resource consumption and redundant storage, making the storage allocation decision more comprehensive and reasonable.

[0148] A specific embodiment 1 of the present invention is provided below, and the specific implementation method of each step in this embodiment 1 is described in detail as follows.

[0149] The specific implementation method of step S01 is to establish a pyramid structure neural network model. The model adopts a bottom-up four-layer architecture design, including a data classification layer, a feature extraction layer, a matrix decomposition layer and a resource assessment layer. The data classification layer serves as the input layer of the model, and uses a unique hot encoding method to process the input data, converting the data access frequency information and the data modification frequency information into a standard numerical feature vector, which specifically includes three parts: an input node, an encoding node and an output node. The input node is responsible for receiving the original data, the encoding node performs the unique hot encoding conversion, and the output node generates a standardized feature vector. Different encoding schemes are used for different types of input information. Unique hot encoding is used for discrete features, interval encoding is used for continuous features, and periodic encoding is used for time features. The mathematical expression of unique hot encoding is:

[0150] X onehot =E·X raw +∈ e ;

[0151] Where, X onehot is the encoded feature vector; X raw is the original input data; E is the encoding matrix; ∈ e is the encoding error term. The feature extraction layer is the second layer of the model, which includes parallel long short-term memory network units and convolutional neural network units. The long short-term memory network unit adopts a bidirectional structure and contains 128 memory units. The activation functions of the input gate, forget gate and output gate all use the Sigmoid function, and the state update uses the hyperbolic tangent function. The mathematical expression of the long short-term memory unit is:

[0152] f t =σ(W f ·[h t-1 , x t ]+b f );

[0153] i t =σ(W i ·[h t-1 , x t ]+b i );

[0154] o t =σ(W o ·[h t-1 , x t ]+b o );

[0155] c t =f t ·c t-1 +i t tanh(W c ·[h t-1, x t ]+b c );

[0156] h t =o t ·tanh(c t );

[0157] In the formula, f t ,i t , o t are the states of the forget gate, input gate, and output gate respectively; c t is the unit state; h t is the hidden state; W f , W i , W o , W c is the weight matrix; b f , b i , b o , b c is the bias vector; σ is the Sigmoid function; tanh is the hyperbolic tangent function. The convolutional neural network unit adopts a depth-separable convolution structure, which contains 3 convolution layers. The convolution kernel size of each layer is 3×3, the step size is 1, the padding method is the same padding, and the activation function uses a linear rectification function. The mathematical expression of the convolution operation is:

[0158] Y = Conv(X, W) + b;

[0159] In the formula, Y is the convolution output; X is the input feature map; W is the convolution kernel weight; and b is the bias term. The matrix decomposition layer, as the third layer of the model, contains a two-level pyramid structure. The first-level pyramid structure adopts a non-negative matrix decomposition algorithm and optimizes the decomposition result by alternating least squares. The second-level pyramid structure adopts a singular value decomposition algorithm to divide the decomposition result into high-frequency components and low-frequency components according to the energy distribution of the singular value. The resource evaluation layer, as the top layer of the model, uses a multi-layer perceptron structure to evaluate the resource consumption during data storage and access. It contains three fully connected layers with 256, 128, and 64 neurons, respectively. The training of the model adopts a layered pre-training and end-to-end fine-tuning strategy. The loss function includes three parts: classification loss, reconstruction loss, and prediction loss.

[0160] The specific implementation method of step S02 is to collect data access frequency information and data modification frequency information in the public cloud business. The collection of data access frequency information is achieved by deploying distributed monitoring probes, which are arranged at key nodes of the system, including load balancers, application servers, and storage nodes. First, the number of data accesses is recorded, and the number of access operations per unit time is counted using a counter. The counter is updated using atomic operations to ensure concurrency safety. The statistical function of the number of accesses is expressed as:

[0161]

[0162] In the formula, C a (t) is the number of visits at time t; a i (t) is the count value of the ith access operation; n is the total number of operations in the sampling period; ∈ c is the counting error term. Then calculate the access time interval and record the time difference between adjacent access operations. The calculation function of the time interval is:

[0163] T i (t) = t i+1 -t i +∈ t ;

[0164] Where, T i (t) is the time interval between the i-th visit; t i is the timestamp of the access operation; ∈ t is the time error term. Then count the concurrent access volume, record the number of access requests that occur simultaneously per unit time, and the statistical function of the concurrent volume is:

[0165]

[0166] Where P a (t) is the concurrency in the time window; Δt is the statistical window size; ∈ p is the concurrent statistical error term. Finally, the data distribution volume is calculated, and the statistical data is the amount of data distributed from the source node to the target node. The calculation function of the distribution volume is:

[0167]

[0168] Where D a (t) is the data distribution amount at time t; d j (t) is the amount of distributed data of the jth node; m is the number of nodes; ∈ d is the distribution statistics error term.

[0169] The collection of data modification frequency information is also achieved through distributed monitoring probes. First, record the number of data modifications and count the number of modification operations per unit time. The statistical function of the number of modifications is:

[0170]

[0171] In the formula, C m (t) is the number of modifications at time t; m k (t) is the count value of the kth modification operation; l is the total number of operations in the sampling period; ∈ mis the modification count error term. Then calculate the modification time interval and record the time difference between adjacent modification operations. The calculation method of the time interval is similar to the access time interval. Then count the modification concurrency and record the number of modification requests that occur simultaneously in a unit of time. The statistical method is similar to the access concurrency. Finally, record the number of data versions and the version change history of the data object. The statistical function of the version number is:

[0172]

[0173] Where V n (t) is the number of versions at time t; v r (t) is the count value of the rth version; s is the total number of versions; ∈ v is the version statistics error term.

[0174] The specific implementation of step S03 is to calculate the unit data storage resource consumption vector, the unit data reading resource consumption vector, and the unit data modification resource consumption vector. The unit data storage resource consumption vector is specifically expressed as:

[0175] V s =[α c C s , α m M s , α d D s , α n N s ]+∈ s ;

[0176] Where V s The resource consumption vector for storing unit data; C s is the CPU usage; M s is the memory usage; D s N is the storage space size; s is the network bandwidth occupancy rate; α c , α m , α d , α n is the weight coefficient of each resource; ∈ s is a random error term. The unit data reading resource consumption vector is specifically expressed as:

[0177] V r =[β c C r , β m M r , β d R d , β n N r ]+∈ r ;

[0178] Where V r Read resource consumption vector for cell data; C r is the CPU usage; M r is the memory usage; R d is the disk read rate; N r is the network transmission rate; β c , β m , β d , β n is the weight coefficient of each resource; ∈ r is a random error term. The unit data modification resource consumption vector is specifically expressed as:

[0179] V m =[γ c C m , γ m M m , γ d W d , γ n N m ]+∈ m ;

[0180] Where V m Modify the resource consumption vector for the cell data; C m is the CPU usage; M m is the memory usage; W d is the disk write rate; N m is the network transmission rate; c , γ m , γ d , γ n is the weight coefficient of each resource; ∈ m is a random error term.

[0181] The specific implementation of step S04 is to use the data classification layer to classify the data access frequency information and generate a data access frequency matrix. First, a three-dimensional tensor structure is constructed to represent the data access frequency matrix, which is specifically represented as follows:

[0182]

[0183] Where A is the data access frequency matrix; a ij represents the access frequency of the jth user in the i-th time segment; m is the number of time segments; n is the number of users. The access frequency matrix is ​​processed using a one-hot encoding method, and the encoding function is expressed as:

[0184] E a =OneHot(A)+δ a ;

[0185] In the formula, Ea is the access frequency matrix after encoding; OneHot(·) is the one-hot encoding function; δ a is the encoding error term. The encoding of the time dimension uses a periodic function:

[0186] T e =sin(2πf t ·t)+cos(2πf t ·t)+∈ t ;

[0187] Where, T e is the time code value; f t is the time period parameter; t is the timestamp; ∈ t is the time coding error term.

[0188] The specific implementation method of step S05 is to use the data classification layer to classify the data modification frequency information and generate a data modification frequency matrix. The representation form of the modification frequency matrix is:

[0189]

[0190] Where M is the data modification frequency matrix; m ij represents the modification frequency of the jth user in the i-th time segment; k is the number of time segments; n is the number of users. The encoding process of the modification frequency matrix is ​​similar to that of the access frequency matrix, but the encoding parameters need to be adjusted according to the characteristics of the modification operation. The encoding function is expressed as:

[0191] E m =OneHot(M)+δ m ;

[0192] In the formula, E m is the modified frequency matrix after encoding; OneHot(·) is the one-hot encoding function; δ m is the coding error term.

[0193] The specific implementation of step S06 is to process the data access frequency matrix using the feature extraction layer. Feature extraction uses the parallel processing of long short-term memory network and convolutional neural network. The processing process of long short-term memory network is expressed as:

[0194] F lstm =LSTM(E a )+∈ lstm ;

[0195] In the formula, F lstm is the time series feature; LSTM(·) is the long short-term memory network function; ∈ lstm is the feature extraction error term. The processing process of the convolutional neural network is expressed as:

[0196] F cnn =CNN(E a )+∈ cnn ;

[0197] In the formula, F cnn is the spatial feature; CNN(·) is the convolutional neural network function; ∈ cnn is the feature extraction error term. The feature fusion process is expressed as:

[0198]

[0199] In the formula, F a is the access frequency feature vector; represents feature fusion operation; δ a is the feature fusion error term.

[0200] The specific implementation of step S07 is to process the data modification frequency matrix using the feature extraction layer. The extraction process of the modification frequency feature is similar to that of the access frequency feature, but the network parameters need to be adjusted according to the characteristics of the modification operation. The temporal feature extraction is expressed as:

[0201] F lstm ′=LSTM(E m )+∈ lstm ′;

[0202] In the formula, F lstm ′ is the modified time series feature; LSTM(·) is the long short-term memory network function; ∈ lstm ′ is the feature extraction error term. Spatial feature extraction is expressed as:

[0203] F cnn ′=CNN(E m )+∈ cnn ′;

[0204] In the formula, F cnn ′ is the modified spatial feature; CNN(·) is the convolutional neural network function; ∈ cnn ′ is the feature extraction error term. The feature fusion process is expressed as:

[0205]

[0206] In the formula, F m To modify the frequency feature vector; represents feature fusion operation; δ m is the feature fusion error term. In the feature extraction process, batch normalization technology is used to process feature data, and the normalization function is expressed as:

[0207]

[0208] In the formula, B n is the normalized result; x is the input feature; μ is the batch mean; σ 2 is the batch variance; ∈ is the numerical stability coefficient.

[0209] The specific implementation of step S08 is to use the matrix decomposition layer to perform matrix decomposition operation on the access frequency feature vector. First, a preliminary decomposition is performed in the first layer of the pyramid structure, using a non-negative matrix decomposition algorithm, and its optimization objective function is:

[0210]

[0211] In the formula, F a is the access frequency feature vector; U, V are decomposition matrices; ||·|| F is the Frobenius norm; α, β are regularization coefficients. The update rule of the decomposition result is:

[0212]

[0213] In the formula, ⊙ represents the Hadamard product; t is the number of iterations. Then, deep decomposition is performed in the second-level pyramid structure using the singular value decomposition algorithm:

[0214]

[0215] Where U a is a left singular matrix; S a is the singular value matrix; V a is a right singular matrix; E a is the decomposed error term.

[0216] The specific implementation of step S09 is to use the matrix decomposition layer to perform matrix decomposition operation on the modified frequency feature vector. The optimization goal of the non-negative matrix decomposition of the first layer of pyramid structure is:

[0217]

[0218] In the formula, F m is the modified frequency feature vector; P, Q are decomposition matrices; γ, δ are regularization coefficients. The update rule of the decomposition result is:

[0219]

[0220] Where t is the number of iterations. The singular value decomposition of the second-level pyramid structure is expressed as:

[0221]

[0222] Where U mis a left singular matrix; S m is the singular value matrix; V m is a right singular matrix; E m is the decomposed error term.

[0223] The specific implementation of step S10 is to use the resource evaluation layer to calculate the data redundant storage index. The calculation function of the redundant storage index is:

[0224] R=ω 1 N c +ω 2 D g +ω 3 R n +∈ r ;

[0225] Where R is the data redundancy storage index; N c is the number of copies; D g is the geographical distance; R n is the node reliability; 1 ,ω 2 ,ω 3 is the weight coefficient; ∈ r To evaluate the error term, the exponential decay model is used to calculate the node reliability:

[0226] R n =e -λt +∈ n ;

[0227] Where λ is the reliability attenuation coefficient; t is the operating time; ∈ n is the reliability error term.

[0228] The specific implementation of step S11 is to establish a determination function. The determination function of the threshold determination unit is:

[0229]

[0230] Where, T a is the access frequency threshold determination result; T m To modify the frequency threshold determination result; f a is the actual access frequency; f m is the actual modification frequency; θ a is the access frequency threshold; θ m To modify the frequency threshold; ∈ t is the judgment error term. The calculation function of the weight calculation unit is:

[0231]

[0232] Where W h is the high frequency component weight; W lis the weight of the low-frequency component; H is the norm of the high-frequency component matrix; L is the norm of the low-frequency component matrix; P i is a set of component matrices; α h , α l , α i is the weight coefficient; ∈ w The error term is calculated for the weight. The decision function for allocating decision units is:

[0233]

[0234] Where D is the final decision result; 1 ,λ 2 ,λ 3 ,λ 4 is the decision coefficient; is the time partial derivative of resource consumption; ∈ d is the decision error term.

[0235] The specific implementation of step S12 is to perform multi-level storage allocation according to the calculation result of the determination function. First, the capacity of the storage layer is initialized, and the cache layer capacity calculation function is:

[0236] C h =α h ·C total +∈ c ;

[0237] In the formula, C h is the cache layer capacity; C total is the total amount of data; α h is the capacity coefficient, which takes a value of 0.1; ∈ c is the capacity error term. The primary storage layer capacity calculation function is:

[0238] C m =α m ·C total +∈ m ;

[0239] In the formula, C m is the capacity of the primary storage layer; α m is the capacity coefficient, which is 0.6; ∈ m is the capacity error term. The auxiliary storage layer capacity calculation function is:

[0240] C l =α l ·C total +∈ l ;

[0241] In the formula, C l is the capacity of the auxiliary storage layer; α l is the capacity coefficient, which is 0.3; ∈l is the capacity error term. The data correlation evaluation function is:

[0242]

[0243] In the formula, R c is the correlation coefficient; x i ,y j is the data item; μ x , μ y is the mean; ∈ r is the estimation error term.

[0244] The specific implementation method of step S13 is to construct a data storage index table. First, a globally unique identifier is assigned to each data object, and the physical location of the data in different storage levels is recorded. Then, three basic permissions of read, write and execute are set, and the permission combination is represented by a bitmap. Then the creation time, last access time and last modification time of the data are recorded, and the timestamp adopts a unified format. Finally, the resource consumption value of the data operation is recorded, including storage space occupancy, read bandwidth consumption and modification operation overhead. The index table is organized in a B+ tree structure to improve query efficiency. The purpose of this step is to establish an efficient data indexing mechanism.

[0245] The specific implementation method of step S14 is to implement data access control based on the index table. First, verify the user identity and operation authority in the access request to ensure that they meet the requirements of the authority identification. Then locate the physical location of the data according to the storage location identification, and select the appropriate storage level for access. Then update the timestamp information to record the time of the access or modification operation. Finally, count the resource consumption and update the relevant values ​​of the resource consumption identification. During the access control process, for cross-level data access, the cost of data migration needs to be considered. The purpose of this step is to ensure the security and efficiency of data access.

[0246] In order to better understand and implement the present invention, the following provides Example 2 of a specific application scenario of the present invention: A cloud computing service provider implemented the data private storage and access method of the present invention to improve the storage efficiency of the data center. The data center contains 10,000 servers, with an average daily data processing volume of 100TB and more than 1 million users. The data storage solution before implementation adopted a unified three-copy storage strategy, resulting in low storage space utilization and uneven data access performance.

[0247] First, a pyramid structure neural network model is constructed and implemented using the PyTorch deep learning framework. The network structure parameters are shown in Table 1:

[0248] Table 1 Neural network model structure parameters

[0249] Network Layer Parameter configuration Activation Function Output Dimensions Data classification layer Input nodes: 256 ReLU 512 Feature extraction layer LSTM Hidden units: 128 Tanh 256 Feature extraction layer CNN Convolution kernel: 3×3 ReLU 256 Matrix factorization layer 1 Decomposition rank: 80% - 128 Matrix factorization layer 2 Number of singular values: 64 - 64 Resource Assessment Layer Fully connected layers: 256 / 128 / 64 PReLU 32

[0250] The system operation data for 30 days was collected, including access frequency and modification frequency information. The data sampling results are shown in Table 2:

[0251] Table 2 Statistics of data access and modification frequency

[0252]

[0253] Figure 2 The comparison of system access volume and concurrency in different time periods is shown. The resource consumption vector is calculated based on the collected data, where the weight coefficients of CPU usage, memory occupancy, storage space and network bandwidth are set to 0.3, 0.2, 0.3 and 0.2 respectively. The resource consumption statistics of unit data volume (1MB) are shown in Table 3:

[0254] Table 3 Statistics of resource consumption per unit data volume

[0255]

[0256] Figure 3 The resource consumption of different operation types is shown, including indicators such as CPU usage, memory occupancy, storage space, and network bandwidth. The data is processed using a pyramid structure neural network model. First, an access frequency matrix and a modification frequency matrix are generated. The matrix size is 24×1000000, which represents the access and modification of 1 million users in 24 time segments. The access frequency feature vector and the modification frequency feature vector are obtained through feature extraction, and the vector dimension is 256. After matrix decomposition, the feature vector obtains high-frequency components and low-frequency components, where the energy distribution of the access frequency is shown in Table 4:

[0257] Table 4 Access frequency energy distribution

[0258] Component Type Singular value energy ratio Data share Access frequency range (times / second) High frequency component 92.5% 15.8% >10 Low frequency component 7.5% 84.2% ≤10

[0259] Figure 4 The heat map shows the data access frequency distribution in different time periods within a week, reflecting the periodic characteristics of the system access pattern. The data redundancy storage index is calculated based on the decomposition results, considering the number of copies, geographical distance and node reliability, with weights of 0.4, 0.3 and 0.3 respectively. The formula for calculating the redundant storage index is: R = 0.4N c +0.3D g +0.3R n +∈ r .

[0260] A judgment function is established based on the calculation results. The specific form of the function is: The decision coefficient λ 1,λ 2 ,λ 3 ,λ 4 The final values ​​are 0.3, 0.2, 0.3, and 0.2 respectively through the multi-objective optimization algorithm. The output of the decision function is used to determine the data storage strategy. The storage level division is shown in Table 5:

[0261] Table 5 Storage level division

[0262] Storage Tiers Capacity ratio Storage Media Access delay Data Types Cache Layer 10% Memory <1ms High-frequency access to data Primary storage layer 60% SSD <10ms IF Access Data Secondary storage layer 30% HDD <100ms Low-frequency access data

[0263] Figure 5 The performance indicators of the three storage levels are compared, including response time, throughput, and resource utilization, and a data storage index table is established, which contains information such as data identifier, storage location, access rights, timestamp, and resource consumption. The index table is organized in a B+ tree structure, the tree order is set to 128, and the fill factor of the leaf node is 0.8. The structure of the index table is shown in Table 6:

[0264] Table 6 Example of data storage index table

[0265] Data Identifiers Storage Location Access Rights Creation time Last access time Resource consumption value 0x1A2B3C Cache / 0x001 rwx—— 2024-01-01 2024-01-21 85.6 0x4D5E6F Main storage / 0x102 rw-rw—- 2024-01-05 2024-01-20 62.3 0x7G8H9I Auxiliary storage / 0x203 rw-r—– 2024-01-10 2024-01-15 28.9

[0266] Compared with the traditional unified three-copy storage solution, the method adopted by the present invention has the following advantages: First, the traditional solution adopts the same storage strategy for all data, while the present invention performs hierarchical storage according to data access characteristics, which improves the utilization rate of storage space. Experimental results show that storage space is saved by 35%. Second, the data access performance of the traditional solution is uneven, while the present invention optimizes access performance through a multi-layer storage structure, and the average response time of high-frequency access data is reduced by 80%. Third, the traditional solution lacks a fine-grained access control mechanism, while the present invention achieves precise access control by establishing a detailed index table, thereby improving data security. Fourth, the traditional solution cannot adapt to the dynamic changes in data access patterns, while the present invention can adjust the storage strategy in real time through a neural network model, and the overall performance of the system is improved by 45%. After applying the storage solution of the present invention, the operating cost of the data center was reduced by 25%, and user satisfaction was improved by 40%, which fully verified the practical value of the method.

[0267] It should be noted that the variables involved in the present invention are explained in detail as shown in Tables 7 and 8 below.

[0268] Table 7 Variable explanation table (Part I)

[0269]

[0270]

[0271] Table 8 Variable explanation table (Part II)

[0272]

[0273]

[0274] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for private storage and access of data for a public cloud service, characterized in that: The following steps are involved: Establishing and training a pyramid structure neural network model, wherein the pyramid structure neural network model includes a data classification layer, a feature extraction layer, a matrix decomposition layer, and a resource evaluation layer; collecting data access frequency information and data modification frequency information in public cloud services; Calculate the unit data storage resource consumption vector, the unit data reading resource consumption vector, and the unit data modification resource consumption vector; use the pyramid structure neural network model to classify the data access frequency information and the data modification frequency information to generate a data access frequency matrix and a data modification frequency matrix; perform feature extraction on the data access frequency matrix and the data modification frequency matrix to obtain an access frequency feature vector and a modification frequency feature vector; Utilizing the matrix decomposition layer to perform matrix decomposition operations, obtaining access to a high-frequency matrix, access to a low-frequency matrix, modification of a high-frequency matrix, and modification of a low-frequency matrix; Calculating the data redundancy storage index and establishing a determination function, and performing multi-level storage allocation according to the calculation result of the determination function; Establish a data storage index table and implement private data storage and access control.

2. The method for private storage and access to public cloud business data according to claim 1, characterized in that: The specific structure of the pyramid structure neural network model includes: the data classification layer processes the input data using a one-hot encoding method, and converts the data access frequency information and the data modification frequency information into standard numerical features; the feature extraction layer includes parallel long short-term memory network units and convolutional neural network units, the long short-term memory network units are used to extract temporal features, and the convolutional neural network units are used to extract spatial features; the matrix decomposition layer includes a first-layer pyramid structure and a second-layer pyramid structure, the first-layer pyramid structure is used to perform preliminary decomposition of the access frequency feature vector and the modification frequency feature vector, and the second-layer pyramid structure is used to perform deep decomposition of the preliminary decomposition matrix; the resource evaluation layer is used to evaluate resource consumption during data storage and access.

3. The method for private storage and access to public cloud business data according to claim 1, characterized in that: The data access frequency information includes the number of data accesses, access time interval, access concurrency, and data distribution volume; the data modification frequency information includes the number of data modifications, modification time interval, modification concurrency, and data version number.

4. The method for private storage and access to public cloud data according to claim 1, characterized in that: The data access frequency matrix includes a time dimension, a user dimension, and an operation dimension, and the data modification frequency matrix includes a time dimension, a user dimension, and a modification dimension.

5. The method for private storage and access to public cloud business data according to claim 1, characterized in that: The determination function includes a threshold determination unit, a weight calculation unit, and an allocation decision unit. The input parameters of the determination function include accessing the high-frequency sub-matrix, accessing the low-frequency sub-matrix, modifying the high-frequency sub-matrix, modifying the low-frequency sub-matrix, unit data storage resource consumption vector, unit data reading resource consumption vector, unit data modification resource consumption vector, and data redundancy storage index.

6. The method for privately storing and accessing data of a public cloud service according to claim 1, characterized in that: The multi-level storage allocation includes a cache layer, a primary storage layer, and an auxiliary storage layer, wherein the cache layer is used to store data with the highest access frequency, the primary storage layer is used to store data with a medium access frequency, and the auxiliary storage layer is used to store data with a low access frequency.

7. The method for private storage and access to public cloud business data according to claim 1, characterized in that: The data storage index table includes a data identifier, a storage location identifier, an access permission identifier, a timestamp identifier, and a resource consumption identifier, wherein the access permission identifier includes a read permission, a write permission, and an execute permission; the timestamp identifier includes a creation timestamp, an access timestamp, and a modification timestamp; the resource consumption identifier includes a storage resource consumption value, a read resource consumption value, and a modification resource consumption value.

8. The method for private storage and access to public cloud business data according to claim 1, characterized in that: The unit data storage resource consumption vector represents the consumption of computing resources, storage resources, and network resources required for storing a unit amount of data; the unit data reading resource consumption vector represents the consumption of computing resources, storage resources, and network resources required for a unit data reading operation; the unit data modification resource consumption vector represents the consumption of computing resources, storage resources, and network resources required for a unit data modification operation; and the data redundancy storage index represents the degree of redundant backup performed during the data storage process to ensure data reliability.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, and when the program instructions are executed in a computer, they are used to execute the method for private storage and access to data of a public cloud service according to any one of claims 1 to 8.

10. A data private storage and access system for public cloud services, characterized in that: The system comprises the computer-readable storage medium as claimed in claim 9, wherein the system is any one of a computer, a server, and a single-chip microcomputer, the computer-readable storage medium is arranged in the system, and the system is provided with a microprocessor for executing program instructions stored in the computer-readable storage medium.

Citation Information

Patent Citations

  • File hierarchical storage method and system taking visit randomness into consideration

    CN104376094A

  • File data hierarchical storage method and device, medium and electronic equipment

    CN118672520A

  • Data dynamic storage method and device, equipment and storage medium

    CN118915978A

  • Data storage method and system based on legal knowledge service platform and storage medium

    CN118964496A

  • Cloud computing resource allocation method based on dynamic optimization, computer device and storage medium

    CN119166354A

Cited By

  • Elastic expansion type hierarchical storage system suitable for campus educational administration and safety information

    CN121029080A

  • Low-energy-consumption positioning data management system for large-scale sensing network

    CN121722312A