Data dynamic storage method and device, equipment and storage medium

By analyzing historical access logs and deep learning models to predict future access popularity, dynamically adjusting the data storage level, solving the problem of difficult balance between cost and performance in traditional storage solutions, and achieving efficient utilization and automated management of storage resources.

CN118915978AInactive Publication Date: 2024-11-08深圳市金政软件技术有限公司

Patent Information

Application Number
CN202411424307.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-12
Publication Date
2024-11-08
Estimated Expiration
Not applicable · inactive patent

Smart Images

  • Figure CN118915978A_ABST
    Figure CN118915978A_ABST
Patent Text Reader

Abstract

The invention relates to the field of computers, and discloses a data dynamic storage method and device, equipment and a storage medium. The method comprises the following steps: acquiring a mapping relationship between data and a data storage layer; when the target data is accessed, routing to the corresponding data storage layer to read the target data, and recording the target data into the data historical access log; according to the data historical access log, performing regular analysis to obtain historical access frequency, time dimension trend and periodic characteristics of the stored data; according to the historical access frequency, the time dimension trend and the periodic characteristics of the storage data, predicting the future access heat of the storage data through an integrated deep learning model; and sequencing the future access popularity of the storage data, and migrating the storage data to different data storage layers according to the sequencing of the future access popularity. In the embodiment of the invention, the data storage can be dynamically adjusted, the storage resources are reasonably distributed, and the utilization rate of the storage resources is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a data dynamic storage method, device, equipment and storage medium. Background Art

[0002] With the rapid development of the Internet and the Internet of Things, related business data has shown an explosive growth trend. According to an IDC report, the amount of global data will grow at a rate of 40+% in the next 10 years, and the difficulty of data storage and system support pressure will increase sharply.

[0003] At present, most data storage solutions are still based on traditional storage solutions. Traditional storage solutions generally use a single storage medium, and it is often difficult to find the best balance between cost and performance. High-performance storage media (such as SSD) have high performance but are expensive, while low-cost storage (such as HDD) is economical but has slow access speed. Since the trend of data changes is difficult to predict in advance, traditional storage solutions are prone to idle resources and over-configuration. In addition, many traditional storage systems lack sufficient automated management capabilities, especially in data classification, migration and lifecycle management. Manual intervention is not only time-consuming, but also prone to errors. It is difficult to accurately predict and adapt to dynamic changes in data access patterns. Summary of the invention

[0004] The main purpose of the present invention is to solve the technical problem of how to dynamically adjust data storage and improve the utilization rate of storage resources.

[0005] A first aspect of the present invention provides a method for dynamically storing data, the method comprising: Get the mapping relationship between data and data storage layer; When accessing target data, according to the key of the target data and the mapping relationship, route to the corresponding data storage layer to read the target data, and record the target data in the data history access log; According to the data historical access log, regularly analyze the historical access frequency, time dimension trend and periodic characteristics of the stored data; According to the historical access frequency, time dimension trend and periodic characteristics of the stored data, the future access popularity of the stored data is predicted through an integrated deep learning model; The future access heat of the stored data is sorted, and the stored data is migrated to different data storage layers according to the sorting of the future access heat.

[0006] Optionally, in a first implementation of the first aspect of the present invention, sorting the future access heat of the stored data, and migrating the stored data to different data storage layers according to the sorting of the future access heat includes: sorting the future access popularity of the stored data; Obtaining the last round ranking of the future access heat of the stored data, and comparing the ranking of the future access heat with the last round ranking of the access heat to obtain the number of differences in the access heat ranking of the stored data; The stored data is migrated to different data storage layers according to the number of differences in the access heat rankings.

[0007] Optionally, in a second implementation of the first aspect of the present invention, migrating the stored data to different data storage layers according to the number of differences in the access popularity sorting includes: Determine whether the number of differences in the access popularity ranking exceeds a preset number threshold; If it exceeds, a data storage level adjustment plan is generated according to the access popularity sorting; According to the data storage level adjustment scheme, the stored data is migrated to different data storage layers.

[0008] Optionally, in a third implementation of the first aspect of the present invention, migrating the stored data to different data storage layers according to the data storage level adjustment scheme includes: Calculate the remaining storage capacity of each data storage layer; Determine whether the remaining storage capacity is lower than a preset first capacity threshold and higher than a preset second capacity threshold, wherein the preset first capacity threshold refers to a threshold for determining whether to perform capacity compression, and the preset second capacity threshold refers to a threshold for determining whether to perform capacity expansion; If so, the stored data is migrated to a different data storage layer according to the data storage level adjustment scheme in combination with TLS / SSL encryption.

[0009] Optionally, in a fourth implementation of the first aspect of the present invention, before migrating the stored data to different data storage layers according to the data storage level adjustment scheme and in combination with TLS / SSL encryption, the method further includes: If the remaining storage capacity is higher than the preset first capacity threshold, reducing the storage nodes of the data storage layer to achieve dynamic compression through a data compression algorithm; If the remaining storage capacity is lower than the preset second capacity threshold, dynamic capacity expansion is achieved by adding storage nodes of the data storage layer through k8s.

[0010] Optionally, in a fifth implementation of the first aspect of the present invention, migrating the stored data to different data storage layers according to the data storage level adjustment scheme in combination with TLS / SSL encryption includes: Obtaining a session request sent by a client, and sending the server's digital certificate and public key to the client; Obtaining a master key sent by the client, where the master key is extracted by the client from the digital certificate and encrypted with the public key; Decrypting the master key using the private key, and generating a session key based on the decrypted master key and random information; According to the data storage level adjustment scheme, the stored data is migrated to different data storage layers through the session key.

[0011] Optionally, in a sixth implementation of the first aspect of the present invention, when accessing the target data, routing to the corresponding data storage layer to read the target data according to the key of the target data and the mapping relationship, and recording the target data in the data history access log includes: When storing the target data, encrypting the target data according to the type of the target data; When accessing the target data, routing to the corresponding data storage layer to read the target data and decrypt the target data according to the key of the target data and the mapping relationship; The decrypted target data is recorded in the data history access log.

[0012] A second aspect of the present invention provides a data dynamic storage device, comprising: a memory and at least one processor, wherein instructions are stored in the memory, and the memory and the at least one processor are interconnected via lines; the at least one processor calls the instructions in the memory so that the data dynamic storage device executes the above-mentioned data dynamic storage method.

[0013] A third aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned data dynamic storage method.

[0014] In an embodiment of the present invention, a mapping relationship between data and a data storage layer is obtained; when accessing target data, according to the key of the target data and the mapping relationship, the target data is routed to the corresponding data storage layer to read the target data, and the target data is recorded in the data history access log; according to the data history access log, the historical access frequency, time dimension trend and periodic characteristics of the stored data are regularly analyzed; according to the historical access frequency, time dimension trend and periodic characteristics of the stored data, the future access popularity of the stored data is predicted through an integrated deep learning model; the future access popularity of the stored data is sorted, and according to the sorting of the future access popularity, the stored data is migrated to different data storage layers. In the present invention, the historical access frequency, time dimension trend and periodic characteristics of the stored data are regularly analyzed, the future access popularity of the stored data is predicted through an integrated deep learning model, and according to the sorting of the future access popularity, the stored data is migrated to different data storage layers, which can dynamically adjust the storage of data, reasonably allocate storage resources, and improve the utilization of storage resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic diagram of an embodiment of a method for dynamically storing data in an embodiment of the present invention; Figure 2 A schematic diagram of an embodiment of a data dynamic storage device in an embodiment of the present invention; Figure 3 It is a schematic diagram of an embodiment of a data dynamic storage device in an embodiment of the present invention. DETAILED DESCRIPTION

[0016] Embodiments of the present invention provide a data dynamic storage method, device, equipment and storage medium.

[0017] The embodiments disclosed in the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments disclosed in the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments disclosed in the present invention are only for exemplary purposes and are not intended to limit the scope of protection disclosed in the present invention.

[0018] In the description of the embodiments disclosed in the present invention, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0019] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , an embodiment of the data dynamic storage method in the embodiment of the present invention includes: S100, obtaining a mapping relationship between data and a data storage layer.

[0020] In this embodiment, the data storage system provides a unified data access and storage entry. When storing data, the data is stored in the system through the entry. When storing data, a unique key is generated for the data, and the data is stored in the corresponding data storage layer according to the service needs of the data. For example, data that needs to be stored quickly is stored in a high-performance storage medium layer, and data that needs to be stored slowly is stored in a low-performance storage medium layer. At the same time, a mapping relationship between the key and the IP information of the data storage layer is generated and stored.

[0021] S200, when accessing target data, route to the corresponding data storage layer to read the target data according to the key and mapping relationship of the target data, and record the target data in the data history access log.

[0022] In this embodiment, when the target data is accessed, the key corresponding to the target data will be returned to the requesting end. When accessing the data, based on the key corresponding to the target data, the IP information of the data storage layer corresponding to the key is obtained through the mapping relationship between the data and the data storage layer. Based on the IP information of the data storage layer, the target data is routed to the corresponding data storage layer to read the target data, record the target data, and generate a data history access log.

[0023] S300, based on the data historical access log, regularly analyze and obtain the historical access frequency, time dimension trend and periodic characteristics of the stored data.

[0024] In this embodiment, according to the data historical access log, the historical access frequency, time dimension trend and periodic characteristics of the stored data are analyzed regularly (configurable interval time), wherein the historical access frequency, that is, the number of times or frequency that the data is accessed, can help understand the popularity or importance of the data, and by counting the number of accesses in different time periods, the activity and usage of the data can be evaluated. The time dimension trend can understand the usage trend of the data by analyzing the changes in the number of accesses to the data in different time periods. For example, it can be observed whether the number of data accesses increases or decreases over time, and the reasons for the changes, which is very helpful for predicting the usage of future data. The periodic characteristics identify the periodic usage pattern of the data, such as daily cycle, weekly cycle or annual cycle, for example, some data may be used more frequently in a specific time period, such as weekdays and weekends, the end of the quarter and the beginning of the year, etc. By regularly analyzing the historical access frequency, time dimension trend and periodic characteristics of the stored data, the usage of the data can be better understood, and the storage strategy can be adjusted according to the access frequency and trend of the data, which can help predict future data needs.

[0025] S400 predicts the future access popularity of the stored data through an integrated deep learning model based on the historical access frequency, time dimension trend and periodic characteristics of the stored data.

[0026] In this embodiment, first, it is necessary to clarify the prediction target, that is, to predict the future access heat of the data, select features related to the future access heat of the data, including the historical access frequency, time dimension trend and periodic features of the stored data, and use these features as input data for training the model. Then, multiple deep learning models are constructed based on the selected features and data. These models may include different types of neural network structures, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs) or long short-term memory networks (LSTMs), etc. The specific selection depends on the nature of the data and the complexity of the prediction task. Secondly, each model is trained, which includes using historical data as a training set, adjusting the weights and parameters of the model through back propagation algorithms and gradient descent optimization techniques to minimize the prediction error. During the training process, it is necessary to define a loss function to measure the prediction error of the model, and use a suitable optimizer to update the weights and biases of the model. Then, the prediction results of multiple basic models are integrated. By combining the prediction results of multiple models, the variance of a single model can be reduced and the overall prediction performance and stability can be improved. Finally, the integrated model is used to predict the future access heat of the data. The integrated deep learning model can effectively utilize the prediction results of multiple models to improve the accuracy and stability of the prediction, thereby better predicting the access popularity of future data.

[0027] Optionally, the ensemble deep learning model method may include methods such as bagging, boosting, and stacking. These methods improve the overall prediction performance by combining the prediction results of multiple models, and have high accuracy and stability. Specifically: ‌Bagging‌ constructs multiple basic models and averages their prediction results to reduce variance by self-sampling the training set. This method improves the stability of prediction by reducing the correlation between models. ‌Boosting‌ iteratively trains multiple weak classifiers, adjusts the sample weights according to the performance of the previous classifier, and finally integrates the prediction results of all classifiers. This method can gradually improve the model and focus on processing samples that were misclassified by the previous round of classifiers. ‌Stacking‌ uses the output results of multiple basic models as new features, and then trains a meta-model to integrate the prediction results of these basic models. This method uses the prediction results of multiple models to train a more advanced model to improve the accuracy of prediction.

[0028] S500, sorting the future access heat of the stored data, and migrating the stored data to different data storage layers according to the sorting of the future access heat.

[0029] In this embodiment, the storage tiers may include high-speed storage media and low-speed but large-capacity storage media, each of which has its own specific performance and cost characteristics and is suitable for storing data with different access frequencies. By monitoring and analyzing the historical data access logs, predicting the future access heat of the stored data, sorting the future access heat of the stored data, and automatically adjusting the location of the data in different storage tiers (such as SSD, HDD, cold storage / cloud storage), high-heat data (such as frequently accessed files or database records) are stored on high-performance storage media to ensure fast access speed, while low-heat data can be stored on low-performance but large-capacity storage media, and an automatic adjustment mechanism can be implemented to dynamically adjust its position in the storage tier according to the actual access mode of the data. For example, if a data that was originally considered to be low-heat becomes high-heat data, the system should be able to automatically migrate it from the low-speed storage medium to the high-speed storage medium to ensure that the cost is optimized while meeting the performance requirements. The system provides an intuitive Web UI and API interface to achieve unified monitoring, configuration and troubleshooting across cloud and local storage resources.

[0030] In an optional implementation of the first aspect of the present invention, sorting the future access heat of the stored data, and migrating the stored data to different data storage layers according to the sorting of the future access heat includes: Sort the future access heat of the stored data; obtain the previous round sorting of the future access heat of the stored data, and compare the sorting of the future access heat with the previous round sorting of the access heat to obtain the number of differences in the access heat sorting of the stored data; migrate the stored data to different data storage layers according to the number of differences in the access heat sorting.

[0031] In this embodiment, the predicted future access heat of the storage data is sorted, and the previous round of sorting of the future access heat of the storage data is obtained, and the current round of sorting of the future access heat of the storage data is compared with the previous round of sorting of the future access heat of the storage data, and the number of differences in the access heat sorting of the storage data is obtained by statistics, and according to the number of differences in the access heat sorting of the storage data, it is determined whether to migrate the storage data to different data storage layers.

[0032] In an optional implementation of the first aspect of the present invention, migrating the stored data to different data storage layers according to the number of differences in access popularity rankings includes: Determine whether the number of differences in access heat ranking exceeds a preset threshold; if so, generate a data storage level adjustment plan based on the access heat ranking; and migrate the stored data to different data storage layers based on the data storage level adjustment plan.

[0033] In this embodiment, the consistency of the current round sorting of the future access heat of the stored data is compared with the previous round sorting of the future access heat, that is, whether the future access heat sorting is consistent is determined. If not, the number of differences in the access heat sorting is obtained, and according to the number of differences in the access heat sorting, it is determined whether to migrate the stored data to different data storage layers. Specifically, it is determined whether the number of differences in the access heat sorting exceeds a preset number threshold, wherein the preset number threshold may be half of the number of data sorted by access heat. If the number of differences in the access heat sorting exceeds the preset number threshold, a data storage level adjustment plan is generated. For example, the number of data sorted by this round of access heat is 10, and the number of differences in the access heat sorting is 6, which exceeds half of the number of data sorted by access heat. Then, a corresponding data storage level adjustment plan is generated, and data with high heat is preferentially migrated to a data storage level with fast reading and writing speeds, and data with low heat is migrated to a data storage level with slow reading and writing speeds.

[0034] In an optional implementation of the first aspect of the present invention, according to the data storage level adjustment scheme, migrating the stored data to different data storage layers includes: Calculate the remaining storage capacity of each data storage layer; determine whether the remaining storage capacity is lower than the preset first capacity threshold and higher than the preset second capacity threshold; if so, migrate the stored data to different data storage layers according to the data storage level adjustment plan combined with TLS / SSL encryption.

[0035] In this embodiment, the remaining storage capacity of each data storage layer is calculated, wherein the calculation of the remaining storage capacity is usually based on the number of storage layer units and the storage word length, and can be performed by bit calculation or byte calculation. Then, it is determined whether to compress or expand the storage capacity of each data storage layer according to the remaining storage capacity. Specifically, it is determined whether the remaining storage capacity is lower than the preset first capacity threshold and higher than the preset second capacity threshold, wherein the preset first capacity threshold refers to the judgment threshold for whether to perform capacity compression, and the preset second capacity threshold refers to the judgment threshold for whether to perform capacity expansion. If the remaining storage capacity is lower than the preset first capacity threshold and higher than the preset second capacity threshold, it means that the data storage layer does not need to be compressed or expanded, and the storage data can be migrated to different data storage layers according to the data storage level adjustment plan generated by the access popularity sorting, combined with TLS / SSL encryption.

[0036] In an optional implementation of the first aspect of the present invention, according to the data storage level adjustment scheme, combined with TLS / SSL encryption, before migrating the stored data to different data storage layers, the method further includes: If the remaining storage capacity is higher than the preset first capacity threshold, the data compression algorithm is used to reduce the storage nodes of the data storage layer to achieve dynamic compression; if the remaining storage capacity is lower than the preset second capacity threshold, k8s is used to increase the storage nodes of the data storage layer to achieve dynamic expansion.

[0037] In this embodiment, the storage capacity of each data storage layer is compressed or expanded by judging whether the remaining storage capacity of each data storage layer is lower than the preset first capacity threshold and higher than the preset second capacity threshold. If the remaining storage capacity is higher than the preset first capacity threshold, the capacity is compressed by the data compression algorithm. If the remaining storage capacity is lower than the preset second capacity threshold, the capacity is expanded by k8s to ensure that the data storage layer has sufficient remaining capacity to meet current or future data storage needs, improve storage efficiency, optimize resource utilization, and enhance the adaptability and flexibility of the system to ensure long-term storage and fast access to data. Specifically, the storage nodes of the data storage layer are reduced by the data compression algorithm LZ77 or LZ78, wherein LZ77 or LZ78 are both dictionary-based algorithms, which reduce storage occupancy and improve storage efficiency by finding repeated sequences in the data and replacing them with references (position and length) pointing to the repeated sequences; combined with containerization (k8s, docker, etc.) and microservice architecture, the on-demand expansion and contraction of storage nodes are realized to improve resource utilization.

[0038] In an optional implementation of the first aspect of the present invention, according to the data storage level adjustment scheme, combined with TLS / SSL encryption, migrating the stored data to different data storage layers includes: Get the session request sent by the client, and send the server's digital certificate and public key to the client; get the master key sent by the client, which is extracted from the digital certificate and encrypted with the public key; use the private key to decrypt the master key, and generate a conversation key based on the decrypted master key and random information; according to the data storage level adjustment plan, migrate the stored data to different data storage layers through the conversation key.

[0039] In this embodiment, during the migration of data between different data storage levels, encryption (TLS / SSL) and dynamic key management are performed in transit. Specifically, the client sends a session request to the server to establish a connection with the server. After receiving the session request, the server sends its digital certificate to the client to prove its identity. The client verifies the legitimacy of the digital certificate. The client extracts the master key from the digital certificate and encrypts it with the server's public key, and then sends it to the server. The server uses its own private key to decrypt the master key, and then uses the master key + random information to generate a session key. The session key is destroyed after the session ends. When the stored data is migrated, the data is encrypted using the session key. When the stored data is migrated to the corresponding data storage layer, the same session key is used to decrypt the data to ensure the security of data transmission.

[0040] In an optional implementation of the first aspect of the present invention, when accessing target data, routing to a corresponding data storage layer to read the target data according to the key and mapping relationship of the target data, and recording the target data in a data history access log includes: When storing the target data, the target data is encrypted according to the type of the target data; when accessing the target data, according to the key and mapping relationship of the target data, it is routed to the corresponding data storage layer to read the target data and decrypt the target data; the decrypted target data is recorded in the data history access log.

[0041] In this embodiment, when the target data is stored, it is determined whether to encrypt the target data based on the type of the target data. Specifically, if the target data is data with common attributes, sensitive data (such as identity card, password, etc.) and important data, then during the process of storing the data at the unified entrance, the entrance application service will perform static data encryption (SM4, etc.), and decryption will be performed during reading, and the decrypted target data will be recorded in the data history access log.

[0042] See also Figure 2 According to a second aspect of the present invention, there is provided a data dynamic storage device, the data dynamic storage device comprising: A mapping relationship acquisition module 10, used to acquire a mapping relationship between data and a data storage layer; The target data reading module 20 is used to route to the corresponding data storage layer to read the target data according to the key and mapping relationship of the target data when accessing the target data, and record the target data in the data history access log; The data analysis module 30 is used to regularly analyze the historical access frequency, time dimension trend and periodic characteristics of the stored data based on the historical data access log; The access popularity prediction module 40 is used to predict the future access popularity of the stored data through an integrated deep learning model based on the historical access frequency, time dimension trend and periodic characteristics of the stored data; The storage data migration module 50 is used to sort the future access heat of the storage data and migrate the storage data to different data storage layers according to the sorting of the future access heat.

[0043] In an optional implementation of the second aspect of the present invention, the storage data migration module 50 is also used to sort the future access heat of the storage data; obtain the previous round of sorting of the future access heat of the storage data, and compare the sorting of the future access heat with the previous round of sorting of the access heat to obtain the number of differences in the access heat sorting of the storage data; and migrate the storage data to different data storage layers according to the number of differences in the access heat sorting.

[0044] In an optional implementation of the second aspect of the present invention, the storage data migration module 50 is also used to determine whether the number of differences in access popularity ranking exceeds a preset threshold; if exceeded, a data storage level adjustment plan is generated based on the access popularity ranking; and based on the data storage level adjustment plan, the storage data is migrated to different data storage layers.

[0045] In an optional implementation of the second aspect of the present invention, the storage data migration module 50 is also used to calculate the remaining storage capacity of each data storage layer; determine whether the remaining storage capacity is lower than a preset first capacity threshold and higher than a preset second capacity threshold, the preset first capacity threshold refers to the judgment threshold for whether to perform capacity compression, and the preset second capacity threshold refers to the judgment threshold for whether to perform capacity expansion; if so, migrate the storage data to different data storage layers according to the data storage level adjustment plan combined with TLS / SSL encryption.

[0046] In an optional implementation of the second aspect of the present invention, the data dynamic storage device further includes: The capacity processing module is used to reduce the storage nodes of the data storage layer to achieve dynamic compression through the data compression algorithm if the remaining storage capacity is higher than the preset first capacity threshold; if the remaining storage capacity is lower than the preset second capacity threshold, the storage nodes of the data storage layer are increased through k8s to achieve dynamic expansion.

[0047] In an optional implementation of the second aspect of the present invention, the storage data migration module 50 is also used to obtain a session request sent by the client, and send the server's digital certificate and public key to the client; obtain a master key sent by the client, which is extracted by the client from the digital certificate and encrypted with the public key; decrypt the master key using the private key, and generate a conversation key based on the decrypted master key and random information; and migrate the storage data to different data storage layers through the conversation key according to the data storage level adjustment plan.

[0048] In an optional implementation of the second aspect of the present invention, the target data reading module 20 is also used to encrypt the target data according to the type of the target data when storing the target data; when accessing the target data, route to the corresponding data storage layer to read the target data according to the key and mapping relationship of the target data, and decrypt the target data; and record the decrypted target data in the data history access log.

[0049] Figure 3 1 is a schematic diagram of the structure of a data dynamic storage device provided by an embodiment of the present invention. The data dynamic storage device 500 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 510 (for example, one or more processors) and a memory 520, and one or more storage media 530 (for example, one or more mass storage devices) storing application programs 533 or data 532. Among them, the memory 520 and the storage medium 530 may be short-term storage or permanent storage. The program stored in the storage medium 530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the data dynamic storage device 500. Furthermore, the processor 510 may be configured to communicate with the storage medium 530 and execute a series of instruction operations in the storage medium 530 on the data dynamic storage device 500.

[0050] The data dynamic storage device 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input and output interfaces 560, and / or one or more operating systems 531, such as Windows Serve, Mac OS X, Unix, Linux, Free BSD, etc. It will be appreciated by those skilled in the art that Figure 3 The data dynamic storage device structure shown does not constitute a limitation on the data dynamic storage device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0051] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the data dynamic storage method.

[0052] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0053] In addition, although each operation is described in a specific order, this should be understood as requiring such operation to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain desired results. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination mode.

[0054] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A method for dynamic data storage, characterized in that: The dynamic data storage method includes: Get the mapping relationship between data and data storage layer; When accessing target data, according to the key of the target data and the mapping relationship, routing to the corresponding data storage layer to read the target data, and recording the target data in the data history access log; Based on the data historical access log, regularly analyze the historical access frequency, time dimension trend and periodic characteristics of the stored data; Based on the historical access frequency, time dimension trend and periodic characteristics of the stored data, an integrated deep learning model is used to predict the future access popularity of the stored data; The future access popularity of the stored data is sorted, and the stored data is migrated to different data storage layers according to the sorting of the future access popularity.

2. The data dynamic storage method according to claim 1, characterized in that: Sorting the future access popularity of the stored data and migrating the stored data to different data storage layers according to the sorting of the future access popularity includes: sorting the future access popularity of the stored data; Obtaining the last round ranking of the future access popularity of the stored data, and comparing the ranking of the future access popularity with the last round ranking of the access popularity to obtain the number of differences in the access popularity ranking of the stored data; The stored data is migrated to different data storage layers according to the number of differences in the access popularity rankings.

3. The data dynamic storage method according to claim 2, characterized in that: Migrating the stored data to different data storage layers according to the number of differences in the access popularity ranking includes: Determine whether the number of differences in the access popularity ranking exceeds a preset threshold; If it exceeds, a data storage level adjustment plan is generated according to the access popularity ranking; According to the data storage level adjustment plan, the stored data is migrated to different data storage layers.

4. The data dynamic storage method according to claim 3, characterized in that: Migrating the stored data to different data storage layers according to the data storage level adjustment scheme includes: Calculate the remaining storage capacity of each data storage layer; determining whether the remaining storage capacity is lower than a preset first capacity threshold and higher than a preset second capacity threshold, wherein the preset first capacity threshold is a threshold for determining whether to perform capacity compression, and the preset second capacity threshold is a threshold for determining whether to perform capacity expansion; If so, the stored data is migrated to a different data storage layer according to the data storage level adjustment scheme in combination with TLS / SSL encryption.

5. The data dynamic storage method according to claim 4, characterized in that: Before migrating the stored data to different data storage layers according to the data storage layer adjustment scheme and in combination with TLS / SSL encryption, the method further includes: If the remaining storage capacity is higher than the preset first capacity threshold, reducing the storage nodes of the data storage layer to achieve dynamic compression through a data compression algorithm; If the remaining storage capacity is lower than the preset second capacity threshold, dynamic capacity expansion is achieved by adding storage nodes of the data storage layer through k8s.

6. The data dynamic storage method according to claim 4, characterized in that: Migrating the stored data to different data storage layers according to the data storage layer adjustment scheme in combination with TLS / SSL encryption includes: Obtaining a session request from a client, and sending the server's digital certificate and public key to the client; Obtaining a master key sent by the client, where the master key is extracted by the client from the digital certificate and encrypted with the public key; Decrypting the master key using the private key, and generating a session key based on the decrypted master key and random information; According to the data storage level adjustment scheme, the stored data is migrated to different data storage layers using the session key.

7. The data dynamic storage method according to claim 1, characterized in that: When accessing target data, routing to the corresponding data storage layer to read the target data according to the key of the target data and the mapping relationship, and recording the target data in the data history access log includes: When storing target data, encrypting the target data according to the type of the target data; When accessing the target data, routing to the corresponding data storage layer to read the target data according to the key of the target data and the mapping relationship, and decrypting the target data; The decrypted target data is recorded in the data history access log.

8. A data dynamic storage device, characterized in that: The data dynamic storage device comprises: A mapping relationship acquisition module, used to obtain the mapping relationship between data and the data storage layer; A target data reading module is used to route to the corresponding data storage layer to read the target data according to the key of the target data and the mapping relationship when accessing the target data, and record the target data in the data history access log; A data analysis module is used to regularly analyze the historical access frequency, time dimension trend and periodic characteristics of the stored data based on the data historical access log; An access popularity prediction module is used to predict the future access popularity of stored data based on the historical access frequency, time dimension trend and periodic characteristics of the stored data through an integrated deep learning model; The storage data migration module is used to sort the future access popularity of the storage data and migrate the storage data to different data storage layers according to the sorting of the future access popularity.

9. A data dynamic storage device, characterized in that: The data dynamic storage device includes: a memory and at least one processor, wherein instructions are stored in the memory, and the memory and the at least one processor are interconnected via a line; The at least one processor calls the instructions in the memory to enable the data dynamic storage device to execute the data dynamic storage method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data dynamic storage method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Method and device for data storage

    CN108932114A

  • Data storage method and device, electronic equipment and computer readable medium

    CN118349178A

  • Memory data management method and system and related equipment

    CN118377624A

  • Data migration schedule prediction using machine learning

    US20230051103A1

Cited By

  • Mass data collection and storage method and system based on distributed architecture

    CN119166077A

  • Cloud storage data access method and device

    CN119484557A

  • Regional education information resource sharing method and system

    CN119621691A

  • Optical fiber data storage management system and method based on big data

    CN120085812A

  • A Fiber Optic Data Storage Management System and Method Based on Big Data

    CN120085812B