A method for automatic management of cluster space based on data hot and cold separation

Abstract

The application provides a kind of based on data hot and cold separation's cluster space automation management method, belong to data processing technical field, specifically include: based on the data type determination core data of the data of cluster storage space, based on the call number of data, the access quantity of the user of data determines heat value, and determines heat data based on heat value;With core data or with heat data having blood relationship data as associated data, and based on associated data, heat data, core data constructs hot spot data, and other data except hot spot data in cluster storage space as cold data, and when the data volume of cold data is greater than the set data volume or the data volume of hot spot data is greater than the set hot spot data volume, heat evaluation value is determined based on heat value, generation time, recent call time, and the storage mode of hot spot data and cold data is determined based on heat evaluation value respectively, to further improve the utilization of storage space.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, and in particular relates to a method for automated management of cluster space based on data hot and cold separation. Background Technology

[0002] To achieve automated management of cluster storage space, the invention patent CN104468654A, "Storage Management Gateway, Method, and Data Tiered Backup System Connected to Cloud Storage," obtains the hot / cold data status in a local virtual storage pool and integrates all different types of local storage resources into a single logical storage resource through the virtual storage pool. Finally, it backs up data with a cold status in the virtual storage pool to the cloud server, thereby reducing local storage space and lowering costs. However, it suffers from the following technical problems:

[0003] 1. Ignoring the lineage relationships between different data: In the process of data processing and storage decisions, some less popular data may have a lineage relationship with more popular data, such as being cited or referenced. If the lineage relationship is not considered and the less popular data is placed directly on the cloud server or offline storage, the data processing efficiency may be slowed down, and the system may not be able to meet the needs of normal operation.

[0004] 2. The importance of data was not assessed by combining popularity and data type. For some data, although its popularity is low, it is still essential data for the system to operate. For example, when approving consumer loans for customers, the customer's basic information is important data for the system to approve. Although its popularity is low, if it is placed directly on the cloud server or offline storage, it may slow down the data processing efficiency and fail to meet the needs of normal system operation.

[0005] To address the aforementioned technical problems, this invention provides a method for automated cluster space management based on data hot and cold separation. Summary of the Invention

[0006] To achieve the objectives of this invention, the following technical solution is adopted:

[0007] According to one aspect of the present invention, a method for automated cluster space management based on data hot and cold separation is provided.

[0008] A method for automated cluster space management based on hot and cold data separation, characterized in that it specifically includes:

[0009] S11 determines the basic importance of the data based on the data type of the cluster storage space, and determines whether the basic importance of the data is greater than the importance setting value. If so, the data is regarded as core data and proceeds to step S13; otherwise, proceeds to step S12.

[0010] S12 determines the number of times the data is called and the number of times the data is accessed by users based on the query history of the data, and determines the popularity value of the data based on the number of times the data is called and the number of times the data is accessed, and data with a popularity value greater than the popularity set amount is regarded as popular data.

[0011] S13 uses data that is related to the core data or the popularity data as associated data, and constructs hot data based on the associated data, popularity data, and core data. It also uses other data in the cluster storage space besides the hot data as cold data, and determines whether the amount of cold data is greater than the set amount of data or the amount of hot data is greater than the set amount of hot data. If yes, proceed to step S14; otherwise, set different storage methods for the hot data and cold data.

[0012] S14 determines the popularity evaluation value of the hot data and the cold data based on the popularity value, generation time, and most recent call time, and determines the storage method of the hot data and the cold data based on the popularity evaluation value respectively.

[0013] By setting importance values, the system first filters data based on its data type, ensuring that data of high importance to business or system operations can be stored more efficiently. This also improves the efficiency of hot and cold data analysis in the integrated storage space.

[0014] By determining the popularity value of data based on the number of calls and accesses, and then filtering the popular data based on the popularity value, the system achieves the filtering of highly popular data from two aspects: the number of calls and the number of users accessing the data. This ensures the accuracy and comprehensiveness of the filtering. At the same time, the filtering of popular data lays the foundation for adopting different storage methods for popular and unpopular data.

[0015] By combining the analysis of blood relations to identify key data, and by identifying hot data based on related data, popularity data, and core data, the technical problem of insufficient data processing efficiency caused by only considering popularity data is avoided. This further improves the system's processing efficiency and ensures the scientific nature of storage and processing.

[0016] By determining the popularity evaluation values ​​of hot and cold data based on popularity value, generation time, and most recent call time, the storage method of hot and cold data can be determined from multiple perspectives, ensuring the accuracy and comprehensiveness of the final storage method determination.

[0017] On the other hand, this application provides a computer system comprising: a memory and a processor connected in communication, and a computer program stored in the memory and capable of running on the processor, characterized in that: when the processor runs the computer program, it executes the above-described method for automated cluster space management based on data hot and cold separation.

[0018] On the other hand, the present invention provides a computer storage medium storing a computer program, which, when executed in a computer, causes the computer to perform the above-described method for automated cluster space management based on data hot and cold separation.

[0019] Other features and advantages will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained through the structures particularly pointed out in the description and the drawings.

[0020] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0021] The above and other features and advantages of the present invention will become more apparent from a detailed description of exemplary embodiments thereof with reference to the accompanying drawings.

[0022] Figure 1 This is a flowchart of a method for automated cluster space management based on data hot and cold separation according to Embodiment 1;

[0023] Figure 2 This is a flowchart illustrating the specific steps involved in constructing the data based on the heat values ​​from Example 1.

[0024] Figure 3 This is a flowchart outlining the specific steps involved in evaluating the popularity rating of hotspot data based on Example 1.

[0025] Figure 4 This is a structural diagram of a computer storage medium according to Embodiment 3. Detailed Implementation

[0026] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that the invention will be thorough and complete, and the concept of the exemplary embodiments will be fully conveyed to those skilled in the art. The same reference numerals in the drawings denote the same or similar structures, and therefore their detailed description will be omitted.

[0027] The terms “a,” “one,” “the,” and “the” are used to indicate the existence of one or more elements / components / etc.; the terms “including” and “having” are used to indicate an open-ended meaning of inclusion and that other elements / components / etc. may exist in addition to the listed elements / components / etc.

[0028] The applicant discovered that existing technologies for automating cluster storage space management neglect the lineage relationships between different data. If these lineage relationships are not considered and data with low popularity is directly placed on cloud servers or offline storage, it may lead to slower data processing efficiency, failing to meet the needs of normal system operation. At the same time, failing to consider assessing the importance of data based on popularity and data type may also lead to slower data processing efficiency, failing to meet the needs of normal system operation.

[0029] Example 1

[0030] To solve the above problems, according to one aspect of the present invention, such as Figure 1 As shown, according to one aspect of the present invention, a method for automated cluster space management based on data hot and cold separation is provided.

[0031] A method for automated cluster space management based on hot and cold data separation, characterized in that it specifically includes:

[0032] S11 determines the basic importance of the data based on the data type of the cluster storage space, and determines whether the basic importance of the data is greater than the importance setting value. If so, the data is regarded as core data and proceeds to step S13; otherwise, proceeds to step S12.

[0033] Specifically, the basic importance of the data is determined based on the data type, and its value ranges from 0 to 1. The more important the data type, the greater the basic importance of the data.

[0034] For example, the data type is determined by the application context. When the data type is important customer data, basic system data, etc., its basic importance is relatively high. Therefore, the above data can be regarded as core data and there is no need to assess its importance.

[0035] For a specific example, the basic importance of the data is determined based on the data type and by methods such as expert scoring. Generally, the importance of the system is determined according to the data type, with a value ranging from 0 to 1.

[0036] By setting importance values, the system first filters data based on its data type, ensuring that data of high importance to business or system operations can be stored more efficiently. This also improves the efficiency of hot and cold data analysis in the integrated storage space.

[0037] S12 determines the number of times the data is called and the number of times the data is accessed by users based on the query history of the data, and determines the popularity value of the data based on the number of times the data is called and the number of times the data is accessed, and data with a popularity value greater than the popularity set amount is regarded as popular data.

[0038] Specifically, the steps for constructing the popularity value of the data are as follows:

[0039] S21 obtains the number of times the data is called within a set time period and uses it as the most recent call count of the data; obtains the number of times the data is accessed by users within a set time period and uses it as the most recent access count value of the data.

[0040] S22 determines whether the data has a popularity based on the number of recent calls or the number of recent accesses to the data. If yes, proceed to step S23; otherwise, set the popularity value of the data to 0.

[0041] For example, if the number of recent calls to the data exceeds 1000 or the number of recent accesses to the data is greater than 200, then proceed to step S23. Otherwise, it means that the data has low popularity, so the popularity value of the data is set to 0.

[0042] S23 determines whether the data is high-popularity data based on the number of times the data is called or the number of users accessing the data. If yes, the popularity value of the data is set to 1; otherwise, proceed to step S24.

[0043] Specifically, if the number of recent calls to data exceeds 10,000 or the number of recent accesses to data is greater than 2,000, the popularity value of the data is set to 1.

[0044] S24 constructs an input set based on the recent call count, call count, recent access count, and access count of the data, and constructs a weight vector to correct the input set to obtain a corrected input set. The popularity value of the data is obtained by using an evaluation model based on a machine learning algorithm.

[0045] For a specific example, the modified input set is Xc = {W1T} j W2T, W3D j ,W4D}, where Tj, T, D j D and W1 are the recent call count, the number of calls, the number of recent visits, and the number of visits, respectively. W1, W2, W3, and W4 are the weight vector values.

[0046] For example, the evaluation model based on the machine learning algorithm adopts an evaluation model based on the PSO-LSTM algorithm, and the specific steps for constructing the evaluation model are as follows:

[0047] Step 1 sets the number of inputs, the number of hidden neurons in the model, the number of outputs, and the maximum allowed difference, and determines the initial values ​​and termination conditions of other factors in PSO to initialize the model.

[0048] Step 2: The input vector of the model is the corrected input set in the dataset, and the output vector is the heat value r of the data. Then the model is trained.

[0049] Step 3: Calculate according to the formula output by the model:

[0050]

[0051] Calculate the popularity value of its data and the expected popularity value r of the data. y If the mean square error between R and r is less than the set maximum error, proceed with the next step; otherwise, compare the calculated R with r. y The mean square error between the two is used as the fitness value of the particle in the PSO algorithm, and the local optimum and global optimum of the particle are obtained by using the improved particle movement speed adjustment equation and the improved particle movement direction adjustment equation.

[0052] Step 4 via formula

[0053]

[0054]

[0055] The particle velocity adjustment equation and the improved particle direction adjustment equation in the PSO algorithm are updated to form a new particle population, so as to achieve the purpose of updating the weights in the LSTM neural network.

[0056] Step 5: Recalculate the mean square error between the heat value of the data output through the output layer of the PSO-LSTM model and the expected value; if the obtained value is smaller than the maximum allowable error, proceed to step 6; if it is larger than the maximum allowable error, proceed to step 4.

[0057] Step 6: If the mean square error is within the acceptable range, save the weights and thresholds between the input neurons and hidden neurons, as well as the weights and thresholds between the hidden neurons and the output neurons during training. The model training is now complete.

[0058] Step 7 modifies the input set as the input to the model and records the model's output, which is the heat value of the data to be evaluated.

[0059] Specifically, the weight vector is determined using the entropy method based on the data type and application scenario of the data in the cluster storage space, and the weight vector value of the most recent call count is greater than the weight vector value of the call count, and the weight vector value of the most recent access count is greater than the weight vector value of the access count.

[0060] By determining the popularity value of data based on the number of calls and accesses, and then filtering the popular data based on the popularity value, the system achieves the filtering of highly popular data from two aspects: the number of calls and the number of users accessing the data. This ensures the accuracy and comprehensiveness of the filtering. At the same time, the filtering of popular data lays the foundation for adopting different storage methods for popular and unpopular data.

[0061] S13 uses data that is related to the core data or the popularity data as associated data, and constructs hot data based on the associated data, popularity data, and core data. It also uses other data in the cluster storage space besides the hot data as cold data, and determines whether the amount of cold data is greater than the set amount of data or the amount of hot data is greater than the set amount of hot data. If yes, proceed to step S14; otherwise, set different storage methods for the hot data and cold data.

[0062] Specifically, based on the reference or being referenced relationship between the core data and the popularity data, data that is related to the core data or the popularity data is identified and treated as associated data.

[0063] Specifically, the provision of different storage methods for hot data and cold data includes:

[0064] When the data is frequently accessed, it is stored using online or near-line storage; when the data is rarely accessed, it is stored using offline storage.

[0065] By combining the analysis of blood relations to identify key data, and by identifying hot data based on related data, popularity data, and core data, the technical problem of insufficient data processing efficiency caused by only considering popularity data is avoided. This further improves the system's processing efficiency and ensures the scientific nature of storage and processing.

[0066] S14 determines the popularity evaluation value of the hot data and the cold data based on the popularity value, generation time, and most recent call time, and determines the storage method of the hot data and the cold data based on the popularity evaluation value respectively.

[0067] Specifically, the evaluation steps for the popularity rating of the hot data are as follows:

[0068] S31 obtains the generation time of the hot data and the most recent call time of the hot data, and obtains the production time difference and call time difference based on the difference between the generation time and the most recent call time and the current time;

[0069] S32 determines whether the hot data is recent hot data based on the production time difference and the call time difference of the hot data. If yes, the heat evaluation value of the hot data is set to 1. If no, proceed to step S33.

[0070] It should be noted that when both the production time difference and the call time difference of the hot data are less than a certain value, it is determined to be recent hot data.

[0071] S33 determines whether the hot data is the recently selected hot data based on the production time difference or the call time difference of the hot data. If yes, proceed to step S34; otherwise, proceed to step S35.

[0072] It should be noted that if either the production time difference or the retrieval time difference of the hot data is less than a certain value, it is determined to be the recent hot data to be filtered.

[0073] S34 determines whether the hot data is filtered data based on the heat value of the hot data. If yes, the heat evaluation value of the hot data is set to 1. If no, proceed to step S35.

[0074] It should be noted that when the popularity value of hot data is high, it is determined to be filtered data.

[0075] S35 constructs a heat evaluation input set based on the heat value, generation time, and most recent call time, and obtains the heat evaluation value of the thermoelectric data by using a heat evaluation model based on intelligent algorithms based on the heat evaluation input set.

[0076] Specifically, when the popularity evaluation value of the hot data is greater than the first evaluation value threshold, or when the popularity evaluation value of the hot data is not greater than the first evaluation value threshold and the required response time value of the hot data is greater than the third time threshold, the hot data is stored in real time. When the popularity evaluation value of the hot data is not greater than the first evaluation value threshold and the required response time value of the hot data is not greater than the third time threshold, the hot data is stored in near-line storage.

[0077] Specifically, when the popularity evaluation value of the unpopular data is greater than the second evaluation value threshold or the difference between the most recent call time and the current time of the unpopular data is not greater than the third time threshold, the unpopular data is stored offline. When the popularity evaluation value of the unpopular data is not greater than the second evaluation value threshold and the difference between the most recent call time and the current time of the unpopular data is greater than the third time threshold, the unpopular data is forgotten.

[0078] By determining the popularity evaluation values ​​of hot and cold data based on popularity value, generation time, and most recent call time, the storage method of hot and cold data can be determined from multiple perspectives, ensuring the accuracy and comprehensiveness of the final storage method determination.

[0079] Example 2

[0080] This application provides a computer system comprising: a memory and a processor connected in communication, and a computer program stored in the memory and capable of running on the processor, characterized in that: when the processor runs the computer program, it executes the above-described method for automated cluster space management based on data hot and cold separation.

[0081] Specifically, this embodiment also provides a computer system including a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements the aforementioned method for automated cluster space management based on data hot / cold separation.

[0082] Example 3

[0083] like Figure 4As shown, the present invention provides a computer storage medium storing a computer program, which, when executed in a computer, causes the computer to perform the aforementioned method for automated cluster space management based on data hot and cold separation.

[0084] Specifically, those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0085] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can also be implemented in other ways. The system embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0086] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0087] If the functionality is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0088] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A method for automated cluster space management based on data hot / cold separation, characterized in that, Specifically, it includes: S11 determines the basic importance of the data based on the data type of the cluster storage space, and determines whether the basic importance of the data is greater than the importance setting value. If so, the data is regarded as core data and proceeds to step S13; otherwise, proceeds to step S12. S12 determines the number of times the data is called and the number of times the data is accessed by users based on the query history of the data, and determines the popularity value of the data based on the number of times the data is called and the number of times the data is accessed, and data with a popularity value greater than the popularity set amount is regarded as popular data. S13 uses data that is related to the core data or the popularity data as associated data, and constructs hot data based on the associated data, popularity data, and core data. It also uses other data in the cluster storage space besides the hot data as cold data, and determines whether the amount of cold data is greater than the set amount of data or the amount of hot data is greater than the set amount of hot data. If yes, proceed to step S14; otherwise, set different storage methods for the hot data and cold data. S14 determines the popularity evaluation value of the hot data and the cold data based on the popularity value, generation time, and most recent call time, and determines the storage method of the hot data and the cold data based on the popularity evaluation value respectively; The specific steps for evaluating the popularity rating of the hot data are as follows: Obtain the generation time of the hot data and the most recent call time of the hot data, and obtain the production time difference and call time difference based on the difference between the generation time and the most recent call time and the current time; Based on the production time difference and the call time difference of the hot data, determine whether the hot data is recent hot data. If yes, set the popularity evaluation value of the hot data to 1. If no, proceed to the next step. Based on the production time difference or the call time difference of the hot data, determine whether the hot data is the recently selected hot data. If yes, proceed to the next step; otherwise, proceed to the last step. Based on the popularity value of the hot data, determine whether the hot data is selected data. If yes, set the popularity evaluation value of the hot data to 1. If no, proceed to the last step. The final step is to construct a popularity evaluation input set based on the popularity value, generation time, and most recent call time, and then use a popularity evaluation model based on intelligent algorithms to obtain the popularity evaluation value of the hot data based on the popularity evaluation input set. When the popularity evaluation value of the hot data is greater than the first evaluation value threshold, or when the popularity evaluation value of the hot data is not greater than the first evaluation value threshold and the required response time value of the hot data is greater than the third time threshold, the hot data is stored in real time; when the popularity evaluation value of the hot data is not greater than the first evaluation value threshold and the required response time value of the hot data is not greater than the third time threshold, the hot data is stored in nearline storage. When the popularity evaluation value of the unpopular data is greater than the second evaluation value threshold, or the difference between the most recent call time and the current time of the unpopular data is not greater than the third time threshold, the unpopular data is stored offline. When the popularity evaluation value of the unpopular data is not greater than the second evaluation value threshold and the difference between the most recent call time and the current time of the unpopular data is greater than the third time threshold, the unpopular data is forgotten.

2. The method for automated cluster space management as described in claim 1, characterized in that, The fundamental importance of the data is determined based on the data type, and its value ranges from 0 to 1. The more important the data type, the greater the fundamental importance of the data.

3. The method for automated cluster space management as described in claim 1, characterized in that, The specific steps for constructing the popularity value of the data are as follows: Get the number of times the data is called within a set time period and use it as the most recent call count of the data; get the number of times the data is accessed by users within a set time period and use it as the most recent access count of the data. The data's popularity is determined based on the number of recent calls or the number of recent accesses. If the data is popular, proceed to the next step; otherwise, set the popularity value of the data to 0. Based on the number of times the data is called or the number of times the data is accessed by users, it is determined whether the data is high-popularity data. If so, the popularity value of the data is set to 1; otherwise, proceed to the next step. An input set is constructed based on the recent call count, call count, recent access count, and access count of the data. A weight vector is then constructed to correct the input set to obtain a corrected input set. An evaluation model based on a machine learning algorithm is used to obtain the popularity value of the data.

4. The method for automated cluster space management as described in claim 3, characterized in that, The weight vector is determined using the entropy method based on the data type and application scenario of the data in the cluster storage space. The weight vector value of the most recent call count is greater than the weight vector value of the call count, and the weight vector value of the most recent access count is greater than the weight vector value of the access count.

5. The method for automated cluster space management as described in claim 1, characterized in that, Based on the reference or being referenced relationship between the core data and the popularity data, data that is related to the core data or the popularity data is identified and treated as associated data.

6. The method for automated cluster space management as described in claim 1, characterized in that, The specific methods for storing hot and cold data include: When the data is frequently accessed, it is stored using online or near-line storage; when the data is rarely accessed, it is stored using offline storage.

7. A computer system, comprising: A memory and processor connected by communication, and a computer program stored on the memory and capable of running on the processor, characterized in that: when the processor runs the computer program, it executes a method for automated cluster space management based on data cold and hot separation as described in any one of claims 1-6.

8. A computer storage medium storing a computer program thereon, which, when executed in a computer, causes the computer to perform a method for automated cluster space management based on data hot and cold separation as described in any one of claims 1-6.

Images