A method and apparatus for determining cold and hot data, and a medium

Abstract

The application discloses a cold and hot data determination method and device and medium, and is suitable for the technical field of data storage. The method obtains service request data in a service layer, adds a decay time label to the service request data according to a current decay time matrix, and optimizes the decay time through an iterative algorithm to determine the state switching frequency of the service request data. When a preset condition is met, the cold and hot data of the service request data is determined according to the average state switching frequency. When the preset condition is not met, the next iteration frequency is taken as a new current iteration frequency. Compared with the cold and hot data determined by a simple time threshold and an LRU algorithm, the application reasonably allocates storage resources according to the decay time of different services in the service layer, so that the determined cold and hot data reflects the personalization of requests of different business departments, and the processing efficiency of the system is improved.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, and in particular to a method, apparatus and medium for determining hot and cold data. Background Technology

[0002] In distributed storage systems, storage is often divided into a caching layer consisting of high-speed, expensive solid-state drives (SSDs) and random access memory (RAM), and a permanent storage layer consisting of low-speed, inexpensive hard disk drives (HDDs). The caching layer is insufficient to store all data, or the cost of storing all data in the caching layer is too high. Therefore, the goal is to store only frequently accessed data in the cache, reduce the occupancy of cold data in the cache through compression, or evict cold data to lower-speed storage devices.

[0003] Typically, all data is classified as hot or cold data using a simple time threshold method (e.g., selecting a fixed time period as the time threshold) or a simple Least Recently Used (LRU) method. Business-level characteristic data is not considered. Since the hot and cold data in the storage system are related to the business requests at the upper layer, and the business requests of different departments have obvious differences, the existing hot and cold data cannot reflect the differences and personalization, which reduces the processing efficiency of the subsequent storage system.

[0004] Therefore, how to improve the rational allocation of storage resources and enhance the processing efficiency of storage systems is a problem that those skilled in the art urgently need to solve. Summary of the Invention

[0005] The purpose of this invention is to provide a method, apparatus, and medium for determining hot and cold data, which is used to rationally allocate storage resources according to the decay time of different business needs at the business level, so that the determined hot and cold data reflects the personalization of different business department requests, thereby improving the system's processing efficiency.

[0006] To address the aforementioned technical problems, this invention provides a method for determining hot and cold data, applied to a server, comprising:

[0007] Obtain the current decay time matrix for the current iteration number;

[0008] Within a preset time period, acquire the client's business request data, and add decay time labels to the business request data according to the current decay time matrix to determine the number of state transitions of the business request data. The decay time labels are obtained by adjusting the decay time through the data status of the business request data, which includes active and inactive states.

[0009] Determine the average number of state transitions based on the number of state transitions;

[0010] When the average number of current iterations or state transitions meets the preset conditions, the hot and cold data of the business request data are determined based on the average number of state transitions.

[0011] If the average number of current iterations or state transitions does not meet the preset conditions, the next iteration number will be used as the new current iteration number, and the process will return to the step of obtaining the current decay time matrix of the current iteration number until the preset conditions are met.

[0012] Preferably, the process of determining the current decay time matrix includes:

[0013] The multiplication factor matrix for the current iteration number is determined based on the relationship between the average number of state transitions in the previous iteration and the number of state transitions, where the multiplication factor matrix for the first iteration number is the initial multiplication factor matrix;

[0014] Obtain the business request data from the client, where each business request data includes different data corresponding to the department making the business request;

[0015] Based on the classification algorithm, the probability distribution matrix is ​​obtained by classifying the data of each business request. Each row in the probability distribution matrix is ​​the same type of data, and each column represents the different business request departments.

[0016] The corresponding weight coefficients are obtained by applying a probability function to the probability distribution matrix to obtain the weights.

[0017] The current decay time matrix is ​​obtained by multiplying the weight coefficients with the multiplication coefficient matrix. The decay time matrix for the first iteration is obtained by multiplying the initial multiplication coefficient matrix with the weight coefficients.

[0018] Preferably, the process of determining the multiplication coefficient matrix for the current iteration number specifically includes:

[0019] Obtain the multiplication factor matrix of the previous iteration number;

[0020] When the number of state transitions is greater than the average number of state transitions, the first coefficient is added to the multiplication factor matrix of the previous iteration number to obtain the multiplication factor matrix of the current iteration number.

[0021] When the number of state transitions equals the average number of state transitions, the multiplier matrix of the previous iteration number is the multiplier matrix of the current iteration number.

[0022] When the number of state transitions is less than the average number of state transitions, the multiplier matrix of the previous iteration number is multiplied by the second coefficient to obtain the multiplier matrix of the current iteration number.

[0023] Preferably, the probability distribution matrix is ​​weighted using a probability function to obtain the corresponding weight coefficients, including:

[0024] The corresponding weight coefficients are obtained by weighting the probability distribution matrix using the normal distribution probability function.

[0025] Preferably, the client's business request data is acquired within a preset time period, and a decay time label is added to the business request data according to the current decay time matrix to determine the number of state transitions of the business request data, including:

[0026] Obtain the previous business request data and the current business request data, and determine the corresponding first business type and second business type, as well as the acquisition time interval, for the previous business request data and the current business request data respectively;

[0027] Within the current decay time matrix, find the corresponding first decay time and second decay time according to the first business type and the second business type respectively, and mark the business request data as active.

[0028] When the time interval is less than the first decay time, the second decay time corresponding to the business request data is adjusted to determine the current final decay time;

[0029] When the time interval is greater than or equal to the first decay time, the second decay time is kept unchanged as the current final decay time, and the business request data is marked as inactive.

[0030] Record the changes in the status of the data marked by the business request data to the status switch count, and count the number of times the status of the business request data switches within a preset time.

[0031] Preferably, when the time interval is less than the first decay time, the second decay time corresponding to the service request data is adjusted, including:

[0032] Determine whether the first decay time is greater than the second decay time;

[0033] If so, the second decay time will be adjusted to the first decay time as the current final decay time;

[0034] If not, the second decay time will remain unchanged as the current final decay time.

[0035] Preferably, determining the average number of state transitions based on the number of state transitions includes:

[0036] The number of state transitions is determined based on the different business types corresponding to the business request data.

[0037] The average number of state transitions is obtained by summing the number of transitions for each state and dividing by the number of business types.

[0038] Preferably, the preset condition is that the current iteration count reaches the total number of iterations or the number of state transitions is less than the average number of state transitions, and the number of business types corresponding to this threshold is less than the total number of business types. When the current iteration count or the average number of state transitions meets the preset condition, the hot and cold data of the business request data are determined based on the average number of state transitions, including:

[0039] Business request data with fewer state transitions than the average number of state transitions is identified as hot data.

[0040] Business request data whose state transition count is greater than or equal to the average state transition count is identified as cold data.

[0041] To address the aforementioned technical problems, the present invention also provides a device for determining hot and cold data, applied on a server side, comprising:

[0042] The acquisition module is used to obtain the current decay time matrix for the current iteration number;

[0043] The first determining module is used to acquire the client's business request data within a preset time, and add decay time labels to the business request data according to the current decay time matrix to determine the number of state switching of the business request data. The decay time labels are obtained by adjusting the decay time through the data status of the business request data, and the data status includes active status and inactive status.

[0044] The second determining module is used to determine the average number of state transitions based on the number of state transitions.

[0045] The third determination module is used to determine the hot and cold data of business request data based on the average number of state transitions when the current iteration number or the average number of state transitions meets the preset conditions.

[0046] The return module is used to take the next iteration number as the new current iteration number and return the steps to obtain the current decay time matrix of the current iteration number until the preset condition is met when the average number of current iterations or state transitions does not meet the preset condition.

[0047] To address the aforementioned technical problems, the present invention also provides an apparatus for determining hot and cold data, comprising:

[0048] Memory, used to store computer programs;

[0049] A processor is used to execute computer programs to implement the steps of the method for determining hot and cold data as described above.

[0050] To address the aforementioned technical problems, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method for determining hot and cold data as described above.

[0051] This invention provides a method for determining hot and cold data, applied to a server, comprising: obtaining the current decay time matrix of the current iteration number; obtaining client business request data within a preset time, and adding decay time labels to the business request data according to the current decay time matrix to determine the number of state transitions of the business request data, wherein the decay time labels are obtained by adjusting the decay time through the data state of the business request data, and the data state includes active state and inactive state; determining the average number of state transitions based on the number of state transitions; when the current iteration number or the average number of state transitions meets a preset condition, determining the hot and cold data of the business request data based on the average number of state transitions; when the current iteration number or the average number of state transitions does not meet the preset condition, taking the next iteration number as the new current iteration number, and returning to the step of obtaining the current decay time matrix of the current iteration number until the preset condition is met. This method acquires business request data at the business level, optimizes the decay time through an iterative algorithm to obtain the number of state transitions for the business request data, and uses the average number of state transitions as a threshold. Compared with existing methods that determine hot and cold data through simple time thresholds and LRU algorithms, this invention rationally allocates storage resources according to the decay time of different business needs at the business level, so that the determined hot and cold data reflects the personalization of requests from different business departments, thereby improving the system's processing efficiency.

[0052] In addition, the present invention also provides a device and medium for determining hot and cold data, which have the same beneficial effects as the method for determining hot and cold data described above. Attached Figure Description

[0053] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 A flowchart illustrating a method for determining hot and cold data provided in an embodiment of the present invention;

[0055] Figure 2 A flowchart illustrating another method for determining hot and cold data provided in an embodiment of the present invention;

[0056] Figure 3 A structural diagram of a device for determining hot and cold data provided in an embodiment of the present invention;

[0057] Figure 4 A structural diagram of another device for determining hot and cold data provided in an embodiment of the present invention. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0059] The core of this invention is to provide a method, apparatus, and medium for determining hot and cold data, which is used to rationally allocate storage resources according to the decay time of different business needs at the business level, so that the determined hot and cold data reflects the personalization of requests from different business departments, thereby improving the processing efficiency of the system.

[0060] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0061] It should be noted that the method for determining hot and cold data provided by this invention is mainly applied to the problem of efficient utilization of storage media with different efficiencies in distributed storage. By identifying hot and cold data in high-efficiency storage media and relocating cold data to inexpensive storage devices, the cluster can utilize storage resources more efficiently, thereby improving cluster storage efficiency.

[0062] In a storage system, cold data generally refers to data that has not been accessed for a long time, while hot data refers to data that is frequently accessed. This characteristic fully demonstrates that cold and hot data are related to upper-level business requests, and the business requests of different departments have obvious differences. Based on these differences, personalized cold and hot data can be defined.

[0063] Figure 1 A flowchart illustrating a method for determining hot and cold data provided in an embodiment of the present invention, applied to a server, such as... Figure 1 As shown, it includes:

[0064] S11: Get the current decay time matrix for the current iteration number;

[0065] S12: Obtain the client's business request data within a preset time, and add a decay time label to the business request data according to the current decay time matrix to determine the number of times the business request data's state switches.

[0066] The decay time label is obtained by adjusting the decay time based on the data status of the business request data. The data status includes active and inactive states.

[0067] S13: Determine the average number of state transitions based on the number of state transitions;

[0068] S14: Determine whether the current iteration count or the average number of state transitions meets the preset conditions. If it does, proceed to step S15; otherwise, proceed to step S16.

[0069] S15: Determine the hot and cold data of business request data based on the average number of state transitions;

[0070] S16: Take the next iteration number as the new current iteration number and return to the step of obtaining the current decay time matrix of the current iteration number until the preset condition is met.

[0071] Specifically, the current decay time matrix for the current iteration number is obtained. Since the decay time matrix differs for different iteration numbers during the iteration process, the current decay time matrix corresponding to the current iteration number is obtained. As a preferred embodiment, the process of determining the current decay time matrix includes:

[0072] The multiplication factor matrix for the current iteration number is determined based on the relationship between the average number of state transitions in the previous iteration and the number of state transitions, where the multiplication factor matrix for the first iteration number is the initial multiplication factor matrix;

[0073] Obtain the business request data from the client, where each business request data includes different data corresponding to the department making the business request;

[0074] Based on the classification algorithm, the probability distribution matrix is ​​obtained by classifying the data of each business request. Each row in the probability distribution matrix is ​​the same type of data, and each column represents the different business request departments.

[0075] The corresponding weight coefficients are obtained by applying a probability function to the probability distribution matrix to obtain the weights.

[0076] The current decay time matrix is ​​obtained by multiplying the weight coefficients with the multiplication coefficient matrix. The decay time matrix for the first iteration is obtained by multiplying the initial multiplication coefficient matrix with the weight coefficients.

[0077] Since the decay time matrix differs in each iteration and is related to the parameters corresponding to the previous iteration number, the relationship between the average number of state transitions and the number of state transitions in the previous iteration is used to determine the multiplication factor matrix for the current iteration number. It can be understood that the multiplication factor matrix at the first iteration is the initial multiplication factor matrix, which can be a general matrix or a diagonal matrix; no limitation is made here. The multiplication factor matrix for the current iteration number is updated based on the iteration number. The updated multiplication factor matrix is ​​related to the average number of state transitions and the number of state transitions. Accordingly, the process of determining the multiplication factor matrix for the current iteration number specifically includes:

[0078] Obtain the multiplication factor matrix of the previous iteration number;

[0079] When the number of state transitions is greater than the average number of state transitions, the first coefficient is added to the multiplication factor matrix of the previous iteration number to obtain the multiplication factor matrix of the current iteration number.

[0080] When the number of state transitions equals the average number of state transitions, the multiplier matrix of the previous iteration number is the multiplier matrix of the current iteration number.

[0081] When the number of state transitions is less than the average number of state transitions, the multiplier matrix of the previous iteration number is multiplied by the second coefficient to obtain the multiplier matrix of the current iteration number.

[0082] Specifically, its updated multiplication coefficient matrix K is obtained through the following relationship:

[0083]

[0084] Where, k i The multiplication coefficient matrix obtained from the previous iteration is calculated using the number of state transitions n. i The relationship between the number of state transitions and the mean μ is used to obtain the corresponding multiplication coefficient matrix, which is then used as the multiplication coefficient matrix for the current iteration number.

[0085] The number of state transitions and the average number of state transitions are both the values ​​of the previous iteration number.

[0086] It is understandable that the first and second coefficients can be set according to the actual situation, and there is no restriction here. They can be the same or different.

[0087] This invention acquires various business request data from the client, including different data corresponding to different business request departments. Existing methods for determining hot and cold data divide large amounts of raw data without considering the characteristics of different business needs. Therefore, this invention statistically analyzes the access frequency of underlying data based on business requests from different departments on the client machine. For example, different business request types include n... q Classification. It should be noted that the same type of data may appear in different departments' business operations, so a classification model needs to be trained for it.

[0088] Correspondingly, the classification algorithm can be a traditional classification algorithm or a neural network classification model based on deep learning to obtain the probability that the data belongs to different categories, and the resulting probability distribution matrix P A The following formula shows that each row in the probability distribution matrix represents the same type of data, and each column represents a different business request department. In other words, each row represents the probability that the same type of data belongs to different businesses.

[0089]

[0090] The probability distribution matrix is ​​weighted using a probability function to obtain the corresponding weight coefficients. The probability function is not limited here; it can be a normal distribution function or a probability distribution function, as long as the discrete variable is expressed in functional form as the probability of each value occurring. As a preferred embodiment, obtaining the corresponding weight coefficients by weighting the probability distribution matrix using a probability function includes:

[0091] The corresponding weight coefficients are obtained by weighting the probability distribution matrix using the normal distribution probability function.

[0092] Specifically, in the probability function of the normal distribution, μ = 0.5 and σ = 0.17, the weighting coefficient B obtained from the normal distribution probability function is... A The formula is as follows:

[0093]

[0094] Among them, P A This is the probability distribution matrix;

[0095] It is understandable that the weight coefficients are in the range (0, 1), and the weight coefficients are multiplied by the multiplier matrix K. i The multiplication process yields the current decay time matrix T. Ai The specific formula is as follows:

[0096] T Ai =K i *BA

[0097] In addition, the decay time matrix for the first iteration is obtained by multiplying the initial multiplication coefficient matrix with the weight coefficients. The specific formula is as follows:

[0098] T A =K×B A

[0099]

[0100] Among them, T A To initialize the decay time matrix, K is the initial multiplication coefficient matrix, and B is the initial multiplication coefficient matrix. A These are the weighting coefficients.

[0101] For example: Initialize the multiplication coefficient matrix during the first iteration. Initialize the decay time matrix T A Initialize the maximum number of optimizations N = 100, the current iteration count i = 1, and the state switching count ni = 0 for each data type.

[0102] The current decay time matrix obtained in step S11 acquires client service request data within a preset time period. Each acquisition of this service request data is of the same type. Multiple acquisitions of service request data are performed within the preset time period. The decay time is calculated to perform state transitions on the service request data, and the number of state transitions is recorded to determine the average number of state transitions for different types of data.

[0103] An iterative algorithm is used to query the corresponding decay time for each business request data within the current decay time matrix based on the request type and data type. Within a preset time period, the corresponding decay time is obtained based on the received data of the same type. It should be noted that the business request data of the same type can correspond to the same or different business types; that is, the decay time for each received data may be the same or different. The data state of the business request data is determined by comparing the time interval between received data with the corresponding decay time. The decay time is adjusted based on different decay times and changes in data state to determine the number of state transitions.

[0104] Understandably, adding decay time tags is to determine the number of times a data state switches by adding tags based on different business types of business request data. Data states include active and inactive states. The active state is when the same type of data is received again before the decay time of the currently received data has expired, while the inactive state is when the same type of data is not received again during the decay time of the currently received data.

[0105] As one embodiment, the system acquires client service request data within a preset time period and adds decay time labels to the service request data according to the current decay time matrix to determine the number of state transitions of the service request data, including:

[0106] Obtain the previous business request data and the current business request data, and determine the corresponding first business type and second business type, as well as the acquisition time interval, for the previous business request data and the current business request data respectively;

[0107] Within the current decay time matrix, find the corresponding first decay time and second decay time according to the first business type and the second business type respectively, and mark the business request data as active.

[0108] When the time interval is less than the first decay time, the second decay time corresponding to the business request data is adjusted to determine the current final decay time;

[0109] When the time interval is greater than or equal to the first decay time, the second decay time is kept unchanged as the current final decay time, and the business request data is marked as inactive.

[0110] Record the changes in the status of the data marked by the business request data to the status switch count, and count the number of times the status of the business request data switches within a preset time.

[0111] Specifically, the previous and current business request data are obtained, and the corresponding first and second business types, as well as the time interval t0 between the previous and current requests, are determined. Within the current decay time matrix, the corresponding first decay time t1 and second decay time t2 are found based on the first and second business types, and the business request data is marked as active. It is understood that the first and second business types may be the same or different; if they are the same, their first and second decay times are the same.

[0112] The system checks if the time interval between two receptions is less than the first decay time. If it is less, it means that the same type of data was received again before the first decay time expired, and this same type of data remained active, so its data state did not change. If it is greater than or equal to the first decay time, it means that the same type of data was received again after the first decay time expired. After the first decay time expired, this same type of data was marked as inactive. During the first decay time, it was active, and a switch occurred. When the same type of data is received again, it becomes active, switching from inactive to active, and another switch occurs. Each data state change is recorded in the state switch count to count the number of state switches for the same type of data within a preset time. When the decay time expires, the data is marked as inactive. When the data state changes, the state switch count ni for this type of data is incremented by one.

[0113] Determining the average number of state transitions based on the number of state transitions means that the same data may have different numbers of state transitions for different business types. The average number of state transitions is calculated by summing the number of state transitions for each business type within a preset time period. As one embodiment, determining the average number of state transitions based on the number of state transitions includes:

[0114] The number of state transitions is determined based on the different business types corresponding to the business request data.

[0115] The average number of state transitions is obtained by summing the number of transitions for each state and dividing by the number of business types.

[0116] Specifically, the average number of state transitions for all types is calculated using the following formula:

[0117]

[0118] Where, n i n represents the number of data state transitions. q The total number of business types, where μ is the average number of state transitions.

[0119] Furthermore, it is determined whether the average number of current iterations or state transitions meets the preset conditions. If it does, the average number of state transitions corresponding to the current iteration is used as the final threshold to determine hot and cold data. If it does not meet the conditions, the iteration calculation continues until the preset conditions are met, and then the iteration is terminated.

[0120] Correspondingly, the preset conditions are set based on the current iteration number or the average number of state transitions. The two are related by OR, and it is acceptable as long as either parameter meets the preset conditions.

[0121] This invention provides a method for determining hot and cold data, applied to a server, comprising: obtaining the current decay time matrix of the current iteration number; obtaining client business request data within a preset time, and adding decay time labels to the business request data according to the current decay time matrix to determine the number of state transitions of the business request data, wherein the decay time labels are obtained by adjusting the decay time through the data state of the business request data, and the data state includes active state and inactive state; determining the average number of state transitions based on the number of state transitions; when the current iteration number or the average number of state transitions meets a preset condition, determining the hot and cold data of the business request data based on the average number of state transitions; when the current iteration number or the average number of state transitions does not meet the preset condition, taking the next iteration number as the new current iteration number, and returning to the step of obtaining the current decay time matrix of the current iteration number until the preset condition is met. This method acquires business request data at the business level, optimizes the decay time through an iterative algorithm to obtain the number of state transitions for the business request data, and uses the average number of state transitions as a threshold. Compared with existing methods that determine hot and cold data through simple time thresholds and LRU algorithms, this invention rationally allocates storage resources according to the decay time of different business needs at the business level, so that the determined hot and cold data reflects the personalization of requests from different business departments, thereby improving the system's processing efficiency.

[0122] Based on the above embodiments, when the time interval is less than the first attenuation time, the second attenuation time corresponding to the service request data is adjusted, including:

[0123] Determine whether the first decay time is greater than the second decay time;

[0124] If so, the second decay time will be adjusted to the first decay time as the current final decay time;

[0125] If not, the second decay time will remain unchanged as the current final decay time.

[0126] Specifically, when the time interval is less than the first decay time, the decay time is adjusted to find the optimal number of state transitions. In this embodiment, the strategy for adjusting the decay time is determined by comparing the first decay time and the second decay time. When the first decay time is greater than the second decay time, it is adjusted to the first decay time as the current final decay time; if it is less than or equal to the second decay time, it remains unchanged. The adjustment strategy can be to use the cumulative first and second decay times as the final decay time, or to calculate their average or other transformations as the final decay time. The adjustment strategy can be set according to the actual situation and is not limited here.

[0127] It is understandable that the adjustment decay time is obtained only by comparing the business request data obtained from every two adjacent acquisitions.

[0128] For example: at a certain time T total Internally, the upper layer runs related business processes. Based on the upper layer business requests, it adds decay time tags to the data and calculates the number of state transitions based on the decay time. The specific method is as follows:

[0129] 1. Upon receiving the first business request, query the decay time t1 based on the request type and data type, and mark the data as active.

[0130] 2. After time t0, if another business request is received, the decay time t2 is obtained based on the request type and data type. If t0 < t1, the data decay time is adjusted.

[0131]

[0132] Where t3 is the final decay time.

[0133] The adjustment decay time strategy provided in this embodiment of the invention facilitates the subsequent search for an optimal number of state switching cycles.

[0134] Based on the above embodiments, the preset condition is that the current iteration count reaches the total number of iterations or the number of state transitions is less than the average number of state transitions, corresponding to a threshold where the number of business types is less than the total number of business types. When the current iteration count or the average number of state transitions meets the preset condition, the hot and cold data of the business request data are determined based on the average number of state transitions, including:

[0135] Business request data with fewer state transitions than the average number of state transitions is identified as hot data.

[0136] Business request data whose state transition count is greater than or equal to the average state transition count is identified as cold data.

[0137] It should be noted that the two parameters in the preset conditions are related by OR. As long as either parameter satisfies the preset condition, the preset condition is considered to be met, and the iteration can be stopped. The average number of state transitions corresponding to the current iteration number is used as a threshold to filter and determine hot and cold data.

[0138] Among them, the threshold that the number of business types corresponding to the number of state transitions being less than the average number of state transitions is less than the total number of business types is n. i The number of business types corresponding to <μ is less than the threshold of the total number of types. For example, satisfying n i If there are 2 types of business categories with a value <μ, and the total number of categories is 20, and the threshold is 20%, then if the corresponding value 2 is less than 20 * 20%, the iterative algorithm can be terminated. The threshold for the total number of categories can be set according to the actual situation and is not specifically limited.

[0139] Business request data with fewer state transitions than the average number of state transitions is identified as hot data.

[0140] Business request data whose state transition count is greater than or equal to the average state transition count is identified as cold data.

[0141] This invention provides a method for classifying data based on the number of state transitions for each data type. Data with fewer than the average number of state transitions is classified as hot data, while data with more than the average number of state transitions is classified as cold data. The filtering threshold is determined by the average number of transitions. Compared to existing methods that determine hot and cold data using simple time thresholds and LRU algorithms, this method ensures that the identified hot and cold data reflects the individualized requests from different business departments.

[0142] Figure 2 A flowchart of another method for determining hot and cold data provided in an embodiment of the present invention is shown below. Figure 2 As shown, the method includes:

[0143] S21: Obtain the probability distribution matrix through a classification algorithm;

[0144] S22: Initialize parameters;

[0145] S23: Run upper-layer services;

[0146] S24: Update data decay time;

[0147] S25: Number of times the statistical status has changed;

[0148] S26: Optimize the multiplication coefficient matrix;

[0149] S27: Determine whether the current iteration count has reached the termination condition. If yes, proceed to step S28; otherwise, return to step S23.

[0150] S28: Classify the data as hot or cold based on the obtained threshold.

[0151] For an introduction to another method for determining hot and cold data provided by the present invention, please refer to the above method embodiments. The present invention will not be described again here, but it has the same beneficial effects as the above method for determining hot and cold data.

[0152] The foregoing has described in detail various embodiments of the method for determining hot and cold data. Based on this, the present invention also discloses an apparatus for determining hot and cold data corresponding to the above-described method. Figure 3 This is a structural diagram of a device for determining hot and cold data provided in an embodiment of the present invention. Figure 3 As shown, the device for determining hot and cold data includes:

[0153] Module 11 is used to obtain the current decay time matrix for the current iteration number;

[0154] The first determining module 12 is used to acquire the client's business request data within a preset time, and add a decay time label to the business request data according to the current decay time matrix to determine the number of state switching of the business request data. The decay time label is obtained by adjusting the decay time through the data state of the business request data, and the data state includes active state and inactive state.

[0155] The second determining module 13 is used to determine the average number of state transitions based on the number of state transitions.

[0156] The third determination module 14 is used to determine the hot and cold data of the business request data based on the average number of state transitions when the average number of current iterations or state transitions meets the preset conditions.

[0157] Return module 15 is used to take the next iteration number as the new current iteration number and return the steps to obtain the current decay time matrix of the current iteration number until the preset conditions are met when the average of the current iteration number or the number of state transitions does not meet the preset conditions.

[0158] Since the embodiments of the device part correspond to the embodiments described above, please refer to the embodiments described in the method part for the embodiments of the device part, and will not be repeated here.

[0159] For a description of the device for determining hot and cold data provided by the present invention, please refer to the above method embodiments. The present invention will not be described again here, but it has the same beneficial effects as the above method for determining hot and cold data.

[0160] Figure 4 A structural diagram of another device for determining hot and cold data provided in an embodiment of the present invention is shown below. Figure 4 As shown, the device includes:

[0161] Memory 21 is used to store computer programs;

[0162] Processor 22 is used to implement the steps of determining hot and cold data when executing a computer program.

[0163] The device for determining hot and cold data provided in this embodiment may include, but is not limited to, smartphones, tablets, laptops, or desktop computers.

[0164] The processor 22 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 22 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 22 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 22 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 22 may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.

[0165] The memory 21 may include one or more computer-readable storage media, which may be non-transitory. The memory 21 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 21 is used to store at least the following computer program 211, which, after being loaded and executed by the processor 22, is capable of implementing the relevant steps of the method for determining hot and cold data disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 21 may also include an operating system 212 and data 213, etc., and the storage method may be temporary storage or permanent storage. The operating system 212 may include Windows, Unix, Linux, etc. The data 213 may include, but is not limited to, the data involved in the method for determining hot and cold data.

[0166] In some embodiments, the device for determining hot and cold data may further include a display screen 23, an input / output interface 24, a communication interface 25, a power supply 26, and a communication bus 27.

[0167] Those skilled in the field can understand, Figure 4 The structure shown does not constitute a limitation on the device for determining hot and cold data and may include more or fewer components than shown.

[0168] The processor 22 implements the method for determining hot and cold data provided in any of the above embodiments by calling instructions stored in the memory 21.

[0169] For a description of the device for determining hot and cold data provided by the present invention, please refer to the above method embodiments. The present invention will not be described again here, but it has the same beneficial effects as the above method for determining hot and cold data.

[0170] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by processor 22, implements the steps of the method for determining hot and cold data as described above.

[0171] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0172] For an introduction to the computer-readable storage medium provided by the present invention, please refer to the above method embodiments. The present invention will not be described again here, but it has the same beneficial effects as the above method for determining hot and cold data.

[0173] The foregoing has provided a detailed description of the method, apparatus, and medium for determining hot and cold data provided by this invention. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from the principles of the invention, and these improvements and modifications also fall within the protection scope of the claims of this invention.

[0174] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for determining hot and cold data, characterized in that, Applied to the server side, including: Obtain the current decay time matrix for the current iteration number; Within a preset time period, the service request data of the client is acquired, and a decay time label is added to the service request data according to the current decay time matrix to determine the number of state switching of the service request data. The decay time label is obtained by adjusting the decay time through the data state of the service request data, and the data state includes active state and inactive state. The average number of state transitions is determined based on the number of state transitions. When a preset condition is met, the hot and cold data of the service request data are determined based on the average number of state transitions; wherein, the preset condition is that the current iteration count reaches the total number of iterations or the number of state transitions is less than the threshold of the number of service types corresponding to the average number of state transitions being less than the total number of service types. If the preset condition is not met, the next iteration number is taken as the new current iteration number, and the process returns to the step of obtaining the current decay time matrix of the current iteration number until the preset condition is met. The process of determining the current decay time matrix includes: The multiplication factor matrix for the current iteration number is determined based on the relationship between the average number of state transitions in the previous iteration and the number of state transitions, wherein the multiplication factor matrix for the first iteration is the initial multiplication factor matrix; Obtain the business request data of the client, wherein each business request data includes different data corresponding to each business request department; Based on the classification algorithm, a probability distribution matrix is ​​obtained by classifying the business request data of each of the above-mentioned business requests. In the probability distribution matrix, each row is the same type of data, and each column represents the different business request departments. The corresponding weight coefficients are obtained by applying a probability function to the probability distribution matrix. The current decay time matrix is ​​obtained by multiplying the weight coefficients with the multiplication coefficient matrix, wherein the decay time matrix of the first iteration number is obtained by multiplying the initial multiplication coefficient matrix with the weight coefficients; The step of acquiring client service request data within a preset time period and adding decay time labels to the service request data according to the current decay time matrix to determine the number of state transitions of the service request data includes: Obtain the previous business request data and the current business request data, and determine the corresponding first business type and second business type, as well as the time interval between obtaining the previous and current business request data, based on the previously obtained business request data and the current business request data; Within the current decay time matrix, the corresponding first decay time and second decay time are found according to the first service type and the second service type, respectively, and the service request data is marked as the active state. When the time interval is less than the first decay time, the second decay time corresponding to the service request data is adjusted to determine the current final decay time; When the time interval is greater than or equal to the first decay time, the second decay time is kept unchanged as the current final decay time, and the service request data is marked as the inactive state. The changes in the state of the business request data are recorded in the state switching count, and the number of state switching times of the business request data within the preset time period is counted.

2. The method for determining hot and cold data according to claim 1, characterized in that, The process of determining the multiplication coefficient matrix for the current iteration number specifically includes: Obtain the multiplication factor matrix of the previous iteration number; When the number of state transitions is greater than the average number of state transitions, the multiplication factor matrix of the previous iteration number is added to the first coefficient to obtain the multiplication factor matrix of the current iteration number. When the number of state transitions is equal to the average number of state transitions, then the multiplication factor matrix of the previous iteration number is the multiplication factor matrix of the current iteration number. When the number of state transitions is less than the average number of state transitions, the multiplication factor matrix of the previous iteration number is multiplied by the second coefficient to obtain the multiplication factor matrix of the current iteration number.

3. The method for determining hot and cold data according to claim 2, characterized in that, The step of obtaining the corresponding weight coefficients by weighting the probability distribution matrix using a probability function includes: The corresponding weight coefficients are obtained by weighting the probability distribution matrix using a normal distribution probability function.

4. The method for determining hot and cold data according to claim 1, characterized in that, When the time interval is less than the first attenuation time, the second attenuation time corresponding to the service request data is adjusted, including: Determine whether the first decay time is greater than the second decay time; If so, the second decay time is adjusted to the first decay time as the current final decay time; If not, the second decay time remains unchanged as the current final decay time.

5. The method for determining hot and cold data according to claim 1, characterized in that, Determining the average number of state transitions based on the number of state transitions includes: The number of state transitions is determined based on the different business types corresponding to the business request data. The average number of state transitions is obtained by summing the number of each state transition and dividing it by the number of service types.

6. The method for determining hot and cold data according to any one of claims 1 to 5, characterized in that, Determining the hot and cold data of the service request data based on the average number of state transitions includes: The business request data corresponding to a state transition count less than the average state transition count is identified as hot data. Business request data whose state switching count is greater than or equal to the average state switching count is identified as cold data.

7. A device for determining hot and cold data, characterized in that, Applied to the server side, including: The acquisition module is used to obtain the current decay time matrix for the current iteration number; The first determining module is used to acquire client service request data within a preset time period, and add a decay time label to the service request data according to the current decay time matrix to determine the number of state transitions of the service request data. The decay time label is obtained by adjusting the decay time based on the data state of the service request data, and the data state includes an active state and an inactive state. Acquiring client service request data within the preset time period and adding a decay time label to the service request data according to the current decay time matrix to determine the number of state transitions of the service request data includes: acquiring the previous service request data and the current service request data, and determining the number of state transitions based on the previously acquired service request data and the currently acquired service request data. Define the corresponding first and second business types and obtain the time interval between the previous and current times; find the corresponding first and second decay times according to the first and second business types in the current decay time matrix, and mark the business request data as active; when the time interval is less than the first decay time, adjust the second decay time corresponding to the business request data to determine the current final decay time; when the time interval is greater than or equal to the first decay time, keep the second decay time unchanged as the current final decay time, and mark the business request data as inactive; record the changes in the marked data state of the business request data in the state switch count, and count the number of state switches of the business request data within a preset time. The second determining module is used to determine the average number of state transitions based on the number of state transitions. The third determining module is used to determine the hot and cold data of the service request data based on the average number of state switching times when a preset condition is met; wherein, the preset condition is that the current iteration number reaches the total number of iterations or the number of state switching times is less than the threshold of the number of service types corresponding to the average number of state switching times is less than the threshold of the total number of service types. The return module is used to, when the preset conditions are not met, take the next iteration number as the new current iteration number and return to the step of obtaining the current decay time matrix of the current iteration number until the preset conditions are met; the process of determining the current decay time matrix includes: determining the multiplication coefficient matrix of the current iteration number based on the relationship between the average number of state transitions and the number of state transitions in the previous iteration number, wherein the multiplication coefficient matrix of the first iteration number is the initialization multiplication coefficient matrix; obtaining the business request data of the client, wherein each business request data includes different data corresponding to each business request department; classifying each business request data based on a classification algorithm to obtain a probability distribution matrix, wherein each row in the probability distribution matrix is ​​the same type of data, and each column is a different business request department; weighting the probability distribution matrix through a probability function to obtain the corresponding weight coefficients; multiplying the weight coefficients with the multiplication coefficient matrix to obtain the current decay time matrix, wherein the decay time matrix of the first iteration number is obtained by multiplying the initialization multiplication coefficient matrix with the weight coefficients.

8. A device for determining hot and cold data, characterized in that, include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the method for determining hot and cold data as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method for determining hot and cold data as described in any one of claims 1 to 6.

Images