A data hotness processing method, system, device and medium
By introducing a time decay factor and a dynamic parameter adjustment mechanism, the problems of inaccurate data heat calculation and resource waste in existing technologies are solved, and accurate data heat management and efficient storage resource utilization are achieved.
Patent Information
- Application Number
- CN202610393091.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-03-27
- Publication Date
- 2026-07-03
Smart Images
Figure CN122332449A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of data processing technology, and in particular to a data popularity processing method, system, device and medium. Background Technology
[0002] With the rapid development of internet technology and the deepening of information technology construction, the business data generated by various industries is growing exponentially. Faced with massive amounts of data, how to formulate the best storage strategy based on factors such as data access frequency, capacity, performance requirements, retention time, and importance has become the key to improving the performance of business systems and reducing operating costs. Typically, in order to optimize storage resource utilization, the system divides data into "hot data" (high-frequency access) and "cold data" (low-frequency access), storing hot data in high-performance storage media (such as SSDs) and migrating cold data to low-cost storage media (such as tape libraries or high-capacity HDDs).
[0003] However, existing data popularity calculation techniques have many shortcomings in practical applications, mainly in the following aspects: First, existing methods for calculating popularity generally employ fixed parameter models. For example, they typically use a fixed decay rate or simple accumulation of visit counts to measure data popularity. This static calculation method cannot reflect the true changes in data activity over time. Especially when processing new data, the system often struggles to capture short-term trends of explosive growth or rapid cooling, resulting in either artificially inflated popularity values (misclassified as long-term hot data) or understated popularity values (failing to enter the hot list in a timely manner), thus affecting the accuracy of data classification.
[0004] Secondly, existing technologies are prone to the "Matthew effect." Due to the lack of an effective dynamic adjustment mechanism, once certain data receives a high number of visits in the initial stage, its popularity value will continue to accumulate and occupy a high position for a long time, causing the system to continuously allocate it to high-performance storage resources, resulting in a waste of storage resources; while other potentially valuable data is suppressed for a long time due to its low initial popularity and has difficulty getting a chance to be displayed.
[0005] Furthermore, existing popularity analysis lacks the ability to integrate multi-source data. Most tools sort data based on a single access characteristic (such as the number of reads), ignoring the impact of data updates and deletions on data value. They also lack comprehensive correlation analysis between historical popularity trends and current access characteristics, resulting in biased popularity calculation results that fail to accurately reflect the true business value of the data.
[0006] Finally, existing computational methods often fall short when faced with the real-time processing needs of massive amounts of data. Due to the lack of efficient dynamic update mechanisms and optimization algorithms, the system struggles to update the popularity of large-scale datasets in a short period of time, preventing database administrators from making decisions based on the latest popularity data, thereby hindering the overall query efficiency of the business system.
[0007] In summary, accurately calculating and updating data popularity while considering data activity and time decay factors, overcoming the limitations of fixed-parameter calculations, and achieving dynamic popularity management based on user behavior and data lifecycle has become a pressing technical challenge. Developing a system capable of dynamically updating data popularity is of great significance for improving database management efficiency, reducing storage costs, and achieving more accurate data analysis and applications. Summary of the Invention
[0008] To address the aforementioned issues, this disclosure provides a data popularity processing method, system, device, and medium.
[0009] Firstly, a data popularity processing method includes: Obtain the historical hot and cold trends of the data and the current access characteristics. The historical hot and cold trends include the number of reads, deletions and updates of historical data, and the current access characteristics include the number of reads, deletions and updates at the current moment. The initial popularity is calculated based on the platform weight at the moment the data is added, historical hot and cold trends, and popularity decay factor. The time decay factor is dynamically adjusted based on the interval between the data addition time and the target time. When the interval is less than the preset target time, the time decay factor is obtained by dynamic calculation. When the interval is greater than or equal to the preset target time, the time decay factor is set to the preset value. The decayed heat is obtained by calculating the decay of the initial heat based on the time decay factor. Based on the decaying popularity and current access characteristics, the latest total popularity is calculated, and the updated data popularity is stored in the database; Data analysis is performed based on popularity data in the database.
[0010] Furthermore, it involves obtaining historical hot / cold trends and current access characteristics of the data, and then includes: Normalize the data in historical hot and cold trends and the data in current access characteristics to make all data points of the same order of magnitude. The normalization process includes dividing each data point by the maximum value of its corresponding data type.
[0011] Furthermore, the time decay factor is obtained through dynamic calculation, including: When the interval duration T is less than the preset target duration T0, the time decay factor K3 is obtained by a dynamic calculation method based on the decay index data, as shown in the following formula:
[0012] in, Here, T is the preset value, T is the interval between the data addition time and the target time, and T0 is the preset target duration. The basic attenuation coefficient.
[0013] Furthermore, the initial popularity HO is calculated based on the platform weight at the moment the data is added, historical hot and cold trends, and popularity decay factor, as shown in the following formula:
[0014] in, , , These are the normalized historical read count, historical delete count, and historical update count, respectively. , , These are the preset weighting coefficients for the number of historical reads, historical deletions, and historical updates; For platform weight, The heat decay factor for new data moments.
[0015] Furthermore, the decay of the initial heat is calculated based on the time decay factor to obtain the decayed heat HF, as shown in the following formula:
[0016] in, Initial popularity, t is the time decay factor, t is the current time, and t0 is the time when the data was added.
[0017] Furthermore, based on the decaying popularity and current access characteristics, the latest total popularity HD is calculated using the following formula:
[0018] in, To reduce heat, , , These are the normalized current read count, current delete count, and current update count, respectively. , , These are the preset weighting coefficients for the current number of reads, the current number of deletes, and the current number of updates.
[0019] Furthermore, it also includes: A single-pass clustering method is used to calculate the similarity between data using the cosine distance of vectors; Process one piece of data at a time in the order of input. For the data stream that arrives in sequence, determine whether the current data belongs to an existing class or a new data class is created based on the similarity between the current data and the existing classes.
[0020] Secondly, a data popularity processing system includes: The data acquisition module is used to acquire the historical hot and cold trends and current access characteristics of the data. The historical hot and cold trends include the number of reads, deletions and updates of historical data, and the current access characteristics include the number of reads, deletions and updates at the current moment. The time decay model construction module is used to construct a time decay model based on the time decay factor. The time decay model includes an initial popularity calculation submodule and a popularity decay calculation submodule. The initial popularity calculation submodule is used to calculate the initial popularity based on the platform weight and historical hot and cold trends at the time of data addition. The popularity decay calculation submodule is used to calculate the decay of the initial popularity using the time decay factor to obtain the decayed popularity. The time decay parameter optimization module is used to dynamically adjust the time decay factor based on the interval between the data addition time and the target time. Specifically, when the interval is less than the preset target time, the time decay factor is obtained by dynamic calculation. When the interval is greater than or equal to the preset target time, the time decay factor is set to the preset target parameter value. The data popularity update module is used to calculate the latest total popularity based on the decay popularity and current access characteristics, and store the updated data popularity in the database; The data analysis application module performs data analysis based on the popularity data in the database.
[0021] Thirdly, an electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, which stores computer programs; When a processor executes a computer program stored in memory, it implements the aforementioned data heat processing method.
[0022] Fourthly, a computer-readable storage medium stores a computer program that, when executed by a processor, implements the aforementioned data heat processing method.
[0023] This disclosure includes at least the following beneficial effects: This disclosure solves the problems of data heat distortion caused by fixed parameter calculation, inaccurate reflection of the heat of newly added data, and Matthew effect in the prior art by introducing a time decay factor and a dynamic parameter adjustment mechanism. It achieves accurate and real-time calculation of data heat, effectively improving database management efficiency and storage resource utilization.
[0024] Other features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the disclosure. The objects and other advantages of this disclosure may be realized and obtained by means of the structures pointed out in the description and the accompanying drawings. Attached Figure Description
[0025] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0026] Figure 1 This is a schematic flowchart of the processing method according to an embodiment of the present disclosure; Figure 2 This is a schematic diagram of the system architecture of an embodiment of this disclosure; Figure 3 This is a schematic diagram of the electronic device structure according to an embodiment of the present disclosure. Detailed Implementation
[0027] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.
[0028] like Figure 1 As shown, a data popularity processing method includes: S101, Obtain the historical hot and cold trends and current access characteristics of the data, where the historical hot and cold trends include the number of reads, deletions and updates of historical data, and the current access characteristics include the number of reads, deletions and updates at the current moment; S102, calculate the initial popularity based on the platform weight, historical hot and cold trends, and popularity decay factor at the moment the data is added; S103, dynamically adjust the time decay factor according to the interval between the data addition time and the target time; wherein, when the interval is less than the preset target time, the time decay factor is obtained by dynamic calculation; when the interval is greater than or equal to the preset target time, the time decay factor is set to the preset value. S104, calculates the decay of the initial heat based on the time decay factor to obtain the decayed heat; S105: Calculate the latest total popularity based on the decay popularity and current access characteristics, and store the updated data popularity in the database; S106, perform data analysis based on popularity data in the database.
[0029] In practice: Obtain historical hot / cold data trends and data access characteristics. Historical hot / cold data trends include the number of reads (PL_H), deletes (CM_H), and updates (Z_H) of historical data. Data access characteristics include the number of reads (PL_N), deletes (CM_N), and updates (Z_N) at the current moment.
[0030] The acquired data is normalized to ensure that the data from various platforms are on the same order of magnitude. Specifically, the normalized data is as follows:
[0031]
[0032]
[0033]
[0034]
[0035]
[0036] Example: Historical data: Read count (PL_H) = 100, Delete count (CM_H) = 50, Update count (Z_H) = 30.
[0037] Current data: Read count (PL_N) = 20, Delete count (CM_N) = 10, Update count (Z_N) = 5. Maximum values are used for normalization: Assume the historical maximum values are MAX(PL_H) = 200, MAX(CM_H) = 100, and MAX(Z_H) = 50; the current maximum values are MAX(PL_N) = 50, MAX(CM_N) = 20, and MAX(Z_N) = 10.
[0038] Data normalization is performed on historical data and current data to make the data at the same order of magnitude. The normalization formula is as follows: = 0.5 = 0.5 = 0.6 = 0.4 = 0.5 = 0.5 The time decay parameter is dynamically adjusted according to the new data addition time. The time decay parameter optimization module dynamically adjusts the time decay parameter according to the interval duration T between the new data addition time and the target time. If T < T0, the dynamic calculation method based on the decay index data is used to obtain the time decay parameter K3:
[0039] If T > T0, the time decay parameter K3 is set to the preset target parameter value .
[0040] Example: Time parameter: The new data addition time t0 = 0, the current time t = 10 hours, the interval duration T = t - t0 = 10 hours, and the threshold T0 = 24 hours. The preset value of the time decay parameter = 0.1.
[0041] It needs to be obtained through the time decay parameter optimization module . Since T = 10 hours < T0 = 24 hours, dynamic calculation is adopted:
[0042] Among them, is the preset value, T is the interval duration between the new data addition time and the target time, T0 is the preset target duration, is the basic decay coefficient.
[0043] When = 0.1, T = 10, T0 = 24, = 0.81, = 0.1944.
[0044] Calculate the initial popularity according to the platform weight at the new data addition time and the data index data:
[0045] Among them, is the platform weight, The heat decay factor for new data moments. , , Based on historical experience, the weighting ratios are determined to be 0.5, 0.25, and 0.25. Platform weight. = 1.0 (adjustable depending on the platform), heat decay factor = 0.9.
[0046] Initial heat calculation, example: Substitute the values into the formula above: HO = (0.5 × 0.5 + 0.25 × 0.5 + 0.25 × 0.6) × 1.0 × 0.9 = (0.25 + 0.125 + 0.15) × 0.9 = 0.525 × 0.9 = 0.4725 The decay of initial heat is calculated using a time decay factor:
[0047] Where t represents the current time, and t0 represents the time when the data was added. This is the time decay factor.
[0048] Substitute the values into the formula above: HF = 0.4725 × (1 - 0.1944)^{10} = 0.4725 × 0.8056^{10} Calculated exponent: 0.8056^{10} ≈ 0.1151 (by step-by-step calculation: 0.8056^2 ≈ 0.6490, 0.8056^4 ≈ 0.4212, 0.8056^8 ≈ 0.1774, 0.8056^{10} ≈ 0.1774 × 0.6490 ≈ 0.1151) HF ≈ 0.4725 × 0.1151 ≈ 0.0544 Based on the time decay model and time decay parameters, the latest total popularity is calculated, and the updated data popularity is stored in the database:
[0049] in , , These are weighting coefficients, which are determined to be 0.1, 0.05, and 0.05 based on historical experience.
[0050] Example: Substitute the values into the formula above: HD = 0.0544 × (1 + 0.1 × 0.4 + 0.05 × 0.5 + 0.05 × 0.5) = 0.0544 × (1 + 0.04 + 0.025 + 0.025) = 0.0544 × 1.09 ≈ 0.0593 Based on the above calculations, the latest total popularity (HD) of the data is approximately 0.0593. This value will be stored in the database for subsequent popularity analysis and recommendation applications.
[0051] Data analysis is performed based on the popularity data in the database to generate a popularity ranking list and provide database administrators with a basis for processing cold / hot data.
[0052] This disclosure uses a time decay factor as its core driver, acquiring historical trends and current behavioral characteristics of data. It utilizes time decay to scientifically assign initial popularity and achieve reasonable decay over time, adaptively adjusting the decay rate according to the data lifecycle stage, thus avoiding a static "one-size-fits-all" approach. Based on this, it integrates multi-dimensional access metrics for weighted aggregation, outputting a high-precision total popularity value. Finally, the popularity results are transformed into actionable hot and cold data management strategies. The entire solution forms a closed-loop process of "perception—modeling—optimization—update—application," significantly improving the accuracy and timeliness of popularity calculation and providing reliable technical support for the intelligent hierarchical storage of massive heterogeneous data.
[0053] This invention, by introducing a time decay factor and a dynamic parameter adjustment mechanism, constructs a complete dynamic data popularity update system, which has the following significant technical effects compared with existing technologies: This disclosure calculates and adjusts the popularity value of data by combining historical hot and cold trends, data access characteristics, and time decay factors. This overcomes the limitations caused by fixed parameter calculations in existing technologies, and more accurately reflects users' real interests and needs. As a result, it improves the accuracy of data popularity ranking results and user experience, especially when adding new data, it can accurately reflect popularity trends.
[0054] This disclosure employs algorithms based on time decay factors, time decay models, and time decay parameters to achieve integrated analysis of multi-source data, solving the problem of the lack of multi-source data integration capabilities in existing technologies and meeting the needs of a wider range of application scenarios.
[0055] This disclosure avoids the problem of popularity spikes caused by a certain data being selected more often in a specific scenario by dynamically updating the data popularity mechanism, effectively improving the data popularity update mechanism and enhancing the accuracy of data popularity.
[0056] This disclosure, based on a dynamic calculation mechanism of time decay factor, enables efficient real-time processing of massive amounts of data, meets users' needs for popularity data, and improves the overall processing efficiency of the system.
[0057] like Figure 2 As shown, a data popularity processing system includes: The data acquisition module is used to acquire the historical hot and cold trends and current access characteristics of the data. The historical hot and cold trends include the number of reads, deletions and updates of historical data, and the current access characteristics include the number of reads, deletions and updates at the current moment. The time decay model construction module is used to construct a time decay model based on the time decay factor. The time decay model includes an initial popularity calculation submodule and a popularity decay calculation submodule. The initial popularity calculation submodule is used to calculate the initial popularity based on the platform weight and historical hot and cold trends at the time of data addition. The popularity decay calculation submodule is used to calculate the decay of the initial popularity using the time decay factor to obtain the decayed popularity. The time decay parameter optimization module is used to dynamically adjust the time decay factor based on the interval between the data addition time and the target time. Specifically, when the interval is less than the preset target time, the time decay factor is obtained by dynamic calculation. When the interval is greater than or equal to the preset target time, the time decay factor is set to the preset target parameter value. The data popularity update module is used to calculate the latest total popularity based on the decay popularity and current access characteristics, and store the updated data popularity in the database; The data analysis application module performs data analysis based on the popularity data in the database.
[0058] In practice: This disclosure calculates a time decay factor based on historical hot and cold data trends (historical read count, deletion count, and update count) and current data access characteristics (current read count, deletion count, and update count). By combining a time decay model and dynamic parameter adjustments, it achieves accurate calculation and real-time updates of data popularity, providing database administrators with accurate basis for cold / hot data processing, effectively reducing storage costs, and improving overall system efficiency.
[0059] The data acquisition module is used to calculate the time decay factor based on the historical hot and cold trends of the data and the data access characteristics. The historical hot and cold trends of the data include the number of reads, deletions and updates of historical data, and the data access characteristics include the number of reads, deletions and updates at the current moment. The time decay model construction module is used to construct a time decay model based on the time decay factor. The time decay model includes an initial popularity calculation submodule and a popularity decay calculation submodule. The initial popularity calculation submodule is used to calculate the initial popularity based on the platform weight and data indicator data at the time of data addition. The popularity decay calculation submodule is used to perform decay calculation on the initial popularity using the time decay factor. The time decay parameter optimization module is used to dynamically adjust the time decay parameter according to the interval between the data addition time and the target time. When the interval is less than the preset target time, the time decay parameter is obtained by dynamic calculation based on decay index data. When the interval is greater than the preset target time, the time decay parameter is set to the preset target parameter value. The data popularity update module is used to calculate the latest total popularity based on the time decay model and time decay parameters, and store the updated data popularity in the database. The data analysis application module is used to perform data analysis based on the popularity data in the database, generate popularity rankings, and provide database administrators with a basis for processing cold / hot data.
[0060] The heat decay calculation submodule in the time decay model adopts a single-pass clustering method, which uses the cosine distance of vectors to calculate the similarity between data. For data streams that arrive sequentially, the method processes one data at a time according to the input order. Based on the similarity between the current data and the existing classes, the data is either classified as an existing class or a new data class is created.
[0061] The time decay factor calculation module also includes a data normalization processing unit, which is used to normalize the platform data to ensure that all platform data are on the same order of magnitude, thus avoiding the influence of deviations caused by differences in data magnitude.
[0062] This disclosure calculates a time decay factor based on historical data hot / cold trends (historical read count, delete count, update count) and current data access characteristics (current read count, delete count, update count) to achieve dynamic updates of data popularity. This disclosure includes an initial popularity calculation submodule (calculating initial popularity based on platform weight and data indicators) and a popularity decay calculation submodule (using the time decay factor to calculate the decay of initial popularity), forming a complete popularity calculation framework.
[0063] This disclosure dynamically adjusts the time decay parameter based on the interval between the data addition time and the target time. When the interval is less than a preset threshold, a dynamic calculation method is used; when it is greater than the threshold, a preset value is used, achieving adaptive parameter adjustment. This disclosure combines the heat decay calculation results with the current data access characteristics, and calculates the latest total heat using a weighted summation method, effectively solving the problem of inaccurate data heat updates.
[0064] This disclosure improves the accuracy and authenticity of popularity calculation. It abandons the existing method of calculating popularity using fixed parameters (such as a fixed decay rate) and instead dynamically adjusts popularity using a time decay factor by comprehensively analyzing historical hot and cold trends (historical reads, deletions, and updates) and current access characteristics (current reads, deletions, and updates). This method can keenly capture changes in data at different stages of its lifecycle, especially for newly added data, accurately reflecting its popularity trend and avoiding distortion of popularity values caused by fixed parameter calculations, thereby significantly improving the accuracy of popularity ranking results.
[0065] This disclosure effectively overcomes the Matthew effect and optimizes storage resource allocation. Existing technologies often lead to a continuous increase in the popularity of frequently selected data, resulting in long-term occupation of high-performance storage resources, while other data is suppressed. This disclosure uses a time decay model to cause data popularity to decay exponentially over time, and adjusts this based on current access behavior. This means that only continuously accessed data can maintain high popularity, while overheated data that is no longer needed will rapidly decrease in popularity. This mechanism effectively breaks the Matthew effect of "the strong get stronger," avoids waste of storage resources, makes the tiered storage of hot and cold data more reasonable, and reduces storage costs.
[0066] This disclosure enables integrated analysis and personalized services for multi-source data. It considers not only the number of reads but also incorporates multi-dimensional operational features such as the number of deletions and updates into the calculation model, and utilizes single-pass clustering and cosine distance for similarity analysis. This integrated analysis capability of multi-source data allows the popularity value to more comprehensively reflect the overall value of the data, meeting the needs of a wider range of application scenarios and providing data support for achieving more accurate and personalized proactive information services.
[0067] This disclosure enhances the real-time performance and system efficiency of massive data processing. For massive data scenarios, this disclosure eliminates biases caused by differences in data volume through data normalization and combines this with an efficient time decay parameter optimization algorithm (adaptively switching calculation strategies based on data age) to achieve rapid calculation and updating of data popularity. This ensures that the system can respond promptly to user needs, providing database administrators with real-time data processing data for cold / hot data, thereby significantly improving the query efficiency and overall management efficiency of the business system.
[0068] like Figure 3 As shown, this disclosure provides an electronic device, including a processor 301, a communication interface 302, a memory 303, and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304; Memory 303 stores computer programs; The processor 301 implements the above method when executing a computer program stored in the memory 303.
[0069] This disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.
[0070] The computer-readable storage medium may be included in the device / apparatus described in the above embodiments; or it may exist independently and not assembled into the device / apparatus. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.
[0071] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
[0072] Although the present disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.
Claims
1. A data hotness processing method, characterized in that, include: Obtain the historical hot and cold trends of the data and the current access characteristics. The historical hot and cold trends include the number of reads, deletions and updates of historical data, and the current access characteristics include the number of reads, deletions and updates at the current moment. The initial popularity is calculated based on the platform weight at the moment the data is added, historical hot and cold trends, and popularity decay factor. The time decay factor is dynamically adjusted based on the interval between the data addition time and the target time. When the interval is less than the preset target time, the time decay factor is obtained by dynamic calculation. When the interval is greater than or equal to the preset target time, the time decay factor is set to the preset value. The decayed heat is obtained by calculating the decay of the initial heat based on the time decay factor. Based on the decaying popularity and current access characteristics, the latest total popularity is calculated, and the updated data popularity is stored in the database; Data analysis is performed based on popularity data in the database.
2. The data popularity processing method according to claim 1, characterized in that, After obtaining historical hot and cold trends and current access characteristics of the data, it also includes: Normalize the data in historical hot and cold trends and the data in current access characteristics to make all data points of the same order of magnitude. The normalization process includes dividing each data point by the maximum value of its corresponding data type.
3. The data popularity processing method according to claim 1, characterized in that, The time decay factor is obtained through dynamic calculation, including: When the interval duration T is less than the preset target duration T0, the time decay factor K3 is obtained by a dynamic calculation method based on the decay index data, as shown in the following formula: wherein, is a preset value, T is an interval duration between the data adding time and the target time, T0 is a preset target duration, is a basic attenuation coefficient.
4. The data popularity processing method according to claim 1, characterized in that, The initial popularity HO is calculated based on the platform weight at the moment the data is added, historical popularity trends, and popularity decay factor, using the following formula: wherein, , , are normalized historical read number, historical deletion number and historical update number respectively; , , are preset historical read number, historical deletion number and historical update number weight coefficients respectively; is a platform weight, is a hotness decay factor at the time of data addition.
5. The data popularity processing method according to claim 1, characterized in that, The decayed heat HF is obtained by calculating the decay of the initial heat based on the time decay factor, as shown in the following formula: wherein, is the initial hotness, is the time decay factor, t is the current time, and t0 is the time when the data is newly added.
6. The data popularity processing method according to claim 1, characterized in that, The latest total popularity (HD) is calculated based on the decaying popularity and current access characteristics, using the following formula: wherein, is the decay temperature, , , are normalized current read, current delete and current update numbers, respectively; , , are preset weight coefficients of the current read, current delete and current update numbers, respectively.
7. The data popularity processing method according to claim 1, characterized in that, Also includes: A single-pass clustering method is used to calculate the similarity between data using the cosine distance of vectors; Process one piece of data at a time in the order of input. For the data stream that arrives in sequence, determine whether the current data belongs to an existing class or a new data class is created based on the similarity between the current data and the existing classes.
8. A data hotness processing system, comprising: include: The data acquisition module is used to acquire the historical hot and cold trends and current access characteristics of the data. The historical hot and cold trends include the number of reads, deletions and updates of historical data, and the current access characteristics include the number of reads, deletions and updates at the current moment. The time decay model construction module is used to construct a time decay model based on the time decay factor. The time decay model includes an initial popularity calculation submodule and a popularity decay calculation submodule. The initial popularity calculation submodule is used to calculate the initial popularity based on the platform weight at the time of data addition and the historical hot and cold trends. The heat decay calculation submodule is used to calculate the decay of the initial heat using the time decay factor, and obtain the decayed heat. The time decay parameter optimization module is used to dynamically adjust the time decay factor based on the interval between the data addition time and the target time. Specifically, when the interval is less than the preset target time, the time decay factor is obtained by dynamic calculation. When the interval is greater than or equal to the preset target time, the time decay factor is set to the preset target parameter value. The data popularity update module is used to calculate the latest total popularity based on the decay popularity and current access characteristics, and store the updated data popularity in the database; The data analysis application module performs data analysis based on the popularity data in the database.
9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, which stores computer programs; A processor, when executing a computer program stored in memory, implements a data heat processing method according to any one of claims 1-7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements a data heat processing method according to any one of claims 1-7.