A data processing method and apparatus

Abstract

The application discloses a data processing method and device, and relates to the technical field of big data. A specific implementation manner of the method comprises the following steps: calculating initial cold and hot values of each data according to a historical data reach log, and storing each data into a hot database or a cold database according to the initial cold and hot values of each time; wherein, the reach time interval of the data is negatively correlated with the initial cold and hot value of the data, and the reach frequency of the data is positively correlated with the initial cold and hot value of the data; migrating the data to the hot database according to a current data reach log; monitoring the proportion of hot data in the hot database, and if the proportion of the hot data exceeds a proportion threshold, migrating the hot data exceeding the threshold to the cold database according to the current time cold and hot value of the hot data. The implementation manner can solve the technical problems of complex logic and large calculation cost.

Description

Technical Field

[0001] This invention relates to the field of big data technology, and in particular to a data processing method and apparatus. Background Technology

[0002] While providing a better user experience, internet companies often accumulate massive amounts of user profile data. Hot and cold data management technology refers to accurately identifying hot and cold data and then dividing and conquering data according to their popularity, minimizing storage costs while ensuring system performance. Currently, methods for identifying hot and cold data suffer from complex implementation logic and high computational overhead. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a data processing method and apparatus to solve the technical problems of complex implementation logic and high computational overhead.

[0004] To achieve the above objectives, according to one aspect of the present invention, a data processing method is provided, comprising: The initial hot / cold value of each data point is calculated based on historical data access logs, and each data point is stored in a hot database or a cold database according to the initial hot / cold value of each data point; wherein, the data access time interval is negatively correlated with the initial hot / cold value of the data, and the data access frequency is positively correlated with the initial hot / cold value of the data. Migrate the data to the hot database based on the current data access logs; The proportion of hot data in the hot database is monitored. If the proportion of hot data exceeds the proportion threshold, the hot data exceeding the limit is migrated to the cold database according to the current hot and cold values ​​of the hot data.

[0005] Optionally, data can be migrated to a hot database based on the current data access logs, including: Extract the device identifier from the current data access logs; If the data corresponding to the device identifier is stored in a thermal database, then calculate the current hot or cold value of the data. If the data corresponding to the device identifier is stored in a cold database, then the data is migrated to a hot database, and the current hot / cold value of the data is calculated. If the device identifier is a new device identifier, then the data corresponding to the device identifier is written into the thermal database, and the initial cold and hot values ​​of the data are calculated.

[0006] Optionally, the current hot / cold value of the data can be calculated using the following formula:

[0007] in, This represents the current hot / cold value of the data. The initial hot and cold values ​​of the data. The hot / cold value of the data at the previous moment. The average hot / cold value of all data from the previous moment. The rate of change of heating and cooling values. This is the gain for both hot and cold values.

[0008] Optionally, the proportion of hot data in the hot database is monitored. If the proportion of hot data exceeds a threshold, the excess hot data is migrated to the cold database based on the current temperature values ​​of the hot data. This includes: Monitor the proportion of heat data in the heat database. If the proportion of heat data exceeds the proportion threshold, calculate the amount of heat data exceeding the limit. Sort the hot data in descending order of their current hot and cold values, and migrate the hot data that are ranked lower and exceed the limit to the cold data database.

[0009] Optionally, the initial hot / cold value of each data point is calculated based on historical data access logs, and each data point is stored in a hot database or a cold database according to its initial hot / cold value, including: Calculate the reach interval characteristics and reach frequency characteristics of each data based on historical data reach logs; Based on the reach interval characteristics and reach frequency characteristics of each data point, the initial hot and cold values ​​of each data point are calculated; Based on the initial hot and cold values ​​of the data and the preset hot and cold value thresholds, the data is divided into hot data and cold data, and stored in a hot database or a cold database respectively.

[0010] Optionally, the reach interval characteristics of each data point are calculated based on historical data reach logs, including: Determine the historical contact time for each device identifier based on historical data contact logs; Calculate the historical contact time interval for each device based on the historical contact time of each device identifier; For each device identifier, the reciprocal of each historical reach time interval corresponding to the device identifier is taken and summed to obtain the reach interval feature of the data corresponding to the device identifier.

[0011] Optionally, the frequency characteristics of each data point are calculated based on historical data reach logs, including: Determine the frequency of each device identifier's reach within each sequence time period based on historical data reach logs; For each device identifier, the reach frequency of each sequence time period is multiplied by its reach frequency and then summed to obtain the reach frequency feature of the data corresponding to the device identifier.

[0012] Optionally, based on the reach interval characteristics and reach frequency characteristics of each data point, the initial hot / cold value of each data point is calculated, including: For each data point, the reach interval feature and reach frequency feature of the data are weighted and summed to obtain the initial hot / cold value of the data.

[0013] Optionally, the method further includes: Monitor the total amount of data in the cold database and the hot database. If the total amount of data exceeds the capacity threshold, calculate the amount of data exceeding the limit. Sort the data in descending order of their current hot / cold values, and compress and transfer the data that is lower in the sort and exceeds the limit.

[0014] Optionally, the method further includes: Data from the hot database is flushed to the Redis cache, and data from the cold database is flushed to the HBase database.

[0015] Additionally, according to another aspect of the present invention, a data processing apparatus is provided, comprising: The cold start module is used to calculate the initial hot and cold values ​​of each data based on historical data access logs, and store each data into a hot database or a cold database respectively based on the initial hot and cold values ​​of each data; wherein, the data access time interval is negatively correlated with the initial hot and cold values ​​of the data, and the data access frequency is positively correlated with the initial hot and cold values ​​of the data. The calculation module is used to migrate data to the hot database based on the current data access log and calculate the current hot / cold value of the data. The monitoring module is used to monitor the proportion of hot data in the hot database. If the proportion of hot data exceeds the proportion threshold, the excess hot data will be migrated to the cold database.

[0016] Optionally, the computing module is further configured to: Extract device identifiers from data access logs; If the data corresponding to the device identifier is stored in a thermal database, then calculate the current hot or cold value of the data. If the data corresponding to the device identifier is stored in a cold database, then the data is migrated to a hot database, and the current hot / cold value of the data is calculated. If the device identifier is a new device identifier, then the data corresponding to the device identifier is written into the thermal database, and the initial cold and hot values ​​of the data are calculated.

[0017] Optionally, the computing module is further configured to: The current hot / cold value of the data is calculated using the following formula:

[0018] in, This represents the current hot / cold value of the data. The initial hot and cold values ​​of the data. The hot / cold value of the data at the previous moment. The average hot / cold value of all data from the previous moment. The rate of change of heating and cooling values. This is the gain for both hot and cold values.

[0019] Optionally, the monitoring module is further configured to: Monitor the proportion of heat data in the heat database. If the proportion of heat data exceeds the proportion threshold, calculate the amount of heat data exceeding the limit. Sort the hot data in descending order of their current hot and cold values, and migrate the hot data that are ranked lower and exceed the limit to the cold data database.

[0020] Optionally, the cold start module is further configured to: Calculate the reach interval characteristics and reach frequency characteristics of each data based on historical data reach logs; Based on the reach interval characteristics and reach frequency characteristics of each data point, the initial hot and cold values ​​of each data point are calculated; Based on the initial hot and cold values ​​of the data and the preset hot and cold value thresholds, the data is divided into hot data and cold data, and stored in a hot database or a cold database respectively.

[0021] Optionally, the cold start module is further configured to: Determine the historical contact time for each device identifier based on historical data contact logs; Calculate the historical contact time interval for each device based on the historical contact time of each device identifier; For each device identifier, the reciprocal of each historical reach time interval corresponding to the device identifier is taken and summed to obtain the reach interval feature of the data corresponding to the device identifier.

[0022] Optionally, the cold start module is further configured to: Determine the frequency of each device identifier's reach within each sequence time period based on historical data reach logs; For each device identifier, the reach frequency of each sequence time period is multiplied by its reach frequency and then summed to obtain the reach frequency feature of the data corresponding to the device identifier.

[0023] Optionally, the cold start module is further configured to: For each data point, the reach interval feature and reach frequency feature of the data are weighted and summed to obtain the initial hot / cold value of the data.

[0024] Optionally, the monitoring module is further configured to: Monitor the total amount of data in the cold database and the hot database. If the total amount of data exceeds the capacity threshold, calculate the amount of data exceeding the limit. Sort the data in descending order of their current hot / cold values, and compress and transfer the data that is lower in the sort and exceeds the limit.

[0025] Optionally, a flashing module is also included for: Data from the hot database is flushed to the Redis cache, and data from the cold database is flushed to the HBase database.

[0026] According to another aspect of the present invention, an electronic device is also provided, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any of the above embodiments.

[0027] According to another aspect of the present invention, a computer-readable medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the methods described in any of the above embodiments.

[0028] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods described in any of the above embodiments.

[0029] One embodiment of the above invention has the following advantages or beneficial effects: Because it employs a technique of calculating the initial hot / cold values ​​of each data point based on historical data access logs, storing each data point in a hot or cold database based on these initial hot / cold values, and migrating data to the hot database based on the current data access logs while monitoring the data volume in the hot database, it overcomes the technical problems of complex implementation logic and high computational overhead in existing technologies. This embodiment calculates the initial hot / cold values ​​of data using historical data access logs, making the data access time interval negatively correlated with the initial hot / cold values ​​and positively correlated with the data access frequency. This accurately distinguishes between hot and cold data, enabling cold starts, and the implementation logic is simple with low computational overhead. Furthermore, this embodiment migrates data and calculates the current hot / cold values ​​using data access logs. When the proportion of hot data exceeds a threshold, it migrates hot data with decreasing hot / cold values ​​to the cold database based on the hot / cold value sorting results. This not only accurately distinguishes between hot and cold data but also simplifies the implementation logic, reduces computational overhead, and accurately controls storage costs, significantly reducing system storage costs and thus meeting the needs of hot / cold data governance.

[0030] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a schematic diagram of the main flow of the data processing method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the main process of migrating data to a hot database according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the main flow of a data processing method according to a possible embodiment of the present invention; Figure 4 This is a schematic diagram of the main process of writing hot and cold data according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the main flow of a data processing method according to another possible embodiment of the present invention; Figure 6 This is a schematic diagram of the main modules of a data processing apparatus according to an embodiment of the present invention; Figure 7This is an exemplary system architecture diagram in which embodiments of the present invention can be applied; Figure 8 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0032] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0033] In existing technologies, the following methods are typically used to identify hot and cold data: (1) Using a data structure-based hot / cold data differentiation scheme, this approach fully utilizes the characteristics of the data structure itself and determines the hotness or coldness of the data based on its relative position. Commonly used data structures include linked lists, hash functions, and Bloom filters. The disadvantage of this approach is that it cannot comprehensively reflect the hotness or coldness of the data, and this identification method is neither quantifiable nor durable.

[0034] (2) A data hot / cold distinction scheme based on mathematical statistics is used, that is, an index is defined to determine whether the data is hot or cold. Although this scheme is beneficial to improving the accuracy of data identification, the implementation logic of hot / cold data identification is relatively complex, and the computational and storage costs are difficult to control.

[0035] All user profile data involved in this application has been agreed to and authorized by the users. The acquisition, storage, use, and processing of user profile data in the technical solution of this application all comply with the relevant provisions of national laws and regulations.

[0036] Figure 1 This is a schematic diagram of the main flow of a data processing method according to an embodiment of the present invention. As one embodiment of the present invention, such as... Figure 1 As shown, the data processing method may include: Step 101: Calculate the initial hot and cold values ​​of each data based on the historical data access logs, and store each data in the hot database or cold database according to the initial hot and cold values ​​of each data.

[0037] Consider the time it takes for the device identifier to reach ( ), frequency of contact ( ) and reach ( Based on the reach characteristics determined by the data, this embodiment of the invention calculates reach interval characteristics and reach frequency characteristics, and calculates the initial hot / cold value of the data based on the reach interval characteristics and reach frequency characteristics. Specifically, the reach interval of the data is negatively correlated with the initial hot / cold value of the data, and the reach frequency of the data is positively correlated with the initial hot / cold value of the data.

[0038] Step 102: Migrate the data to the hot database based on the current data access logs.

[0039] Data access logs from the previous day can be retrieved at a daily granularity. Data is then migrated to a hot database based on these logs, and the current hot / cold value of the data is calculated to ensure it remains up-to-date. The data access logs may include the device identifier, device type, and access time. The data stored in the hot / cold database can be user profile data, specifically including multi-dimensional user profile data such as device identifier, user gender, user age, user behavior, and user preferences. It should be noted that user profile data in the hot / cold database is stored by device identifier to facilitate searching for user profile data by device identifier.

[0040] Optionally, step 102 may include: extracting a device identifier from the data access log; if the data corresponding to the device identifier is stored in a hot database, calculating the current hot / cold value of the data; if the data corresponding to the device identifier is stored in a cold database, calculating the migration of the data to the hot database and calculating the current hot / cold value of the data; if the device identifier is a new device identifier, writing the data corresponding to the device identifier into the hot database and calculating the current hot / cold value of the data. Figure 2 As shown, the scan data reaches the logs and hot / cold databases, extracting the reached device identifiers. If the data corresponding to the device identifier is stored in the hot database, it indicates that the device identifier is a hot device identifier and the data is hot data. In this case, the current hot / cold value of the data is recalculated and overwritten with the previous hot / cold value. If the data corresponding to the device identifier is stored in the cold database, it indicates that the device identifier is a cold device identifier and the data is cold data. In this case, the data is migrated to the hot database to become hot data, and the current hot / cold value of the data is recalculated and overwritten with the previous hot / cold value. If the data corresponding to the device identifier is neither stored in the hot database nor the cold database, it indicates that the device identifier is a new device identifier that has not been accessed. In this case, the data corresponding to the device identifier is written to the hot database, and the initial hot / cold value of the data (i.e., the current hot / cold value) is calculated.

[0041] In the technical solution of this application, the hot / cold value of the data is used to characterize the degree of hotness or coldness of the data. The higher the hot / cold value, the hotter the data; the lower the hot / cold value, the colder the data.

[0042] For each data access log, scan the hot and cold databases one by one to determine if the data has been accessed. After scanning all data access logs, proceed to step 103.

[0043] Newton's law of cooling describes how a hot object in a cold environment gradually cools down by transferring heat to its surroundings. The greater the object's temperature is above its environment, the faster its temperature drops. Newton's law of cooling is shown below:

[0044] in, express The temperature of an object at any time. Indicates the rate of temperature change. Indicates ambient temperature.

[0045] This invention calculates the current thermal value of the data based on Newton's law of cooling, taking into full account the influence of the surrounding thermal values ​​on the thermal value of the object, thereby accurately calculating the current thermal value of the data.

[0046] Optionally, the current hot / cold value of the data can be calculated using the following formula:

[0047] in, This represents the current hot / cold value of the data. The initial hot and cold values ​​of the data. The hot / cold value of the data at the previous moment. The average hot / cold value of all data from the previous moment. The rate of change of heating and cooling values. This is the gain for both hot and cold values.

[0048] It should be noted that when When the time data is accessed, the gain of the hot / cold value is Otherwise, it is 0, which can be represented as:

[0049] Among them, the rate of change of heating and cooling values and gain of heating and cooling values As hyperparameters, they can be tuned according to actual needs and different scenarios.

[0050] Step 103: Monitor the proportion of hot data in the hot database. If the proportion of hot data exceeds the proportion threshold, migrate the excess hot data to the cold database according to the current hot and cold values ​​of the hot data.

[0051] Optionally, step 103 may include: monitoring the proportion of hot data in the hot database; if the proportion of hot data exceeds a threshold, calculating the amount of hot data exceeding the threshold; sorting each hot data in descending order of its current hot / cold value, and migrating the hot data that is ranked lower and exceeds the threshold to the cold database. The proportion of hot data in the hot database can be monitored by a scheduled task (generally executed in the early morning). If the proportion of hot data exceeds a preset threshold (e.g., 20%, 30%, or 35%), the amount of hot data exceeding the threshold is calculated. In this step, the current hot / cold value needs to be calculated for each data point (i.e., hot data), and then the hot data is sorted in descending order of its current hot / cold value, migrating the hot data with lower current hot / cold values ​​and exceeding the threshold to the cold database, thus making these hot data points cold data.

[0052] It should be noted that, in order to avoid frequently migrating data from the hot database to the cold database, when calculating the amount of hot data exceeding the limit, the amount of hot data exceeding the limit can be appropriately increased, for example by 10%, 15%, 30% or 50%.

[0053] Based on the various embodiments described above, it can be seen that the present invention solves the technical problems of complex implementation logic and high computational overhead in the prior art by calculating the initial hot and cold values ​​of each data based on historical data access logs, storing each data in a hot database or a cold database according to the initial hot and cold values, migrating data to the hot database according to the current data access logs, and monitoring the data volume in the hot database. The present invention calculates the initial hot and cold values ​​of data using historical data access logs, making the data access time interval negatively correlated with the initial hot and cold values, and the data access frequency positively correlated with the initial hot and cold values. This accurately distinguishes between hot and cold data, enabling cold starts, and the implementation logic is simple with low computational overhead. Furthermore, the present invention migrates data and calculates the current hot and cold values ​​using data access logs, and migrates hot data with reduced hot and cold values ​​that exceed the threshold to the cold database based on the hot and cold value sorting results when the proportion of hot data exceeds the threshold. This not only accurately distinguishes between hot and cold data, but also simplifies the implementation logic, reduces computational overhead, and accurately controls storage overhead, significantly reducing system storage costs, thereby meeting the needs of hot and cold data governance.

[0054] Figure 3 This is a schematic diagram of the main flow of a data processing method according to a possible embodiment of the present invention. As another embodiment of the present invention, such as... Figure 3 As shown, the data processing method may include: Step 301: Calculate the reach interval characteristics and reach frequency characteristics of each data based on the historical data reach logs.

[0055] Steps 301-303 constitute the cold start phase. In step 301, historical data reach logs are first acquired, which can be at the daily or hourly granularity; this embodiment of the invention does not impose any limitation on this. Then, the reach interval characteristics and reach frequency characteristics of each data item are calculated based on the historical data reach logs. The historical data reach logs may include the device identifier, device type, and reach time of the historical reach; the data corresponding to the device identifier may be user profile data, specifically including multi-dimensional user profile data such as device identifier, user gender, user age, user behavior, and user preferences.

[0056] Optionally, after obtaining historical data access logs, the historical data access logs are preprocessed, mainly including several steps such as data cleaning, data integration, and data normalization. Among them, data cleaning includes several steps such as missing value handling, discrete point handling, and data noise handling.

[0057] Consider the time it takes for the device identifier to reach ( ), frequency of contact ( ) and reach ( Based on the reach characteristics determined by the reach characteristics, this embodiment of the invention calculates the reach interval characteristics and reach frequency characteristics, and calculates the initial cold and hot values ​​of the data based on the reach interval characteristics and reach frequency characteristics.

[0058] Optionally, the reach interval characteristics of each data are calculated based on historical data reach logs, including: determining the historical reach time of each device identifier based on historical data reach logs; calculating the historical reach time length interval of each device based on the historical reach time of each device identifier; and for each device identifier, summing the reciprocals of the historical reach time length intervals corresponding to the device identifier to obtain the reach interval characteristics of the data corresponding to the device identifier.

[0059] Recently accessed device identifiers are more likely to be accessed again, resulting in a higher "hot / cold" value. However, the longer the time after an identifier is accessed and left unused, the lower its "hot / cold" value becomes, and the less likely it is to be accessed again. If the time between each access or modification of a device identifier is... , …, These time points are relative to the current time. The duration is: , ,…, Assuming the above time intervals are , ,…, ,Right now = , = ,…, = The reach interval feature can be calculated using the following formula. :

[0060] Optionally, the reach frequency characteristics of each data are calculated based on historical data reach logs, including: determining the reach frequency of each device identifier in each sequence time period based on historical data reach logs; for each device identifier, multiplying each sequence time period by its reach frequency and summing the results to obtain the reach frequency characteristics of the data corresponding to the device identifier.

[0061] The higher the frequency of data access, the more frequently the device identifier is accessed, which can be understood as a greater value for its "hot" or "cold" value. This is especially true for recent access frequencies closer to the current time, where the gain in the "hot" or "cold" value is greater. Assuming the device identifier is within a fixed-length sequence time period... , ,…, The frequency of being reached is , … The reach frequency characteristics can be calculated using the following formula. :

[0062] in, , ,…, It can be configured as needed, for example Indicates the most recent day, This indicates the last three days. Indicates the past week, This indicates the past month, etc.

[0063] Step 302: Calculate the initial hot / cold value of each data based on the reach interval characteristics and reach frequency characteristics of each data.

[0064] This invention uses the reach interval and reach frequency characteristics of integrated data to calculate the initial problem, which can accurately calculate the initial cold and hot values ​​of each data point and achieve cold start.

[0065] Optionally, step 302 may include: for each data point, performing a weighted summation of the reach interval characteristic and reach frequency characteristic of the data to obtain the initial hot / cold value of the data. Specifically, the initial hot / cold value of the data can be calculated using the following formula. :

[0066] in, Weights for reaching interval features, Weights of reach frequency features.

[0067] Data administrators can configure the weights of the reach interval feature and the reach frequency feature as needed.

[0068] Step 303: Based on the initial hot and cold values ​​of the data and the preset hot and cold value thresholds, the data is divided into hot data and cold data, and stored in a hot database or a cold database respectively.

[0069] After calculating the initial hot / cold values ​​for each data point, they are sorted according to their values, and then the data is processed based on a preset hot / cold value threshold. The data is divided into hot and cold data, and then stored separately in a hot database or a cold database from the data warehouse, thus achieving the separation of hot and cold data. It should be noted that user profile data in the hot and cold databases is stored by device identifier, facilitating the retrieval of user profile data by device identifier.

[0070] Data administrators can manually set hot and cold value thresholds. For example, based on Pareto's Law (the 80 / 20 rule), the threshold for hot and cold values ​​is set to 20% of the total database capacity.

[0071] Step 304: Write the data from the hot database to the Redis cache, and write the data from the cold database to the HBase database.

[0072] like Figure 4As shown, after storing hot and cold data in the cold and hot databases in the data warehouse respectively, the hot data is flushed to the Redis cache and the cold data is flushed to the HBase database. When data is requested (such as when providing data services to external parties), the system first searches for the hot data in the Redis cache. If the data is found, it is returned directly. If the data is not found, the system accesses the HBase database. If the data is found, the data is returned directly. Otherwise, the data is recorded as a new identifier to satisfy the data request service. At the same time, user touch information is embedded in the front end, and all data touch behaviors are collected and aggregated into a daily data touch log.

[0073] Step 305: Migrate the data to the hot database based on the current data access log, and calculate the current hot / cold value of the data.

[0074] Step 306: Monitor the proportion of thermal data. If the proportion of thermal data exceeds the proportion threshold, calculate the amount of thermal data exceeding the limit.

[0075] Step 307: Sort the hot data in descending order of the current hot and cold values, and migrate the hot data that are ranked lower and exceed the limit to the cold data database.

[0076] Steps 305-307 are similar to steps 102-103, and will not be repeated here.

[0077] Furthermore, the specific implementation details of the data processing method in one of the reference embodiments of the present invention have been described in detail in the data processing method described above, so the details will not be repeated here.

[0078] To improve the management efficiency of hot and cold data, this embodiment of the invention monitors the total amount of hot and cold data and the proportion of hot data through a timed task (generally executed in the early morning), and processes the data based on the monitoring results, thereby meeting the needs of hot and cold data governance.

[0079] like Figure 5 As shown, the total amount of hot and cold data is first calculated. If the total amount of data exceeds the capacity threshold, a capacity alarm is issued to the data administrator, who then decides whether to perform data compression and transfer. The specific process of compression and transfer is as follows: calculate the amount of data exceeding the limit, sort the hot and cold data based on the current hot and cold values, and then filter out the data with lower hot and cold values ​​that exceed the limit according to the sorting. These data are then compressed and transferred. Commonly used compression algorithms include GZIP and LZO.

[0080] If the total data volume does not exceed the capacity threshold, the proportion of hot data is calculated. If the proportion of hot data is greater than or equal to the maximum proportion threshold, it indicates that the hot data has exceeded the limit. In this case, the amount of hot data to be migrated is calculated, and then the hot data are sorted in descending order of their current hot and cold values. Finally, the hot data that is ranked lower and exceeds the limit is migrated to the cold database. Similarly, if the proportion of hot data is less than the minimum proportion threshold, it indicates that there is too little hot data. In this case, the amount of cold data to be migrated is calculated, and then the cold data are sorted in descending order of their current hot and cold values. Finally, the cold data that is ranked higher is migrated to the hot database.

[0081] Furthermore, the specific implementation details of the data processing method in one of the reference embodiments of the present invention have been described in detail in the data processing method described above, so the details will not be repeated here.

[0082] Figure 6 This is a schematic diagram of the main modules of a data processing apparatus according to an embodiment of the present invention. Figure 6 As shown, the data processing device 600 includes a cold start module 601, a calculation module 602, and a monitoring module 603. The cold start module 601 calculates the initial hot / cold values ​​of each data point based on historical data access logs, and stores each data point in a hot database or a cold database based on these initial hot / cold values. The data access time interval is negatively correlated with the initial hot / cold value, while the data access frequency is positively correlated with the initial hot / cold value. The calculation module 602 migrates data to the hot database based on the current data access logs and calculates the current hot / cold value of the data. The monitoring module 603 monitors the proportion of hot data in the hot database; if the proportion of hot data exceeds a threshold, the excess hot data is migrated to the cold database.

[0083] Optionally, the computing module 602 is further configured to: Extract device identifiers from data access logs; If the data corresponding to the device identifier is stored in a thermal database, then calculate the current hot or cold value of the data. If the data corresponding to the device identifier is stored in a cold database, then the data is migrated to a hot database, and the current hot / cold value of the data is calculated. If the device identifier is a new device identifier, then the data corresponding to the device identifier is written into the thermal database, and the initial cold and hot values ​​of the data are calculated.

[0084] Optionally, the computing module 602 is further configured to: The current hot / cold value of the data is calculated using the following formula:

[0085] in, This represents the current hot / cold value of the data. The initial hot and cold values ​​of the data. The hot / cold value of the data at the previous moment. The average hot / cold value of all data from the previous moment. The rate of change of heating and cooling values. This is the gain for both hot and cold values.

[0086] Optionally, the monitoring module 603 is further configured to: Monitor the proportion of heat data in the heat database. If the proportion of heat data exceeds the proportion threshold, calculate the amount of heat data exceeding the limit. Sort the hot data in descending order of their current hot and cold values, and migrate the hot data that are ranked lower and exceed the limit to the cold data database.

[0087] Optionally, the cold start module 601 is further configured to: Calculate the reach interval characteristics and reach frequency characteristics of each data based on historical data reach logs; Based on the reach interval characteristics and reach frequency characteristics of each data point, the initial hot and cold values ​​of each data point are calculated; Based on the initial hot and cold values ​​of the data and the preset hot and cold value thresholds, the data is divided into hot data and cold data, and stored in a hot database or a cold database respectively.

[0088] Optionally, the cold start module 601 is further configured to: Determine the historical contact time for each device identifier based on historical data contact logs; Calculate the historical contact time interval for each device based on the historical contact time of each device identifier; For each device identifier, the reciprocal of each historical reach time interval corresponding to the device identifier is taken and summed to obtain the reach interval feature of the data corresponding to the device identifier.

[0089] Optionally, the cold start module 601 is further configured to: Determine the frequency of each device identifier's reach within each sequence time period based on historical data reach logs; For each device identifier, the reach frequency of each sequence time period is multiplied by its reach frequency and then summed to obtain the reach frequency feature of the data corresponding to the device identifier.

[0090] Optionally, the cold start module 601 is further configured to: For each data point, the reach interval feature and reach frequency feature of the data are weighted and summed to obtain the initial hot / cold value of the data.

[0091] Optionally, the monitoring module is also used for 603: Monitor the total amount of data in the cold database and the hot database. If the total amount of data exceeds the capacity threshold, calculate the amount of data exceeding the limit. Sort the data in descending order of their current hot / cold values, and compress and transfer the data that is lower in the sort and exceeds the limit.

[0092] Optionally, a flashing module is also included for: Data from the hot database is flushed to the Redis cache, and data from the cold database is flushed to the HBase database.

[0093] It should be noted that the specific implementation details of the data processing device described in this invention have been described in detail in the data processing method described above, so the details will not be repeated here.

[0094] Figure 7 An exemplary system architecture 700 is shown that can be applied to the data processing method or data processing apparatus of the present invention.

[0095] like Figure 7 As shown, system architecture 700 may include terminal devices 701, 702, and 703, a network 704, and a server 705. Network 704 serves as the medium for providing communication links between terminal devices 701, 702, and 703 and server 705. Network 704 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0096] Users can use terminal devices 701, 702, and 703 to interact with server 705 via network 704 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 701, 702, and 703, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0097] Terminal devices 701, 702, and 703 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0098] Server 705 can be a server providing various services, such as a backend management server supporting shopping websites browsed by users using terminal devices 701, 702, and 703 (this is just an example). The backend management server can analyze and process data such as received item information query requests, and then feed the processing results back to the terminal devices.

[0099] It should be noted that the data processing method provided in the embodiments of the present invention is generally executed by server 705, and correspondingly, the data processing device is generally set in server 705.

[0100] It should be understood that Figure 7 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0101] The following is for reference. Figure 8 It shows a schematic diagram of the structure of a computer system 800 suitable for implementing a terminal device of the present invention. Figure 8 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0102] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 802 or programs loaded from storage section 808 into random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the system 800. The CPU 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0103] The following components are connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 810 as needed so that computer programs read from it can be installed into storage section 808 as needed.

[0104] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by central processing unit (CPU) 801, it performs the functions defined above in the system of this invention.

[0105] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0106] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer programs according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0107] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor can be described as including a cold start module, a computing module, and a monitoring module. The names of these modules do not necessarily limit the functionality of the module itself.

[0108] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by the device, the device implements the following method: calculating the initial hot / cold values ​​of each data point based on historical data access logs; storing each data point to a hot database or a cold database respectively based on the initial hot / cold values; wherein the data access time interval is negatively correlated with the initial hot / cold values, and the data access frequency is positively correlated with the initial hot / cold values; migrating data to the hot database based on the current data access logs; monitoring the proportion of hot data in the hot database; if the proportion of hot data exceeds a proportion threshold, migrating the excess hot data to the cold database based on the current hot / cold values ​​of the hot data.

[0109] In another aspect, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the methods described in any of the above embodiments.

[0110] According to the technical solution of the present invention, by employing the method of calculating the initial hot and cold values ​​of each data based on historical data access logs, storing each data in a hot database or a cold database according to the initial hot and cold values, and migrating the data to the hot database according to the current data access logs and monitoring the data volume in the hot database, the technical problems of complex implementation logic and high computational overhead in the prior art are overcome. The present invention calculates the initial hot and cold values ​​of data through historical data access logs, making the data access time interval negatively correlated with the initial hot and cold values, and the data access frequency positively correlated with the initial hot and cold values. This allows for accurate differentiation between hot and cold data, enabling cold starts, and the implementation logic is simple with low computational overhead. Furthermore, the present invention migrates data and calculates the current hot and cold values ​​through data access logs, and when the proportion of hot data exceeds a threshold, migrates hot data with decreasing and exceeding limits to the cold database based on the hot and cold value sorting results. This not only accurately differentiates between hot and cold data, but also simplifies the implementation logic, reduces computational overhead, and allows for accurate control of storage overhead, significantly reducing system storage costs, thereby meeting the needs of hot and cold data governance.

[0111] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data processing method, characterized in that, include: The initial hot / cold value of each data point is calculated based on historical data access logs, and each data point is stored in a hot database or a cold database according to the initial hot / cold value of each data point; wherein, the data access time interval is negatively correlated with the initial hot / cold value of the data, and the data access frequency is positively correlated with the initial hot / cold value of the data. Migrate the data to the hot database based on the current data access logs; The proportion of hot data in the hot database is monitored. If the proportion of hot data exceeds the proportion threshold, the hot data exceeding the limit is migrated to the cold database according to the current hot and cold values ​​of the hot data. Calculate the initial hot / cold value of each data point based on historical data access logs, and store each data point into a hot database or a cold database according to the initial hot / cold value, including: Calculate the reach interval characteristics and reach frequency characteristics of each data based on historical data reach logs; Based on the reach interval characteristics and reach frequency characteristics of each data point, the initial hot and cold values ​​of each data point are calculated; Based on the initial hot and cold values ​​of the data and the preset hot and cold value thresholds, the data is divided into hot data and cold data, and stored in a hot database or a cold database respectively.

2. The method according to claim 1, characterized in that, Migrate data to the hot database based on the current data access logs, including: Extract the device identifier from the current data access logs; If the data corresponding to the device identifier is stored in a thermal database, then calculate the current hot or cold value of the data. If the data corresponding to the device identifier is stored in a cold database, then the data is migrated to a hot database, and the current hot / cold value of the data is calculated. If the device identifier is a new device identifier, then the data corresponding to the device identifier is written into the thermal database, and the initial cold and hot values ​​of the data are calculated.

3. The method according to claim 2, characterized in that, The current hot / cold value of the data is calculated using the following formula: in, This represents the current hot / cold value of the data. The initial hot and cold values ​​of the data. The hot / cold value of the data at the previous moment. The average hot / cold value of all data from the previous moment. The rate of change of heating and cooling values. This is the gain for both hot and cold values.

4. The method according to claim 1, characterized in that, Monitor the proportion of hot data in the hot database. If the proportion of hot data exceeds a threshold, migrate the excess hot data to the cold database based on the current temperature values ​​of the hot data. This includes: Monitor the proportion of heat data in the heat database. If the proportion of heat data exceeds the proportion threshold, calculate the amount of heat data exceeding the limit. Sort the hot data in descending order of their current hot and cold values, and migrate the hot data that are ranked lower and exceed the limit to the cold data database.

5. The method according to claim 1, characterized in that, The reach interval characteristics of each data point are calculated based on historical data reach logs, including: Determine the historical contact time for each device identifier based on historical data contact logs; Calculate the historical contact time interval for each device based on the historical contact time of each device identifier; For each device identifier, the reciprocal of each historical reach time interval corresponding to the device identifier is taken and summed to obtain the reach interval feature of the data corresponding to the device identifier.

6. The method according to claim 1, characterized in that, The frequency characteristics of each data point are calculated based on historical data access logs, including: Determine the frequency of each device identifier's reach within each sequence time period based on historical data reach logs; For each device identifier, the reach frequency of each sequence time period is multiplied by its reach frequency and then summed to obtain the reach frequency feature of the data corresponding to the device identifier.

7. The method according to claim 1, characterized in that, Based on the reach interval characteristics and reach frequency characteristics of each data point, the initial hot / cold values ​​of each data point are calculated, including: For each data point, the reach interval feature and reach frequency feature of the data are weighted and summed to obtain the initial hot / cold value of the data.

8. The method according to claim 1, characterized in that, Also includes: Monitor the total amount of data in the cold database and the hot database. If the total amount of data exceeds the capacity threshold, calculate the amount of data exceeding the limit. Sort the data in descending order of their current hot / cold values, and compress and transfer the data that is lower in the sort and exceeds the limit.

9. The method according to claim 1, characterized in that, Also includes: Data from the hot database is flushed to the Redis cache, and data from the cold database is flushed to the HBase database.

10. A data processing apparatus, characterized in that, include: The cold start module is used to calculate the initial hot and cold values ​​of each data based on historical data access logs, and store each data into a hot database or a cold database respectively based on the initial hot and cold values ​​of each data; wherein, the data access time interval is negatively correlated with the initial hot and cold values ​​of the data, and the data access frequency is positively correlated with the initial hot and cold values ​​of the data. The calculation module is used to migrate data to the hot database based on the current data access logs; The monitoring module is used to monitor the proportion of hot data in the hot database. If the proportion of hot data exceeds the proportion threshold, the excess hot data will be migrated to the cold database. The cold start module is also used for: Calculate the reach interval characteristics and reach frequency characteristics of each data based on historical data reach logs; Based on the reach interval characteristics and reach frequency characteristics of each data point, the initial hot and cold values ​​of each data point are calculated; Based on the initial hot and cold values ​​of the data and the preset hot and cold value thresholds, the data is divided into hot data and cold data, and stored in a hot database or a cold database respectively.

11. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-9.

12. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-9.

Images