Data processing method and device, equipment and storage medium

Abstract

A data processing method, device and equipment and storage medium, relating to the technical field of computer. The method comprises: determining target data to be accessed in a storage space; updating a first frequency parameter of the target data and a second frequency parameter of the target data, the first frequency parameter being used to represent the number of non-repeated data accessed in the latest two access intervals of the target data, and the second frequency parameter being used to represent the number of accesses of the target data in the latest n accesses; in the case that the target data belongs to a retention cold data set, if the first frequency parameter of the target data is less than or equal to a first threshold value, and the second frequency parameter of the target data is greater than or equal to a second threshold value, the target data is transferred from the retention cold data set to a hot data set. The present application is applicable to multimedia data storage and push scenarios, and can also avoid frequent data exchange processes in the case of some sudden batch data operations and periodic data access, which helps to reduce device processing overhead.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, device, and storage medium. Background Technology

[0002] In the process of data processing and statistics, data needs to be processed.

[0003] In related technologies, the LRU (Least Recently Used) algorithm is used to process data. The LRU algorithm assumes that if a piece of data has been accessed once, it will likely be accessed again within a certain period of time. If a piece of data has not been accessed for a long time, it is less likely to be accessed again. The core of the LRU algorithm is to exchange the data that has not been accessed for the longest time.

[0004] However, for situations involving sudden batch data operations or periodic data access, the LRU algorithm can lead to frequent data exchange processes, increasing the number of unnecessary data exchanges. Summary of the Invention

[0005] This application provides a data processing method, apparatus, device, and storage medium that can avoid frequent data exchange processes, especially for situations involving sudden batch data operations or periodic data access. The technical solution is as follows:

[0006] According to one aspect of the embodiments of this application, a data processing method is provided, the method comprising:

[0007] Identify the target data to be accessed in the storage space, which includes cache space and non-cache space, and the cache space includes a hot data set and a retained cold data set;

[0008] If the target data belongs to the retained cold data set, update the first frequency parameter and the second frequency parameter of the target data. The first frequency parameter of the target data is used to represent the number of non-repeating data accessed within the interval between the two most recent accesses of the target data, and the second frequency parameter of the target data is used to represent the number of times the target data was accessed in the last n accesses, where n is a positive integer.

[0009] If the first frequency parameter of the target data is less than or equal to the first threshold, and the second frequency parameter of the target data is greater than or equal to the second threshold, then the target data is transferred from the retained cold data set to the hot data set.

[0010] According to another aspect of the embodiments of this application, a data processing apparatus is provided, the apparatus comprising:

[0011] The first determining module is used to determine the target data to be accessed in the storage space, the storage space including cache space and non-cache space, the cache space including a hot data set and a retained cold data set;

[0012] The parameter update module is used to update the first frequency parameter and the second frequency parameter of the target data when the target data belongs to the retained cold data set. The first frequency parameter of the target data is used to represent the number of non-repeating data accessed within the interval between the two most recent accesses of the target data, and the second frequency parameter of the target data is used to represent the number of times the target data was accessed in the last n accesses, where n is a positive integer.

[0013] The first transfer module is used to transfer the target data from the retained cold data set to the hot data set when the first frequency parameter of the target data is less than or equal to a first threshold and the second frequency parameter of the target data is greater than or equal to a second threshold.

[0014] According to another aspect of the embodiments of this application, a computer device is provided, the computer device comprising: a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, wherein the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the above-described data processing method.

[0015] According to another aspect of this application, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set, or instruction set is stored therein, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the above-described data processing method.

[0016] According to another aspect of the embodiments of this application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium, and a processor reading from the computer-readable storage medium and executing the computer instructions to implement the above-described data processing method.

[0017] The beneficial effects of the technical solutions provided in this application include at least the following:

[0018] By identifying the target data to be accessed and updating its first and second frequency parameters, where the first frequency parameter represents the number of non-repeating data accessed within the interval between the two most recent accesses of the target data, and the second frequency parameter represents the number of times the target data has been accessed in the last n accesses, the target data is transferred from the cold data set in the cache space to the hot data set when both frequency parameters simultaneously meet the threshold conditions. This raises the threshold for moving target data into the hot data set, ensuring that the data stored in the hot data set is the most frequently accessed data in the recent access process, reducing the number of data exchanges between the cold data set and the hot data set, and helping to reduce device processing overhead. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a structural block diagram of a computer system provided in an exemplary embodiment of this application;

[0021] Figure 2 This is a flowchart of a data processing method provided in an exemplary embodiment of this application;

[0022] Figure 3 This is a schematic diagram illustrating the relationship between various storage spaces provided in an exemplary embodiment of this application;

[0023] Figure 4 This is a schematic diagram of a first frequency parameter provided in an exemplary embodiment of this application;

[0024] Figure 5 This is a schematic diagram of a second frequency parameter provided in an exemplary embodiment of this application;

[0025] Figure 6 This is a schematic diagram of the target data update frequency parameter provided in an exemplary embodiment of this application;

[0026] Figure 7 This is a block diagram of a data processing apparatus provided in an exemplary embodiment of this application;

[0027] Figure 8 This is a block diagram of a data processing apparatus provided in another exemplary embodiment of this application;

[0028] Figure 9This is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0030] First, let's introduce the terms used in the embodiments of this application:

[0031] Cache eviction: Since the cache space is limited, when the data stored in the cache space reaches the cache space size and new data needs to be written to the cache space, the data stored in the cache space needs to be evicted and the new data written to the cache space.

[0032] Hot data refers to data that needs to be frequently accessed by computing nodes during the computation process. Hot data typically accounts for about 20% of the total data. Of course, this percentage can be set as needed, and this application does not limit it.

[0033] Streaming data, also known as data streams, refers to a collection of dynamic data with no upper limit on its volume. The dynamic data within this collection is distributed in chronological order and has a time-sensitive nature; its value decreases over time. Operations performed on streaming data are called streaming operations, and the time-sensitive nature of streaming data requires streaming operations to have a real-time, big data computing model.

[0034] Real-time computing refers to data processing on massive datasets, with feedback times typically required to be in the order of seconds. Real-time computing mainly consists of two parts: real-time data ingestion and real-time data processing. Currently, the main streaming frameworks used for real-time computing include Storm, Spark Streaming, and Flink.

[0035] Figure 1 A structural block diagram of a computer system provided in an exemplary embodiment of this application is given. The computer system may include: a terminal device 10 and a server 20.

[0036] Terminal device 10 has data transmission capabilities and runs a client application for a target application. This target application can be a video application, news application, social application, or other applications with push data functionality. The data can be multimedia data (such as video, audio, and images) or other types of data; this application does not limit the specific types of data. Terminal device 10 includes, but is not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, and in-vehicle terminals. Terminal device 10 can upload real-time data generated by the target application to server 20.

[0037] Server 20 provides backend support for terminal device 10; for example, server 20 can be a backend server for the target application. Server 20 has the function of receiving and statistically analyzing data, used to receive data uploaded by terminal device 10 in real time and perform statistical analysis on the data. Server 20 can be a standalone server device, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud computing, cloud functions, cloud storage, network services, cloud communication, domain name services, security services, and big data and artificial intelligence platforms. In some embodiments, during the data analysis and processing process by server 20, the data is stored in a cache space to implement the following data processing method.

[0038] The system architecture and business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of the implementation environment and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0039] Figure 2 This is a flowchart illustrating a data processing method provided in an exemplary embodiment of this application. The subject executing this method is a computer device with storage and processing capabilities. Figure 2 As shown, the method may include the following steps (210-230):

[0040] Step 210: Determine the target data to be accessed in the storage space. The storage space includes cache space and non-cache space. The cache space includes a hot data set and a retained cold data set.

[0041] Please refer to Figure 3 This diagram illustrates the relationships between various storage spaces. The cache space stores data needed during statistical analysis. In some embodiments, the cache space is a fixed-size data storage space. Data in the cache space is read quickly and is used to temporarily store raw or intermediate data during data processing. After processing, the data in the cache space is discarded or stored in the non-cache space. In some embodiments, each piece of data is stored only once in the cache space, with only one storage location for each data point, avoiding unnecessary storage space usage. In some embodiments, during streaming operations, the cache space is a fixed-size storage space created by the Flink framework. The non-cache space can store data for extended periods, but data storage in the non-cache space is slower.

[0042] The target data to be accessed is the raw or intermediate data needed during the data statistics process. In some embodiments, when a computing node needs to access certain data during the statistical analysis of data uploaded by a terminal device by the server, this data is the target data to be accessed.

[0043] The cache space includes a hot data set and a cold data set. The hot data set is the cache area in the cache space used to store data that is accessed frequently. In some embodiments, the data stored in the hot data set is called hot data. The retained cold data set refers to the cache area in the cache space used to store data other than hot data. In some embodiments, the data stored in the hot data set may change as the data access process proceeds. In some embodiments, the data in the hot data set may be swapped with the data in the retained cold data set to ensure that the data stored in the hot data set is always the data that was accessed most frequently in the most recent access process.

[0044] In some embodiments, the hot data set and the retained cold data set use a data stack structure to store data, while the non-cached space uses a linked list structure to store data. Optionally, the non-cached space can also be referred to as the non-retained cold data set. In some embodiments, the retained cold data set and the non-retained cold data set are collectively referred to as the cold data set, and the data stored in the cold data set is called cold data.

[0045] In some embodiments, the capacity of the hot dataset is smaller than the capacity of the retained cold dataset. The capacity of the hot dataset and the capacity of the retained cold dataset are determined based on the characteristics of the data that needs to be accessed during data processing. For example, if the data that needs to be accessed frequently during data statistical analysis constitutes a large proportion of the total data, then the capacity of the hot dataset should be increased accordingly. The capacity of the hot dataset and the cold dataset can be set according to actual circumstances and are not limited here.

[0046] Step 220: Update the first frequency parameter and the second frequency parameter of the target data. The first frequency parameter of the target data is used to represent the number of non-repeating data accessed in the interval between the two most recent accesses of the target data. The second frequency parameter of the target data is used to represent the number of times the target data has been accessed in the last n accesses, where n is a positive integer.

[0047] Please refer to Figure 4The diagram illustrates the first frequency parameter, which reflects, to some extent, the frequency of the target data's occurrence during recent data access. A higher first frequency parameter indicates that more other data was accessed between the two most recent accesses, meaning a longer interval and a lower access frequency. Conversely, a lower first frequency parameter indicates that fewer other data was accessed between the two most recent accesses, meaning a shorter interval and a higher access frequency. Optionally, the first frequency parameter can be referred to as IRR (Inter-Reference Recency).

[0048] Please refer to Figure 5 The diagram illustrates the second frequency parameter. A larger second frequency parameter indicates that the target data has been accessed more frequently in the last n accesses, meaning the target data appears more frequently in the recent data access process; conversely, a smaller second frequency parameter indicates that the target data has been accessed less frequently in the last n accesses, meaning the target data appears less frequently in the recent data access process. In some embodiments, during real-time statistical analysis of the data generated by the target application, the average amount of data generated by the target application can be calculated based on experience, and the value of n can be equal to the calculated average. n can be set according to the actual application scenario; for example, n is 10 or 15, etc., and the value of n is not limited here. Optionally, the second frequency parameter can be called Freq.

[0049] In the storage space, for the purpose of saving storage space and facilitating data access, a piece of data has only one storage location in the storage space. Therefore, the first frequency parameter only records the number of times non-repeating data appears.

[0050] Updating the first frequency parameter and the second frequency parameter of the target data means performing update operations on the first frequency parameter and the second frequency parameter of the target data respectively to obtain the updated first frequency parameter and second frequency parameter of the target data.

[0051] In some embodiments, updating the first frequency parameter and the second frequency parameter of the target data includes: assigning a third frequency parameter of the target data to the first frequency parameter of the target data, wherein the third frequency parameter of the target data represents the number of non-repeating data accessed since the most recent access to the target data; and incrementing the current value of the second frequency parameter of the target data by 1 to obtain the updated second frequency parameter of the target data. Optionally, the third frequency parameter may be referred to as Recency.

[0052] The third frequency parameter of the target data is used to record the number of non-duplicate data accessed since the most recent access. After updating the first frequency parameter of the target data, the third frequency parameter is updated and set to 0. Please refer to [reference needed]. Figure 6 This illustrates the process of updating the target data frequency parameters, where the frequency parameters include a first frequency parameter, a second frequency parameter, and a third frequency parameter. When the currently accessed target data is determined to be data 1, before data 1 is determined to be accessed, the value of the first frequency parameter for data 1 is 5, the value of the second frequency parameter is 2, and the value of the third frequency parameter is 1. After the target data is determined to be the next data to be accessed, the current value of the updated first frequency parameter for the target data is 1, the current value of the second frequency parameter is 3, and the current value of the third frequency parameter is 0.

[0053] Step 230: If the target data belongs to the retained cold data set, and the first frequency parameter of the target data is less than or equal to the first threshold, and the second frequency parameter of the target data is greater than or equal to the second threshold, then the target data is transferred from the retained cold data set to the hot data set.

[0054] In some embodiments, the first threshold is determined by the top x percent of all data in the storage space sorted in ascending order according to a first frequency parameter, where x is a positive integer. For example, if the first frequency parameters of the data in the storage space are arranged in ascending order as (1, 2, 2, 3, 4, 5, 5, 5, 7, 8), and x = 20%, then the first threshold is 2. In some embodiments, the second threshold is determined based on a percentage of the value n. For example, if n = 100 and y = 5%, then the value of the second threshold is 100 * 5% = 5. In some embodiments, the first threshold and the second threshold can be fixed values ​​set empirically.

[0055] If the target data belongs to the cold data set, it means that the target data to be accessed has been hit in the cache space. Therefore, it is not necessary to remove the data from the cache space. It is only necessary to determine whether to replace the data between the hot data set and the cold data set in the cache space.

[0056] If the first frequency parameter of the target data is less than or equal to the first threshold, and the second frequency parameter of the target data is greater than or equal to the second threshold, it indicates that the target data has a high access frequency in the recent access process and needs to be transferred to the hot data set for storage. If the first frequency parameter of the target data is greater than or equal to the first threshold, or the second frequency parameter of the target data is less than or equal to the second threshold, it indicates that the target data does not meet the conditions for storage in the hot data set and does not need to be transferred to the hot data set for storage. That is, the target data is still stored in the retained cold database set.

[0057] In summary, by identifying the target data to be accessed in the storage space, updating the first and second frequency parameters of the target data, and transferring the target data whose first and second frequency parameters respectively satisfy a first threshold and a second threshold to the hot data set for storage, the goal of timely updating the hot data set as the data access process progresses is achieved. In addition, by simultaneously limiting the frequency parameters of the target data through the first and second thresholds, the threshold for moving the target data into the hot data set is increased, ensuring that the data stored in the hot data set is the most frequently accessed data in the recent access process, and reducing the number of data swaps between the cold and hot data sets.

[0058] The process of moving target data from a retained cold dataset to a hot dataset will be described below through the following two examples.

[0059] In some embodiments, transferring target data from a retained cold data set to a hot data set includes: determining a first frequency parameter for each hot data in the hot data set, the first frequency parameter representing the number of non-repeating data accessed within the interval between the two most recent accesses of the hot data; selecting hot data from the hot data set whose first frequency parameter meets a first condition as hot data; removing the target data from the retained cold data set and adding it to the hot data set; and removing the target hot data from the hot data set and adding it to the retained cold data set.

[0060] The first frequency parameter of a certain hot data point represents the number of non-repeating data points accessed within the interval between the two most recent accesses of that hot data point. If the target data to be accessed is stored in a cold data set, the first frequency parameter corresponding to each hot data point in the hot data set remains unchanged.

[0061] In some embodiments, selecting thermal data whose first frequency parameter meets a first condition as target thermal data includes: selecting the data with the largest first frequency parameter from the thermal data set and using that thermal data as the target thermal data. In some embodiments, by comparing the first frequency parameters of each thermal data in the thermal data set, the thermal data with the largest first frequency parameter is selected as the target thermal data.

[0062] This method ensures that at least one piece of hot data in the hot data set satisfies the first condition and can be used as the target hot data.

[0063] In some embodiments, selecting hot data whose first frequency parameter meets a first condition as target hot data includes: when the hot data set uses a data stack to store the hot data, selecting *a* hot data points stored at the bottom of the data stack of the hot data set, where *a* is a positive integer, as candidate target hot data, and then randomly selecting one of the candidate target hot data points as the target hot data. Optionally, the hot data stored at the lowest end of the hot data set is selected as the target hot data. The data stack of the hot data set writes the hot data to the hot data set from the top of the stack; therefore, the first frequency parameter of the hot data stored at the bottom of the data stack of the hot data set is relatively large.

[0064] This method can quickly find hot data that meets the first condition, reducing the access latency of the target data.

[0065] In some embodiments, selecting thermal data whose first frequency parameter meets a first condition as target thermal data includes: comparing the first frequency parameter of the target data with the first frequency parameter of any thermal data in the thermal data set, and selecting the first thermal data whose first frequency parameter is less than or equal to the first frequency parameter of the target data as target thermal data; optionally, when the thermal data set stores thermal data in order of magnitude of the first frequency parameter, determining thermal data whose first frequency parameter is greater than or equal to the first frequency parameter of the target data by a binary search method, and selecting such thermal data as target thermal data.

[0066] This method can be used to maintain the state of the most frequently accessed data in the hot dataset.

[0067] In some embodiments, both the hot data set and the retained cold data set in the cache space store data in a data stack manner. The hot data stored at the bottom of the data stack of the hot data set has the largest first frequency parameter. When performing operations such as removing target data from the retained cold data set and adding it to the hot data set, and removing target hot data from the hot data set and adding it to the retained cold data set, it is necessary to move the target data to the top of the data stack of the hot data set, move the target hot data out from the bottom of the data stack of the hot data set, and move the target hot data to the top of the data stack of the retained cold data set to complete the process of transferring the target data from the retained cold data set to the hot data set.

[0068] In some embodiments, transferring target data from a retained cold data set to a hot data set includes: determining a second frequency parameter for each hot data in the hot data set, the second frequency parameter representing the number of times the hot data has been accessed in the most recent n accesses; selecting hot data from the hot data set whose second frequency parameter meets a second condition as target hot data; removing the target data from the retained cold data set and adding it to the hot data set; and removing the target hot data from the hot data set and adding it to the retained cold data set.

[0069] The larger the second frequency parameter of a hot data point, the more times the hot data point has been accessed in the most recent n accesses, meaning that the hot data point has appeared more frequently in the recent data access process; conversely, the smaller the second frequency parameter of a hot data point, the fewer times the hot data point has been accessed in the most recent n accesses, meaning that the hot data point has appeared less frequently in the recent data access process.

[0070] In some embodiments, selecting thermal data whose second frequency parameter meets the second condition as target thermal data includes: selecting the data with the smallest second frequency parameter from the thermal data set and using that thermal data as the target thermal data.

[0071] In some embodiments, selecting thermal data whose second frequency parameter meets the first condition as target thermal data includes: comparing the second frequency parameter of the target data with the second frequency parameter of any thermal data in the thermal data set, and selecting thermal data whose first second frequency parameter obtained by comparison is less than or equal to the second frequency parameter of the target data as target thermal data; optionally, when the thermal data set stores thermal data in order of magnitude of the second frequency parameter, the thermal data whose second frequency parameter is less than or equal to the second frequency parameter of the target data is determined by a binary search method, and the thermal data is selected as target thermal data.

[0072] In some embodiments, if a hot data point in the hot data set is the nth accessed target data point before the current target data is accessed, the third frequency parameter of that hot data point is decremented by 1. For example, if n = 6, the current target data is data 7, and data 2 is hot data, before the current target data becomes the data to be accessed, the target data accessed in chronological order are (2, 2, 1, 5, 2, 4), and the value of the second frequency parameter corresponding to data 2 is 3. When target data 7 becomes the data to be accessed, the n (n = 6) target data points accessed in chronological order become (2, 1, 5, 2, 4, 7), and at this time, the value of the second frequency parameter corresponding to data 2 becomes 2.

[0073] In some embodiments, after determining the data to be accessed in the storage space, the third frequency parameter of each hot data in the hot data set needs to be updated. The third frequency parameter of the hot data is used to characterize the number of non-repeating data accessed after the most recent access of the hot data. Optionally, the third frequency parameter can be called Recency. When updating the third frequency parameter of the hot data, the current value of the third frequency parameter of the hot data needs to be incremented by 1.

[0074] In data statistical analysis, there exists a situation where if a certain piece of data has a high access frequency in the last n accesses, the probability of that data being accessed again increases. Therefore, by removing the target hot data with the smallest second frequency parameter from the hot data set, we can maintain the hot data stored in the hot data set as the data with the highest access frequency in the last n accesses, thereby increasing the hit rate of the hot data set for the target data.

[0075] In other embodiments, when the target data is in a hot data set, the first frequency parameter, the second frequency parameter, and the third frequency parameter of the target data are updated. The update method is the same as the update method for the first frequency parameter, the second frequency parameter, and the third frequency parameter of the target data in the previous embodiment, and will not be repeated here. In this case, the target data continues to be stored in the hot data set. Optionally, if the hot data set stores hot data through a data stack structure, the target data is moved to the top of the data stack of the hot data set.

[0076] In some embodiments, after determining the target data to be accessed, it is necessary to update the first frequency parameter, the second frequency parameter, and the third frequency parameter of all data in the storage space. For the update method, please refer to the above-mentioned update method for the first frequency parameter, the second frequency parameter, and the third frequency parameter of hot data.

[0077] By updating the first and second frequency parameters of the target data, the storage location of the target data in the cache space can be adjusted in a timely manner, which helps to improve the data hit rate of the cache space.

[0078] The following two examples illustrate the data processing method when the target data is not in the cache space.

[0079] In some embodiments, after determining the target data to be accessed in the storage space, the method further includes: if the target data belongs to the non-cached space, determining the target retained data to be removed from the retained data contained in the retained cold data set; removing the target data from the non-cached space and adding it to the retained cold data set; and removing the target retained data from the retained data set and adding it to the non-cached space.

[0080] Non-cached space refers to the space in the storage space other than the cache space that can store data. Data read speed in the cache space is higher than data read speed in the non-cached space. In some real-time examples, when the target data is in the non-cached space, it is necessary to move the target data from the non-cached space to the cache space to enable access to the target data or reduce access latency. In some embodiments, when the target data is in the non-cached space, it is necessary to remove a piece of data from the cache space, store that data in the non-cached space, and write the target data into the cache space. In other embodiments, when a piece of target data is accessed for the first time, i.e., when the target data is not stored anywhere in the storage space, it is first necessary to create and initialize the first, second, and third frequency parameters of the target data. The current values ​​of the first and second frequency parameters are assigned to -1 or m (m is close to infinity or m exceeds the storage space capacity), and the value of the third frequency parameter is assigned to 0. Then, a piece of data is removed from the cache space, stored in the non-cached space, and the target data is written into the cache space.

[0081] After determining the target data to be accessed in the storage space, it is necessary to update the first frequency parameter, the second frequency parameter, and the third frequency parameter of the target data. For the update method of the first frequency parameter, the second frequency parameter, and the third frequency parameter of the target data, please refer to the above embodiment, which will not be repeated here.

[0082] When the target data belongs to the non-cached space, the target retained data that needs to be removed is determined from the retained data contained in the retained cold data set. That is, when evicting data stored in the cache space, data in the retained cold data set is selectively evicted first, rather than data in the hot data set. This is because, through the continuous maintenance and updating of the hot data set in steps 220 and 230, the hot data stored in the hot data set represents the group of data accessed most frequently during the recent data access process. During data periodicity or statistical analysis, the probability of the hot data in the hot data set being accessed again is higher. Selecting the target retained data to be removed first from the retained cold data set, without changing the data in the hot data set, helps improve the cache space's hit rate for accessed data and reduces the number of times data stored in the cache space is moved out of the cache space.

[0083] In some embodiments, determining target retained data to be removed from retained data included in a retained cold data set includes: determining a second frequency parameter for each retained cold data in the retained cold data set, wherein the second frequency parameter of the retained cold data is used to represent the number of times the retained cold data has been accessed in the most recent n accesses; and selecting retained data from the retained cold data set whose second frequency parameter meets a third condition as target retained data.

[0084] In some embodiments, selecting retention data whose second frequency parameter meets the third condition as target retention data includes: selecting target retention data with the smallest second frequency parameter from the set of retained cold data.

[0085] For the method of determining the second frequency parameter of each retained cold data in the retained cold data set, please refer to the method of determining the second frequency parameter of each hot data in the hot data set, which will not be repeated here.

[0086] In some embodiments, determining the target retained data to be removed from the retained data included in the retained cold data set includes: determining a third frequency parameter for each retained cold data in the retained cold data set, wherein the third frequency parameter of the retained cold data is used to represent the number of non-duplicate data accessed after the most recent access of the retained cold data; and selecting retained data from the retained cold data set whose third frequency parameter meets a fourth condition as the target retained data.

[0087] In some embodiments, selecting retention data from the retention cold data set whose third frequency parameter meets the fourth condition as target retention data includes: selecting target retention data from the retention cold data set whose third frequency parameter is less than the first frequency parameter of the target data.

[0088] In some embodiments, determining the third frequency parameter of each retained cold data in the retained cold data set includes: incrementing the current value of the third frequency parameter of each retained cold data in the retained cold data set by 1.

[0089] Because the third frequency parameter of the retained cold data represents the number of non-duplicate data accessed since the most recent access to the retained cold data, determining the target data to be accessed in the storage space is equivalent to increasing the number of non-duplicate data accessed since the most recent access to the retained cold data by one. In some embodiments, if the current target data is accessed for the first time after the most recent access to a certain retained cold data, the current value of the third frequency parameter in the retained cold data set needs to be incremented by 1. In other embodiments, if the current target data is not accessed for the first time after the most recent access to a certain retained cold data, the third frequency parameter of the retained cold data remains unchanged.

[0090] In some embodiments, after determining the target data to be accessed in the storage space, the third frequency parameter of each retained cold data in the retained cold data set is first determined, and the third frequency parameter of each retained cold data is compared with the first frequency parameter of the target data. A retained cold data whose third frequency parameter is less than or equal to the first frequency parameter of the target data is determined as the target retained data. The storage locations of the target retained data and the target data are interchanged. Optionally, the target data is written to the storage location of the target retained data, and the target retained data is written to the storage location of the target data. Optionally, the retained cold data set adopts a data stack structure, and the non-cached space adopts a linked list structure. The target data is moved to the top of the data stack of the retained cold data set, the target retained data is moved out from the bottom of the data stack, and the target retained data is stored in the linked list of the non-cached space.

[0091] The processes of removing target data from the non-cached space and adding it to the retained cold data set, and removing target retained data from the retained data set and adding it to the non-cached space, can be performed simultaneously or sequentially in actual operation, and are not limited here.

[0092] In some embodiments, the cache space is created and initialized by a streaming data statistics system. The cache space is used to read and parse log data in the message system. Access to target data in the cache space is performed through hash keys, which are used to improve the speed of determining the data storage location of the target data. The cache space stores both the target data and its corresponding hash key.

[0093] In some embodiments, after transferring the target data from the retained cold dataset to the hot dataset, the method further includes:

[0094] The system uses aggregation operators in streaming data statistics to read target data from the cache space; performs statistical analysis on the target data to obtain statistical analysis results; writes the statistical analysis results into the database and displays them in the form of reports.

[0095] In some embodiments, the above data processing method can be applied to data analysis within a streaming framework. Before data processing, Flink is used to read and parse Kafaka log data. A fixed-size cache space is created and initialized in Flink. Each piece of data in the Kafaka log is parsed by Flink, and each piece of data has a unique key-value pair. The MD5 (Message Digest Algorithm 5) function is used to convert the key-value pair corresponding to the data into a hash key-value pair, and the hash value corresponding to the data is stored in the cache. The hash key-value pair is used to improve the speed of finding the target data and reduce latency when accessing the target data. If the data stored in the cache space exceeds the cache's capacity, the above data processing method is used to transfer the data in the cache space, store the target data in the cache space, and access the target data through its hash key-value pair.

[0096] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0097] Please refer to Figure 7 This diagram illustrates a block diagram of a data processing apparatus according to an embodiment of this application. The apparatus has the function of implementing the above-described data processing method; this function can be implemented in hardware or by hardware executing corresponding software. The apparatus 700 can be a computer device or can be installed within a computer device. The apparatus 700 may include: a first determining module 710, a parameter updating module 720, and a first transferring module 730.

[0098] The first determining module 710 is used to determine the target data to be accessed in the storage space, the storage space including cache space and non-cache space, the cache space including a hot data set and a retained cold data set.

[0099] The parameter update module 720 is used to update the first frequency parameter and the second frequency parameter of the target data. The first frequency parameter of the target data is used to represent the number of non-repeating data accessed in the interval between the two most recent accesses of the target data, and the second frequency parameter of the target data is used to represent the number of times the target data has been accessed in the last n accesses, where n is a positive integer.

[0100] The first transfer module 730 is configured to transfer the target data from the retained cold data set to the hot data set if the first frequency parameter of the target data is less than or equal to a first threshold and the second frequency parameter of the target data is greater than or equal to a second threshold when the target data belongs to the retained cold data set.

[0101] In some embodiments, the first transfer module 730 is configured to:

[0102] A first frequency parameter is determined for each hot data in the hot data set, the first frequency parameter representing the number of non-repeating data accessed within the interval between the two most recent accesses of the hot data; hot data whose first frequency parameter meets a first condition is selected from the hot data set as target hot data; the target data is removed from the retained cold data set and added to the hot data set; and the target hot data is removed from the hot data set and added to the retained cold data set.

[0103] In some embodiments, the first transfer module 730 is configured to:

[0104] A second frequency parameter is determined for each hot data in the hot data set, the second frequency parameter representing the number of times the hot data has been accessed in the most recent n accesses; hot data whose second frequency parameter meets a second condition is selected from the hot data set as target hot data; the target data is removed from the retained cold data set and added to the hot data set; and the target hot data is removed from the hot data set and added to the retained cold data set.

[0105] In some embodiments, the parameter update module 720 is used for:

[0106] Assign the third frequency parameter of the target data to the first frequency parameter of the target data. The third frequency parameter of the target data is used to represent the number of non-duplicate data accessed after the most recent access of the target data. Increment the current value of the second frequency parameter of the target data by 1 to obtain the updated second frequency parameter of the target data.

[0107] In some embodiments, such as Figure 8 As shown, the device 700 further includes: a second determining module 740 and a second transferring module 750.

[0108] The second determining module 740 is used to determine the target retained data that needs to be removed from the retained data included in the retained cold data set when the target data belongs to the non-cache space.

[0109] The second transfer module 750 is used to remove the target data from the non-cached space and add it to the retained cold data set; and to remove the target retained data from the retained data set and add it to the non-cached space.

[0110] In some embodiments, the second transfer module 750 is configured to:

[0111] A second frequency parameter is determined for each piece of retained cold data in the retained cold data set, wherein the second frequency parameter of the retained cold data is used to represent the number of times the retained cold data has been accessed in the most recent n accesses; from the retained cold data set, retained data whose second frequency parameter meets a third condition is selected as the target retained data.

[0112] In some embodiments, the second transfer module 750 is configured to:

[0113] A third frequency parameter is determined for each retained cold data in the retained cold data set. The third frequency parameter of the retained cold data is used to represent the number of non-duplicate data accessed after the most recent access of the retained cold data. From the retained cold data set, retained data whose third frequency parameter meets the fourth condition and is less than the first frequency parameter of the target data are selected as the target retained data.

[0114] In some embodiments, such as Figure 8 As shown, the device 700 further includes a statistical recording module 760, used for:

[0115] The target data is read from the cache space using aggregation operators in the streaming data statistics system, and statistical analysis is performed on the target data. The final results obtained from the statistical analysis are written into the database and presented in the form of reports.

[0116] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0117] Please refer to Figure 9This illustration shows a schematic diagram of a computer device provided in one embodiment of this application. The computer device can be any electronic device with data computing, processing, and storage functions, such as a mobile phone, tablet computer, PC (Personal Computer), or server. This computer device is used to implement the data processing method provided in the above embodiments. Specifically:

[0118] The computer device 900 includes a processing unit (such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), and FPGA (Field Programmable Gate Array)) 901, a system memory 904 including RAM (Random-Access Memory) 902 and ROM (Read-Only Memory) 903, and a system bus 905 connecting the system memory 904 and the central processing unit 901. The computer device 900 also includes a basic input / output system 909 that facilitates information transfer between various devices within the server, and a large-capacity storage device 907 for storing the operating system 913, application programs 914, and other program modules 915.

[0119] The basic input / output system 909 includes a display 909 for displaying information and an input device 909 for user input, such as a mouse or keyboard. Both the display 909 and the input device 909 are connected to the central processing unit 901 via an input / output controller 910 connected to the system bus 905. The basic input / output system 909 may also include the input / output controller 910 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 910 also provides output to a display screen, printer, or other types of output devices.

[0120] The mass storage device 907 is connected to the central processing unit 901 via a mass storage controller (not shown) connected to the system bus 905. The mass storage device 907 and its associated computer-readable media provide non-volatile storage for the computer device 900. That is, the mass storage device 907 may include computer-readable media (not shown) such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive.

[0121] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other solid-state storage technologies, CD-ROM, DVD (Digital Video Disc) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage medium is not limited to the above-mentioned types. The system memory 904 and mass storage device 907 described above can be collectively referred to as memory.

[0122] According to an embodiment of this application, the computer device 900 can also be connected to a remote computer on a network, such as the Internet, for operation. That is, the computer device 900 can be connected to a network 912 via a network interface unit 911 connected to the system bus 905, or the network interface unit 911 can be used to connect to other types of networks or remote computer systems (not shown).

[0123] The memory further includes at least one instruction, at least one program, code set, or instruction set, which are stored in the memory and configured to be executed by one or more processors to implement the above data processing method.

[0124] This application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, which is loaded and executed by a processor to implement the above-described data processing method.

[0125] This application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. The processor reads and executes the computer instructions from the computer-readable storage medium to implement the above-described data processing method.

[0126] It should be understood that "multiple" as used in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0127] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent switching, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A data processing method, characterized in that, The method includes: Identify the target data to be accessed in the storage space, which includes cache space and non-cache space, and the cache space includes a set of hot data and a set of retained cold data; The third frequency parameter of the target data is assigned to the first frequency parameter of the target data. The third frequency parameter of the target data is used to represent the number of non-repeating data accessed after the most recent access to the target data. The first frequency parameter of the target data is used to represent the number of non-repeating data accessed within the interval between the two most recent accesses to the target data. The current value of the second frequency parameter of the target data is incremented by 1 to obtain the updated second frequency parameter of the target data. The second frequency parameter of the target data is used to represent the number of times the target data has been accessed in the most recent n accesses, where n is a positive integer. If the target data belongs to the retained cold data set, and if the first frequency parameter of the target data is less than or equal to the first threshold, and the second frequency parameter of the target data is greater than or equal to the second threshold, then the target data is transferred from the retained cold data set to the hot data set.

2. The method according to claim 1, characterized in that, The step of transferring the target data from the retained cold data set to the hot data set includes: Determine a first frequency parameter for each hot data in the hot data set, wherein the first frequency parameter of the hot data is used to represent the number of non-repeating data accessed within the interval between the two most recent accesses of the hot data; From the set of thermal data, select thermal data whose first frequency parameter meets the first condition as target thermal data; The target data is removed from the retained cold data set and added to the hot data set; and the target hot data is removed from the hot data set and added to the retained cold data set.

3. The method according to claim 1, characterized in that, The step of transferring the target data from the retained cold data set to the hot data set includes: Determine a second frequency parameter for each hot data in the hot data set, wherein the second frequency parameter of the hot data is used to represent the number of times the hot data has been accessed in the most recent n accesses; From the set of thermal data, select thermal data whose second frequency parameter meets the second condition as target thermal data; The target data is removed from the retained cold data set and added to the hot data set; and the target hot data is removed from the hot data set and added to the retained cold data set.

4. The method according to claim 1, characterized in that, After determining the target data to be accessed in the storage space, the method further includes: If the target data belongs to the non-cache space, determine the target retained data that needs to be removed from the retained data included in the retained cold data set; Remove the target data from the non-cached space and add it to the retained cold data set; and remove the target retained data from the retained data set and add it to the non-cached space.

5. The method according to claim 4, characterized in that, The step of determining the target retained data to be removed from the retained data included in the retained cold data set includes: Determine a second frequency parameter for each retained cold data in the retained cold data set, wherein the second frequency parameter of the retained cold data is used to represent the number of times the retained cold data has been accessed in the most recent n accesses; From the set of retained cold data, select the retained data whose second frequency parameter meets the third condition as the target retained data.

6. The method according to claim 4, characterized in that, The step of determining the target retained data to be removed from the retained data included in the retained cold data set includes: The third frequency parameter of each retained cold data in the retained cold data set is determined. The third frequency parameter of the retained cold data is used to represent the number of non-repeating data accessed after the most recent access of the retained cold data. From the set of retained cold data, select the retained data whose third frequency parameter meets the fourth condition as the target retained data.

7. The method according to any one of claims 1 to 6, characterized in that, After transferring the target data from the retained cold data set to the hot data set, the method further includes: The target data is read from the cache space using aggregation operators in the streaming data statistics system; Statistical analysis was performed on the target data to obtain the statistical analysis results; The statistical analysis results are written into the database and presented in the form of reports.

8. A data processing apparatus, characterized in that, The device includes: The first determining module is used to determine the target data to be accessed in the storage space, the storage space including cache space and non-cache space, the cache space including a hot data set and a retained cold data set; The parameter update module is used to assign the third frequency parameter of the target data to the first frequency parameter of the target data. The third frequency parameter of the target data is used to represent the number of non-repeating data accessed after the most recent access of the target data, and the first frequency parameter of the target data is used to represent the number of non-repeating data accessed within the interval between the two most recent accesses of the target data. The parameter update module is further configured to increment the current value of the second frequency parameter of the target data by 1 to obtain the updated second frequency parameter of the target data. The second frequency parameter of the target data is used to represent the number of times the target data has been accessed in the most recent n accesses, where n is a positive integer. The first transfer module is configured to, when the target data belongs to the retained cold data set, transfer the target data from the retained cold data set to the hot data set if the first frequency parameter of the target data is less than or equal to a first threshold and the second frequency parameter of the target data is greater than or equal to a second threshold.

9. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one program, which is loaded and executed by the processor to implement the data processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores at least one program, which is loaded and executed by a processor to implement the data processing method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium, which a processor reads from and executes to implement the method as described in any one of claims 1 to 7.

Images