Data Processing Method, Device and Storage Medium

Abstract

The present application provides a data processing method, device, and storage medium. The method includes obtaining data to be processed, determining a first Key of the data to be processed according to a preset rule and the data characteristics of the data to be processed. If there is data corresponding to the first Key in the local cache, then update the most recent access time and the cumulative access count of the first Key, and calculate the current popularity value of the first Key according to the minimum timestamp of the local cache, the updated most recent access time of the first Key, and the updated cumulative access count of the first Key; store the data corresponding to the first Key in the local cache or non-local cache according to the current popularity value of the first Key, and the popularity values corresponding to the data in the non-local cache are lower than the popularity values corresponding to the data in the local cache. The data processing method provided by the present application improves the data processing efficiency.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to a data processing method, device, and storage medium. Background Art

[0002] With the development of communication technology, the big data era has arrived. Processing a large amount of hot data is a key link for enterprises to conduct business management and provide better services for users.

[0003] In the prior art, a large amount of hot data can usually be stored in a cache cluster, and by interacting with the cache cluster, the processing of a large amount of hot data can be achieved.

[0004] However, in the process of implementing this application, the inventor found that there are at least the following problems in the prior art: The interaction efficiency with the cache cluster is limited by the network bandwidth, and data transmission has a high latency, which reduces the data processing efficiency. Summary of the Invention

[0005] This application provides a data processing method, device, and storage medium to improve the efficiency of data processing.

[0006] In a first aspect, this application provides a data processing method, including:

[0007] Obtain the data to be processed, and determine the first Key of the data to be processed according to a preset rule and the data characteristics of the data to be processed;

[0008] If the data corresponding to the first Key exists in the local cache, update the most recent access time and the cumulative access count of the first Key;

[0009] Calculate the current popularity value of the first Key according to the minimum timestamp of the local cache, the updated most recent access time of the first Key, and the updated cumulative access count of the first Key; the minimum timestamp of the local cache is the minimum value of the updated most recent access times of each Key in the local cache;

[0010] Store the data corresponding to the first Key in the local cache or a non-local cache according to the current popularity value of the first Key; the popularity values corresponding to the data in the non-local cache are lower than the popularity values corresponding to the data in the local cache.

[0011] In a possible design, after determining the first Key of the data to be processed according to the preset rule and the data characteristics of the data to be processed, it further includes:

[0012] If there is a second Key in the local cache that needs to be accessed in association with the first Key, update the most recent access time and the cumulative access count of the second Key;

[0013] Calculate the current popularity value of the second Key based on the minimum timestamp of the local cache, the updated most recent access time of the second Key, and the updated cumulative access count of the second Key;

[0014] Store the data corresponding to the second Key in the local cache or in a non-local cache according to the current popularity value of the second Key.

[0015] In a possible design, the calculating the current popularity value of the first Key according to the minimum timestamp of the local cache, the updated most recent access time of the first Key, and the updated cumulative access count of the first Key includes:

[0016] Determine the popularity value of the first Key as the weighted sum of the difference between the most recent access time of the first Key and the minimum timestamp of the local cache and the cumulative access count of the first Key.

[0017] In a possible design, the storing the data corresponding to the first Key in the local cache or in a non-local cache according to the current popularity value of the first Key includes:

[0018] Add the data corresponding to the first Key to the priority queue of the local cache according to the popularity value; the popularity value of the data at the head of the priority queue is higher than that of the data at the tail;

[0019] Swap out the tail data that meets the preset conditions in the priority queue to the non-local cache according to the current number of elements in the priority queue and the preset threshold.

[0020] In a possible design, the non-local cache includes a cache cluster, and the preset threshold includes the maximum value allowed by the window, the reference value, and the preset minimum popularity value; the swapping out the tail data that meets the preset conditions in the priority queue to the non-local cache according to the current number of elements in the priority queue and the preset threshold includes:

[0021] If the number of elements in the priority queue is greater than the maximum value allowed by the window, swap out the tail data in the priority queue that is greater than the maximum value allowed by the window to the eliminated queue;

[0022] If the number of elements in the priority queue is less than the maximum value allowed by the window and greater than the reference value, swap out the tail data in the priority queue whose popularity value is less than the preset minimum popularity value to the eliminated queue;

[0023] Periodically scan the eliminated queue, delete the data in the eliminated queue, and batch-evict it to a non-local cache.

[0024] In a possible design, the non-local cache further includes a persistence layer. After evicting the tail data that meets the preset conditions in the priority queue to the non-local cache according to the current number of elements in the priority queue and a preset threshold, it further includes:

[0025] Set an expiration duration for each data evicted to the cache cluster;

[0026] Write the Key and expiration time corresponding to each data into a persistent ordered queue;

[0027] Periodically scan the persistent ordered queue, obtain the third Key that reaches the expiration time, and evict the data corresponding to the third Key to the persistence layer.

[0028] In a possible design, the method further includes:

[0029] Obtain a query request from a user, parse the query request, and obtain a fourth Key corresponding to the query request;

[0030] According to the fourth Key, read the data corresponding to the fourth Key from the local cache. If the data corresponding to the fourth Key is not read, read the data corresponding to the fourth Key from the non-local cache.

[0031] In a second aspect, the present application provides a data processing device, including:

[0032] An acquisition module, configured to acquire data to be processed, and determine a first Key of the data to be processed according to a preset rule and data characteristics of the data to be processed;

[0033] An update module, configured to update the most recent access time and cumulative access count of the first Key if the data corresponding to the first Key exists in the local cache;

[0034] A calculation module, configured to calculate a current popularity value of the first Key according to a local cache minimum timestamp, the updated most recent access time of the first Key, and the updated cumulative access count of the first Key; the local cache minimum timestamp is the minimum value among the updated most recent access times of each Key in the local cache;

[0035] A storage module, configured to store the data corresponding to the first Key in a local cache or a non-local cache according to the current popularity value of the first Key; the popularity values of the data corresponding to each item in the non-local cache are lower than the popularity values of the data corresponding to each item in the local cache.

[0036] In a third aspect, the present application provides a data processing device, including: at least one processor and a memory;

[0037] The memory stores computer-executable instructions;

[0038] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the method described in the first aspect above and various possible designs of the first aspect.

[0039] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method described in the first aspect above and various possible designs of the first aspect is implemented.

[0040] In a fifth aspect, the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, the method described in the first aspect above and various possible designs of the first aspect is implemented.

[0041] The data processing method, device and storage medium provided by the present application first obtain data to be processed, and determine the first Key of the data to be processed according to a preset rule and the data characteristics of the data to be processed. If the data corresponding to the first Key exists in the local cache, the most recent access time and the cumulative access times of the first Key are updated. According to the minimum timestamp of the local cache, the updated most recent access time of the first Key, and the updated cumulative access times of the first Key, the current popularity value of the first Key is calculated; the minimum timestamp of the local cache is the minimum value among the updated most recent access times of each Key in the local cache. According to the current popularity value of the first Key, the data corresponding to the first Key is stored in the local cache or the non-local cache, and the popularity values of the data corresponding to each item in the non-local cache are lower than the popularity values of the data corresponding to each item in the local cache. By arranging the data to be processed into Keys according to data characteristics, calculating the popularity values of each Key in the local cache based on the minimum timestamp of the local cache, the most recent access time, and the cumulative access times, the present application improves the calculation accuracy of the popularity values. On this basis, according to the level of the popularity values, the data with high popularity values is stored in the local cache, and the data with low popularity values is swapped out to the non-local cache, so that more accesses to the local cache are made during the data processing process, with almost no delay, improving the data processing efficiency. Brief Description of the Drawings

[0042] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0043] Figure 1 Schematic diagram of an application scenario of a data processing method provided by an embodiment of the present application;

[0044] Figure 2 Flow chart of the data processing method provided by an embodiment of the present application Figure 1 ;

[0045] Figure 3 Schematic diagram of the data structure in the local cache provided by an embodiment of the present application;

[0046] Figure 4 Schematic diagram of the data structure of the priority queue provided by an embodiment of the present application;

[0047] Figure 5 Schematic diagram of the data structure of the eliminated data queue provided by an embodiment of the present application;

[0048] Figure 6 Schematic diagram of the data structure of the persistent ordered queue provided by an embodiment of the present application;

[0049] Figure 7 Schematic diagram of the process of data synchronization between the cache cluster and the persistent layer provided by an embodiment of the present application;

[0050] Figure 8 Flow chart of the data processing method provided by an embodiment of the present application Figure 2 ;

[0051] Figure 9 Schematic diagram of the structure of the data processing device provided by an embodiment of the present application;

[0052] Figure 10 Block diagram of a data processing device provided by an embodiment of the present application. Detailed Description of the Embodiments

[0053] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the following will clearly and completely describe the technical solutions in the embodiments of this application with reference to the accompanying drawings in the embodiments of this application. Apparently, the described embodiments are some, but not all, of the embodiments of this application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts shall fall within the protection scope of this application.

[0054] The internal channel processes and call record flows converged from various peripheral systems are aggregated through other systems.

[0055] With the development of communication technologies, the big data era has arrived. Processing massive amounts of hot data is a key link for enterprises to conduct business management and provide better services to users. The processing of massive amounts of hot data can be to sort and process the massive amounts of hot data based on rules.

[0056] In the prior art, hot data can be stored in a cache cluster, such as a cache cluster implemented using Redis, memcached, etc. When sorting and processing data, each piece of data needs to request the current cached data information from the cache cluster and then cache the processed data into the cache cluster. Since the cache cluster has a faster read and write speed than a persistent storage device or a file system, efficient data sorting can be achieved. However, all operations on hot data need to interact with the cache cluster, and the interaction efficiency is limited by the network bandwidth. The direct data request has a high latency, which greatly reduces the throughput of the massive data processing system.

[0057] To solve the above technical problems, the inventors of this application have found that keys can be arranged according to the data characteristics of the hot data to be processed, and the heat values of each key can be calculated based on factors such as the recent access time, cumulative access times, and local cache minimum timestamp of the key, and the data corresponding to the key with a high heat value is stored in the local cache. Based on this, this embodiment provides a data processing method that can improve the calculation accuracy of the heat value. On this basis, more accesses to the local cache are made during the data processing process, with almost no latency, improving the data processing efficiency.

[0058] Figure 1 It is a schematic diagram of the application scenario of a data processing method provided by an embodiment of this application. As Figure 1As shown in the figure, the background service includes multiple services, and each service includes a local cache. The background service obtains the data to be processed from the real-time data generated by each system, and determines the first Key of the data to be processed according to the preset rules and the data characteristics of the data to be processed. If the data corresponding to the first Key exists in the local cache, the most recent access time and the cumulative access times of the first Key are updated. According to the minimum timestamp of the local cache, the updated most recent access time of the first Key, and the updated cumulative access times of the first Key, the current popularity value of the first Key is calculated; the minimum timestamp of the local cache is the minimum value of the updated most recent access times of each Key in the local cache; according to the size of the current popularity value of the first Key, the data corresponding to the first Key is stored in the local cache, or the cache cluster, or the persistent layer. Exemplarily, the data with a high popularity value can be stored in the local cache, the data with a low popularity value can be swapped out to the cache cluster, and the data with an even lower popularity value can be swapped out to the persistent layer. In addition, the client can receive the query request sent by the user and send the query request to any service in the background service through the load balancing for data query. In this application, the data to be processed is arranged with Keys according to the data characteristics, and the popularity values of each Key in the local cache are calculated based on the minimum timestamp of the local cache, the most recent access time, and the cumulative access times, improving the calculation accuracy of the popularity values. On this basis, according to the level of the popularity values, the data with high popularity values is stored in the local cache, and the data with low popularity values is swapped out to non-local caches, enabling more access to the local cache during the data processing process with almost no delay, and improving the data processing efficiency.

[0059] The technical solution of this application will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0060] Figure 2 It is a schematic flow of the data processing method provided by the embodiment of this application Figure 1 As Figure 2 shown, this method includes:

[0061] 201. Obtain the data to be processed, and determine the first Key of the data to be processed according to the preset rules and the data characteristics of the data to be processed.

[0062] The execution subject of this embodiment can be a data processing device such as a computer or a server.

[0063] In this embodiment, the source of the data to be processed can be the real-time data generated by the data generation system, such as the log data generated in real time by the log system, the call data generated in real time by the call detail record system, the shopping system, etc.

[0064] In this embodiment, data features refer to features such as data content, data source, and data collection location. For example, if the data to be processed is a call record, the data features can be the phone prefix, such as 135, 186, etc., and can also be the place of registration; if the data to be processed is a log, the data features can be the log source, log collection point, etc.; if the data to be processed is a commodity order, the data features can be the order number, commodity name, commodity origin, commodity price, etc.

[0065] In this embodiment, the process of arranging a Key for the data to be processed can be understood as classifying the data to be processed. Taking the phone prefix as an example. Data with a phone prefix of 135 can all be arranged as Key1, and data with a phone prefix of 186 can all be arranged as Key2. That is, multiple data can be arranged as the same Key, and each Key can correspond to multiple data.

[0066] 202. If the data corresponding to the first Key exists in the local cache, update the most recent access time and the cumulative access count of the first Key.

[0067] Specifically, after arranging the Key for the data to be processed, it is possible to query the local cache based on the first Key of the data to be processed to check whether there is already cached data belonging to the same first Key. If so, after processing the data to be processed, the processed data to be processed and the above-mentioned cached data can be stored together, that is, data belonging to the same first key are stored together, so as to facilitate processing the data corresponding to the first Key according to the first Key. It can be understood that it is also possible that the storage locations of data belonging to the same first Key are not continuous, and this embodiment does not make any restrictions on this.

[0068] If the data corresponding to the first Key does not exist in the local cache, it is possible to access non-local caches, such as a cache cluster and a persistence layer. If the data corresponding to the first Key exists in the non-local cache, the data can be loaded into the local cache and a cache object can be created. Since this access can be counted as one access record, the cumulative access count can be set to 1, and the most recent access time can be set to the current time. Then, based on the subsequent heat calculation method, the heat value of the first Key can be calculated.

[0069] If the data corresponding to the first Key does not exist in both the local cache and the non-local cache, a cache object can be created. Since this access can be counted as one access record, the cumulative access count can be set to 1, and the most recent access time can be set to the current time. Then, based on the subsequent heat calculation method, the heat value of the first Key can be calculated.

[0070] In this embodiment, during the process of querying the data corresponding to the first key, it is equivalent to accessing the data corresponding to the first key once. Therefore, the most recent access time corresponding to the first key can be updated to the current time, and the cumulative access count of the first key can be incremented by one.

[0071] 203. Calculate the current popularity value of the first key based on the local cache minimum timestamp, the updated most recent access time of the first key, and the updated cumulative access count of the first key; the local cache minimum timestamp is the minimum value among the updated most recent access times of each key in the local cache.

[0072] In this embodiment, there can be multiple ways to calculate the popularity value of the first key. In one implementable way, the sum of the difference between the most recent access time of the first key and the local cache minimum timestamp and the cumulative access count of the first key can be determined as the popularity value.

[0073] In another implementable way, the weighted sum of the difference between the most recent access time of the first key and the local cache minimum timestamp and the cumulative access count of the first key can be determined as the popularity value of the first key.

[0074] Specifically, when sorting and processing the data to be processed with timeliness, the data characteristics of the data to be processed are arranged as the first key according to the preset rules. After processing the data to be processed, the most recent access time of the first key is updated to the current time, and the cumulative access count is increased by 1. The first key is calculated for popularity by the data hotspot calculation component. The popularity value is positively correlated with the most recent access time and the access count. For the scenario where the access frequency of the data gradually decreases over time, the specific popularity calculation is: Popularity value = (Most recent access timestamp - Local cache window data minimum timestamp) * Time impact weighting coefficient + Cumulative access count * Access volume weighting coefficient, where the time impact weighting coefficient and the access volume weighting coefficient can be set according to actual needs. As Figure 3 shown, after calculating the popularity value of the first key, attributes such as the content after data processing, the most recent access time, the access count, and the popularity value are added to the memory cache implemented using Java ConcurrentHashMap.

[0075] 204. Store the data corresponding to the first key in the local cache or non-local cache according to the current popularity value of the first key; the popularity values corresponding to the data in the non-local cache are lower than the popularity values corresponding to the data in the local cache.

[0076] In this embodiment, there are multiple ways to determine whether to retain the first Key in the local cache or evict it to a non-local cache based on the current popularity value of the first Key.

[0077] Exemplarily, in one implementable manner, after calculating the popularity value of the first Key, the popularity value of the first Key can be compared with the popularity values of each Key in the local cache, and the data of the Key with the lowest popularity value is evicted to the non-local cache, otherwise it continues to be retained in the local cache; in another implementable manner, the popularity value of the first Key can be compared with a preset threshold. If it is lower than the preset threshold, it is evicted to the non-local cache, otherwise it continues to be retained in the local cache.

[0078] In practical applications, there may be a dependency relationship between each Key. For example, a certain logging system needs to know the number of faults that occurred on the current day. Suppose a fault report log has been obtained from real-time data and the fault report log is arranged with Key1. When the second fault report log is obtained, it is arranged with Key2. Since the number of faults needs to be known, Key1 also needs to be accessed. Therefore, it can be said that Key2 and Key1 have an associated access relationship. In view of the above situation, in some embodiments, on the basis of the above embodiments, after determining the first Key of the data to be processed according to the preset rules and the data characteristics of the data to be processed, it may further include: if there is a second Key in the local cache that needs to be associated with the first Key for access, then update the most recent access time and the cumulative access times of the second Key; calculate the current popularity value of the second Key according to the minimum timestamp in the local cache, the updated most recent access time of the second Key, and the updated cumulative access times of the second Key; store the data corresponding to the second Key in the local cache or the non-local cache according to the current popularity value of the second Key.

[0079] The data processing method provided in this embodiment arranges Keys for the data to be processed according to the data characteristics, calculates the popularity values of each Key in the local cache based on the minimum timestamp in the local cache, the most recent access time, and the cumulative access times, improving the calculation accuracy of the popularity values. On this basis, according to the high and low of the popularity values, the data with high popularity values is stored in the local cache, and the data with low popularity values is evicted to the non-local cache, enabling more access to the local cache during the data processing process with almost no delay, and improving the data processing efficiency.

[0080] In some embodiments, in order to ensure that the heat value corresponding to the data stored in the local cache is high enough to increase the number of accesses to the local cache, thereby improving the data processing efficiency. Based on the above embodiments, step 204 may specifically include: adding the data corresponding to the first Key to the priority queue of the local cache according to the heat value; the heat value of the data at the head of the priority queue is higher than that of the data at the tail; according to the current number of elements in the priority queue and a preset threshold, the tail data in the priority queue that meets the preset conditions is swapped out to a non-local cache.

[0081] Optionally, the non-local cache includes a cache cluster, and the preset threshold includes a window allowable maximum value, a reference value, and a preset minimum heat value; the step of swapping out the tail data in the priority queue that meets the preset conditions to the non-local cache according to the current number of elements in the priority queue and the preset threshold may include: if the number of elements in the priority queue is greater than the window allowable maximum value, swapping out the tail data in the priority queue that is greater than the window allowable maximum value to the eliminated queue; if the number of elements in the priority queue is less than the window allowable maximum value and greater than the reference value, swapping out the tail data in the priority queue with a heat value less than the preset minimum heat value to the eliminated queue; periodically scanning the eliminated queue, and deleting the data in the eliminated queue and batch swapping it out to the non-local cache.

[0082] Specifically, the Key and heat value of the data to be processed after processing are added to the priority queue of the local cache window through the cache replacement component, and the data elements are sorted according to the heat value. The elements with high heat values are located at the head of the queue, and the elements with low heat values are located at the tail of the queue. The size of the local cache window is controlled by three parameters: the reference value, the allowable maximum value, and the minimum heat value. When the number of elements in the priority queue of the cache window exceeds the allowable maximum value, the low heat value elements at the tail of the queue that exceed the allowable maximum value are removed and written to the eliminated queue; when the number of elements in the priority queue of the cache window exceeds the reference value, it is checked whether the heat value of the data element at the tail of the queue is less than the set minimum heat value parameter. If it is less than the minimum heat value parameter, the data elements in this part are also removed and written to the eliminated queue. The cache replacement component will periodically scan the eliminated data queue and delete the local cache data and batch write it to the distributed cache according to the key in the queue arrangement characteristics. As Figure 4 shown, for the convenience of sorting and improving the data processing efficiency, the priority queue can be implemented using Java's PriorityBlockingQueue. As Figure 5 shown, since sorting is not required, the eliminated data queue can be implemented using Java's LinkedBlockingQueue.

[0083] Optionally, the non-local cache may further include a persistence layer. After periodically scanning the eviction queue, deleting the data in the eviction queue, and swapping out the data in batches to the cache cluster, it may further include: setting an expiration duration for each piece of data swapped out to the cache cluster; writing the Key and expiration time corresponding to each piece of data into a persistent ordered queue; periodically scanning the persistent ordered queue, obtaining the third Key that has reached the expiration time, and swapping out the data corresponding to the third Key to the persistence layer.

[0084] Lifting weights. By means of the strategy of periodically synchronizing cache cluster data to the persistence layer, low-popularity data is swapped out to the persistence layer to avoid the increase in the amount of cached data in the cache cluster over time, which may affect the performance of the host.

[0085] In this embodiment, for the scenario where the access frequency of data gradually decreases over time, Redis can be used as the cache cluster. When hot data is written into Redis, the cache cluster data is automatically deleted by setting the expiration duration of the Key. At the same time, the orchestration feature Key and the expiration time are written into the persistent ordered queue. The expiration time calculation rule is the most recent access time of the Key plus the expiration duration of the Key in Redis. As Figure 6 shown, the persistent ordered queue can be implemented using the Redis sorted set ZSET, and the expiration time is used as the sorting score of the ZSET.

[0086] As Figure 7 shown, the implementation steps of data synchronization between the Redis cache cluster and the persistence layer may include: first, for the swapped-in data swapped into the Redis cache cluster from the local cache, generating the expiration time of the swapped-in data, and writing the data into the persistent ordered queue. Periodically read the persistent ordered queue, query the list of orchestration feature Keys whose expiration time is less than the current time, batch-read the data elements according to the Key list, swap out the read data elements to the persistence layer, and delete the data elements in the persistent ordered queue that are lower than the current synchronization time.

[0087] Figure 8 This is the flowchart of the data processing method provided by the embodiment of the present application. Figure 2 As Figure 8 shown, on the basis of the above embodiment, for example, on the basis of the embodiment shown in Figure 2 this embodiment adds an example description of data reading. The method includes:

[0088] 801. Obtain the data to be processed, and determine the first Key of the data to be processed according to the preset rules and the data characteristics of the data to be processed.

[0089] 802. If the data corresponding to the first Key exists in the local cache, update the most recent access time and the cumulative access count of the first Key.

[0090] 803. Calculate the current popularity value of the first Key based on the minimum timestamp of the local cache, the updated most recent access time of the first Key, and the updated cumulative access count of the first Key; the minimum timestamp of the local cache is the minimum value among the updated most recent access times of each Key in the local cache.

[0091] 804. Store the data corresponding to the first Key in the local cache or non-local cache according to the current popularity value of the first Key; the popularity values corresponding to the data in the non-local cache are lower than the popularity values corresponding to the data in the local cache.

[0092] In this embodiment, steps 801 to 804 are similar to steps 201 to 204 in the above embodiment, and will not be elaborated here.

[0093] 805. Obtain the query request of the user, and parse the query request to obtain the fourth Key corresponding to the query request.

[0094] 806. Read the data corresponding to the fourth Key from the local cache according to the fourth Key. If the data corresponding to the fourth Key is not read, read the data corresponding to the fourth Key from the non-local cache.

[0095] In this embodiment, a query request may correspond to multiple Keys, that is, the fourth Key may be multiple Keys. After reading, the parameter values of the most recent access time and the cumulative access count of the fourth Key can be updated to recalculate the popularity value corresponding to the fourth Key, and based on the current popularity value of the fourth Key, perform the replacement strategy in the above embodiment. For example, if the popularity value is high, the data of the fourth Key can be retained in the local cache, and if the popularity value is low, the data of the fourth Key can be swapped out to the non-local cache.

[0096] The data processing method provided in this embodiment can improve the data processing efficiency by parsing the query request input by the user to obtain the Key to be accessed and reading the data based on the Key.

[0097] Figure 9 It is a schematic structural diagram of a data processing device provided in an embodiment of the present application. As Figure 9 shown, the data processing device 90 includes: an acquisition module 901, an update module 902, a calculation module 903, and a storage module 904.

[0098] An acquisition module 901, configured to acquire data to be processed, and determine a first Key of the data to be processed according to a preset rule and a data feature of the data to be processed.

[0099] An update module 902, configured to update the most recent access time and the cumulative access count of the first Key if data corresponding to the first Key exists in the local cache.

[0100] A calculation module 903, configured to calculate a current popularity value of the first Key according to a minimum timestamp of the local cache, the updated most recent access time of the first Key, and the updated cumulative access count of the first Key; the minimum timestamp of the local cache is the minimum value of the updated most recent access times of each Key in the local cache.

[0101] A storage module 904, configured to store data corresponding to the first Key in the local cache or a non-local cache according to the current popularity value of the first Key; the popularity values corresponding to the data in the non-local cache are lower than the popularity values corresponding to the data in the local cache.

[0102] The data processing device provided by the embodiment of the present application arranges the Key according to the data feature for the data to be processed, calculates the popularity value of each Key in the local cache based on the minimum timestamp of the local cache, the most recent access time, and the cumulative access count, improves the calculation accuracy of the popularity value. On this basis, according to the level of the popularity value, the data with a high popularity value is stored in the local cache, and the data with a low popularity value is swapped out to the non-local cache, so that more accesses to the local cache are made during the data processing process, with almost no delay, improving the data processing efficiency.

[0103] The data processing device provided by the embodiment of the present application can be used to execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be elaborated here in this embodiment.

[0104] Figure 10 It is a block diagram of a data processing device provided by an embodiment of the present application, and the device can be a data processing device such as a computer or a server.

[0105] The apparatus 100 may include one or more of the following components: a processing component 1001, a memory 1002, a power component 1003, a multimedia component 1004, an audio component 1005, an input / output (I / O) interface 1006, a sensor component 1007, and a communication component 1008.

[0106] The processing component 1001 generally controls the overall operation of the device 100, such as operations associated with display, telephone calls, data communication, camera operations, and recording operations. The processing component 1001 may include one or more processors 1009 to execute instructions to complete all or part of the steps of the above-described methods. In addition, the processing component 1001 may include one or more modules to facilitate the interaction between the processing component 1001 and other components. For example, the processing component 1001 may include a multimedia module to facilitate the interaction between the multimedia component 1004 and the processing component 1001.

[0107] The memory 1002 is configured to store various types of data to support the operation of the device 100. Examples of such data include instructions for any application or method operating on the device 100, contact data, phone book data, messages, pictures, videos, and the like. The memory 1002 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk.

[0108] The power component 1003 provides power to various components of the device 100. The power component 1003 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 100.

[0109] The multimedia component 1004 includes a screen that provides an output interface between the device 100 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe actions, but also detect the duration and pressure associated with the touch or swipe operations. In some embodiments, the multimedia component 1004 includes a front camera and / or a rear camera. When the device 100 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each of the front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capabilities.

[0110] The audio component 1005 is configured to output and / or input audio signals. For example, the audio component 1005 includes a microphone (MIC), which is configured to receive external audio signals when the device 100 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 1002 or transmitted via the communication component 1008. In some embodiments, the audio component 1005 further includes a speaker for outputting audio signals.

[0111] The I / O interface 1006 provides an interface between the processing component 1001 and a peripheral interface module, and the peripheral interface module can be a keyboard, a click wheel, buttons, etc. These buttons can include, but are not limited to: a home button, a volume button, a start button, and a lock button.

[0112] The sensor component 1007 includes one or more sensors for providing a status assessment of various aspects of the device 100. For example, the sensor component 1007 can detect the on / off state of the device 100, the relative positioning of components, such as the display and keypad of the device 100. The sensor component 1007 can also detect a change in the position of the device 100 or a component of the device 100, the presence or absence of user contact with the device 100, the orientation or acceleration / deceleration of the device 100, and the temperature change of the device 100. The sensor component 1007 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor component 1007 can also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 1007 can further include an acceleration sensor, a gyro sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0113] The communication component 1008 is configured to facilitate communication between the device 100 and other devices in a wired or wireless manner. The device 100 can access a wireless network based on a communication standard, such as WiFi, 2G, or 3G, or a combination thereof. In an exemplary embodiment, the communication component 1008 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1008 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0114] In an exemplary embodiment, the apparatus 100 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the above method.

[0115] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1002 including instructions, and the above instructions can be executed by a processor 1009 of the apparatus 100 to complete the above method. For example, the non-transitory computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0116] The above computer-readable storage medium may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk, or an optical disk. The readable storage medium may be any available medium accessible by a general-purpose or special-purpose computer.

[0117] An exemplary readable storage medium is coupled to the processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium may also be a component of the processor. The processor and the readable storage medium may be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium may also exist as discrete components in the device.

[0118] Those of ordinary skill in the art can understand that all or part of the steps for implementing the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps including the above method embodiments; and the foregoing storage medium includes: ROM, RAM, magnetic disk, or optical disk and other media that can store program codes.

[0119] The embodiments of the present application also provide a computer program product including a computer program, and when the computer program is executed by a processor, it implements the data processing method executed by the above data processing device.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than limiting them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that, Including: Obtain the data to be processed, and determine the first Key of the data to be processed according to a preset rule and the data characteristics of the data to be processed; If the data corresponding to the first Key exists in the local cache, update the most recent access time and the cumulative access count of the first Key; Calculate the current popularity value of the first Key according to the minimum timestamp of the local cache, the updated most recent access time of the first Key, and the updated cumulative access count of the first Key; the minimum timestamp of the local cache is the minimum value among the updated most recent access times of each Key in the local cache; Store the data corresponding to the first Key in the local cache or non-local cache according to the current popularity value of the first Key; The popularity value corresponding to each data in the non-local cache is lower than the popularity value corresponding to each data in the local cache; After determining the first Key of the data to be processed according to the preset rule and the data characteristics of the data to be processed, it further includes: If a second Key that needs to be accessed in association with the first Key exists in the local cache, update the most recent access time and the cumulative access count of the second Key; Calculate the current popularity value of the second Key according to the minimum timestamp of the local cache, the updated most recent access time of the second Key, and the updated cumulative access count of the second Key; Store the data corresponding to the second Key in the local cache or non-local cache according to the current popularity value of the second Key; The calculating the current popularity value of the first Key according to the minimum timestamp of the local cache, the updated most recent access time of the first Key, and the updated cumulative access count of the first Key includes: Determine the popularity value of the first Key as the weighted sum of the difference between the most recent access time of the first Key and the minimum timestamp of the local cache and the cumulative access count of the first Key.

2. The method according to claim 1, characterized in that, The storing the data corresponding to the first Key in the local cache or non-local cache according to the current popularity value of the first Key includes: Add the data corresponding to the first Key to the priority queue of the local cache according to the popularity value; the popularity value of the data at the head of the priority queue is higher than that of the data at the tail; Swap out the tail data that meets the preset conditions in the priority queue to the non-local cache according to the current number of elements in the priority queue and the preset threshold.

3. The method according to claim 2, wherein The non-local cache includes a cache cluster, and the preset threshold includes the maximum value allowed in the window, the reference value, and the preset minimum popularity value; the swapping out the tail data that meets the preset conditions in the priority queue to the non-local cache according to the current number of elements in the priority queue and the preset threshold includes: If the number of elements in the priority queue is greater than the maximum value allowed in the window, swap out the tail data in the priority queue that is greater than the maximum value allowed in the window to the eliminated queue; If the number of elements in the priority queue is less than the maximum value allowed by the window and greater than the reference value, the tail data in the priority queue with a heat value less than the preset minimum heat value is swapped out to the eliminated queue; Periodically scan the eliminated queue, and delete the data in the eliminated queue and batch-swap it out to the cache cluster.

4. The method according to claim 3, wherein The non-local cache further includes a persistent layer. After periodically scanning the eliminated queue and deleting the data in the eliminated queue and batch-swap it out to the cache cluster, it further includes: Set an expiration duration for each data swapped out to the cache cluster; Write the Key and expiration time corresponding to each data into the persistent ordered queue; Periodically scan the persistent ordered queue, obtain the third Key that reaches the expiration time, and swap out the data corresponding to the third Key to the persistent layer.

5. The method according to any one of claims 1 to 4, characterized in that The method further includes: Obtain a query request of a user, and parse the query request to obtain a fourth Key corresponding to the query request; According to the fourth Key, read the data corresponding to the fourth Key from the local cache. If the data corresponding to the fourth Key is not read, read the data corresponding to the fourth Key from the non-local cache.

6. A data processing device, characterized in that, It includes: An acquisition module, configured to acquire data to be processed, and determine a first Key of the data to be processed according to a preset rule and data characteristics of the data to be processed; An update module, configured to update the most recent access time and cumulative access times of the first Key if data corresponding to the first Key exists in the local cache; A calculation module, configured to calculate a current heat value of the first Key according to a local cache minimum timestamp, the updated most recent access time of the first Key, and the updated cumulative access times of the first Key; the local cache minimum timestamp is the minimum value among the updated most recent access times of each Key in the local cache; A storage module, configured to store the data corresponding to the first Key in the local cache or the non-local cache according to the current heat value of the first Key; The heat value corresponding to each data in the non-local cache is lower than the heat value corresponding to each data in the local cache; After the acquisition module is configured to determine the first Key of the data to be processed according to the preset rule and the data characteristics of the data to be processed, through the calculation module, if a second Key that needs to be associated with the first Key exists in the local cache, update the most recent access time and cumulative access times of the second Key; Calculate the current heat value of the second Key according to the local cache minimum timestamp, the updated most recent access time of the second Key, and the updated cumulative access times of the second Key; The storage module is further configured to store the data corresponding to the second Key in the local cache or the non-local cache according to the current heat value of the second Key; The calculation module is specifically configured to determine the popularity value of the first Key as the weighted sum of the difference between the most recent access time of the first Key and the local cache minimum timestamp and the cumulative access count of the first Key.

7. A data processing device, characterized in that, It includes: At least one processor and a memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the data processing method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, Computer-executable instructions are stored in the computer-readable storage medium, and when the processor executes the computer-executable instructions, the data processing method according to any one of claims 1 to 5 is implemented.

Images