Data processing method and device, electronic equipment and medium

Abstract

The present disclosure discloses a data processing method and device, an electronic device, and a storage medium. The data processing method may include: determining the access mode of a first user based on the first user's access information to data pages in a first-level cache; wherein the first-level cache is set in the memory of the first user's service process; based on the first user's access mode, transferring data of a target data page whose access popularity meets a preset condition to a target partition of a second-level cache; wherein the second-level cache is set on a hard disk; the target partition is a cache partition corresponding to the first user's access mode among multiple cache partitions of the second-level cache, and different access modes correspond to different cache partitions.

Description

Technical Field

[0001] The present disclosure relates to the technical field of database cache, and in particular to a data processing method and device, an electronic device, and a storage medium. Background Art

[0002] A cloud database is an online relational database deployed and virtualized in a cloud computing environment. It has the characteristics of high scalability, high availability, ease of use, and support for large-scale parallel processing.

[0003] To cope with the demands of massive data storage and multi-tenant data access, caching technology has been introduced. Existing caching technologies can be divided into local caching and distributed caching, depending on whether the processes performing data caching and data modification operations belong to the same process.

[0004] Local caching refers to caching data in the memory of the service process. The data does not need to be transmitted over the network and the access speed is fast, but it will occupy the memory space of the service process and is destined to not be able to store large amounts of data.

[0005] The disadvantage of the local cache architecture is that it needs to be coupled with the service process. The service processes cannot share the same cache. Each service process and cluster node needs to provide additional memory space for caching. When the application process crashes or restarts, the cached data will be lost.

[0006] Distributed caching refers to caching using cache components or services that are separate from the service process. Multiple service processes can share a cache with independent memory space, supporting cluster expansion and caching of large amounts of data.

[0007] The disadvantage of distributed caching is that in high-concurrency, large-scale data read and write scenarios, data cache avalanche and / or cache breakdown may occur.

[0008] Cache avalanche: A large number of cache failures occur at the same time, causing a large number of requests to directly access the database, which puts too much pressure on the database and may cause the database to crash.

[0009] Cache breakdown: The cache of a certain hot data expires at a certain moment. At this time, there are a large number of concurrent requests for the data, which may also cause the database to be overloaded and crash. Summary of the Invention

[0010] The embodiments of the present disclosure provide a data processing method and apparatus, an electronic device, and a storage medium to solve the problems of slow data access response rate and downtime caused by accumulation of access volume.

[0011] A first aspect of the present disclosure provides a data processing method, including:

[0012] Determining an access pattern of the first user based on access information of the first user to data pages in a first-level cache, wherein the first-level cache is set in a memory of a service process of the first user;

[0013] According to the access pattern of the first user, data of the target data page whose access popularity meets the preset conditions is transferred to the target partition of the secondary cache; wherein, the secondary cache is set on the hard disk; the target partition is the cache partition corresponding to the access pattern of the first user among the multiple cache partitions of the secondary cache, and different access patterns correspond to different cache partitions.

[0014] A second aspect of the disclosed embodiment provides a data processing device, including:

[0015] a determination module, configured to determine an access pattern of the first user based on access information of the first user to data pages in a first-level cache, wherein the first-level cache is set in a memory of a service process of the first user;

[0016] A transfer module is used to transfer the data of the data page whose access popularity meets the preset conditions to the target partition of the secondary cache according to the access pattern of the first user; wherein the secondary cache is set on the hard disk; the target partition is the cache partition corresponding to the access pattern of the first user among the multiple cache partitions of the secondary cache, and different access patterns correspond to different cache partitions.

[0017] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0018] Memory;

[0019] A processor is connected to the memory and is used to implement the data processing method provided by any one of the technical solutions of the first aspect by executing computer executable instructions stored in the memory.

[0020] A fourth aspect of the embodiments of the present disclosure provides a computer storage medium, which stores computer-executable instructions. After the computer-executable instructions are executed by a processor, the data processing method provided by any one of the technical solutions in the first aspect can be implemented.

[0021] The technical solution provided by the embodiment of the present disclosure is to cache data pages in a hierarchical manner using a first-level cache and a second-level cache. In this way, the data pages with the highest access popularity will be stored in the first-level cache (i.e., memory), so that the data pages with higher access popularity can respond to access quickly because they are stored in the first-level cache, and the data pages with lower access popularity are transferred to the second-level cache. In this way, the second-level cache is the hard disk of the cluster node, which improves the response efficiency compared to reading data directly from the database disk. On the one hand, compared with pure local memory caching, it reduces the limitations of local memory, and compared with distributed caching, it reduces cache avalanche and / or cache breakdown and other problems. And more importantly, based on the access information of the user accessing its own first-level cache, the user's access pattern is determined, and the data pages accessed by the user are cached in the corresponding cache partition in the second-level cache according to the access pattern. In this way, the data pages stored in the target partition corresponding to the user's access pattern in the second-level cache can be used for priority response, thereby improving the response rate, and based on big data statistics, unnecessary data statistics can be reduced, and the downtime caused by the accumulation of access requests can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A flowchart of a data processing method provided in an embodiment of the present disclosure;

[0023] Figure 2 A flowchart of a data processing method provided in an embodiment of the present disclosure;

[0024] Figure 3 A flowchart of a data processing method provided in an embodiment of the present disclosure;

[0025] Figure 4 A schematic diagram of a cache queue provided in an embodiment of the present disclosure;

[0026] Figure 5 A schematic structural diagram of a data cache device provided in an embodiment of the present disclosure;

[0027] Figure 6 A schematic structural diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] In order to enable a more detailed understanding of the features and technical contents of the present disclosure, the implementation of the present disclosure is described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference only and are not intended to limit the present disclosure.

[0029] like Figure 1 As shown, the embodiment of the present disclosure provides a data processing method, including:

[0030] S1110: Determine an access pattern of a first user based on access information of the first user to data pages in a first-level cache, wherein the first-level cache is set in a memory of a service process of the first user;

[0031] S1120: According to the access pattern of the first user, the data of the target data page whose access popularity meets the preset conditions is transferred to the target partition of the secondary cache; wherein, the secondary cache is set on the hard disk; the target partition is the cache partition corresponding to the access pattern of the first user among the multiple cache partitions of the secondary cache, and different access patterns correspond to different cache partitions.

[0032] The data processing method can be applied to data processing in a database system, for example, in a cloud database system. The database system may include a computer cluster, with one or more nodes in a cluster.

[0033] The data processing method may be used in cluster nodes in a database system, for example, in cache nodes in the cluster nodes.

[0034] The execution node of the data processing method of the embodiment of the present disclosure may be: the first user may be any tenant.

[0035] A first-level cache is configured for each user in the cluster node. The first-level cache is configured in the memory of the cluster node. In this way, the local cache can be used to provide a fast response to highly accessed data for the user.

[0036] Because cluster nodes have limited memory, a second level cache is deployed on the node's solid-state drive. The capacity of a user's second-level cache can be larger than that of their first-level cache. Typically, data pages stored in the second-level cache are less popular than data pages cached in the first-level cache at the current moment.

[0037] The access popularity is a parameter value that measures the probability that a corresponding data page will be accessed. It can be related to the historical access and / or modification history of the corresponding data page before the current moment. For example, if a data page has been frequently accessed before the current moment and the content of this data page is content that users like to read repeatedly, the access popularity of this data page will be relatively high. For another example, if a data page is currently being modified and the modification has not yet been completed, the probability of this data page continuing to be accessed is very high, and is generally higher than the access popularity of a data page that has already been modified.

[0038] Because the first-level cache is within the cluster node's memory, the data pages stored in the memory can be quickly responded to the first user's access request. The second-level cache can be located in the cluster node's hard disk. The cluster node's hard disk may include a solid-state drive and a regular hard disk, and the second-level cache can further be located in the cluster node's solid-state drive. The access rate of the solid-state drive is lower than the access rate of the memory; and the access rate of the solid-state drive is higher than the access rate of the regular hard disk. Regular hard disks include, but are not limited to, mechanical hard disks. However, the cost of a solid-state drive is higher than that of a regular hard disk.

[0039] Since the secondary cache is located in the cluster nodes, the response rate of data pages accessed is higher than that of data pages located on the database disk.

[0040] It is worth noting that, to further improve access response rates, in the disclosed embodiment, the first user's access pattern is determined based on the first user's access information to data pages in the first-level cache. Different access patterns may result in different data page types preferred by the first user, or different frequency distributions of the first user's access to different types of data pages.

[0041] A L2 cache in a cluster node can be shared by multiple users. For example, a first user and a second user share a L2 cache and a single cache partition. For example, the service processes of the first and second users are created in the same cluster node. Specifically, the target partition is the cache interval in the L2 cache corresponding to the access patterns of the first and second users; the access pattern of the second user satisfies the similarity condition with the access pattern of the first user.

[0042] In the embodiment of the present disclosure, a correspondence between different cache partitions and access patterns in the secondary cache is established, so that data pages eliminated from the primary cache of multiple users with similar access patterns are transferred to the same cache partition. When the access request of the user with the access pattern subsequently cannot be responded to based on the data page in the primary cache, the cache partition in the secondary cache corresponding to the access pattern of the user is preferentially traversed based on this correspondence. If the data page that responds to its access request is retrieved in the cache partition, it is responded directly; if not retrieved, it continues to traverse other partitions in the secondary cache. This can reduce the rate at which the secondary cache responds to user data access and reduce unnecessary data retrieval in the secondary cache, thereby improving the quality of data access service.

[0043] The determining, based on the first user's access information to the data pages in the first-level cache, the access pattern of the first user includes:

[0044] determining an access mode value of the first user based on access information of the first user to the data page in the first-level cache;

[0045] determining an access mode of the first user according to the access mode value;

[0046] The access information includes at least one of the following: the type of data page accessed by the first user in the first-level cache; the access frequency of the first user to each type of data page; and the first weight of each type of data page.

[0047] Exemplarily, determining the access pattern of the first user based on access information of the first user to the data pages in the first-level cache includes:

[0048] Calculating an access mode value according to types of data pages accessed by the first user in the first-level cache and access frequencies of the data pages of each type;

[0049] The access mode of the second user and the access mode of the first user meet a similarity condition, including: the access mode value of the second user and the access mode value of the first user are in the same mode value range.

[0050] Furthermore, calculating the access pattern value according to the type of data pages accessed by the first user in the first-level cache and the access frequency of each type of the data pages may include:

[0051] The access mode value is calculated according to the type of data pages accessed by the first user in the first-level cache, the first weight of each type of data page, and the access frequency of each type of the data page.

[0052] For example, a tenant (or user) accesses data pages in the first-level cache. Different types of data pages may be accessed at different frequencies, which also reflects the access characteristics of the user's data pages.

[0053] For example, Among them, J is the total number of types of the data page, W i is the access weight of the i-th type data page; f i is the frequency of the first user accessing the data page of type i. M is the aforementioned access mode value; and N is the total number of times the first user accesses the first-level cache per unit time.

[0054] In this way, different users can calculate different access pattern values. Based on the different access pattern values, and according to the intervals in which the access pattern values ​​are distributed, multiple users with the same or similar access pattern values ​​can be mapped to the same cache partition of the secondary cache. For example, a cluster node provides data caching and application services for 100 users. Then, each of the 100 users has its own first-level cache, and based on the access of these 100 users to the data pages in their respective first-level caches, the pattern value interval corresponding to the access pattern values ​​of the 100 users can be determined, and cache intervals can be allocated to the pattern value intervals distributed by users in the cache partitions of the secondary cache. A cache interval can correspond to a pattern value interval, and can correspond to one or more users with similar access pattern values.

[0055] like Figure 2 As shown, an embodiment of the present disclosure provides a data processing method, which is executed by a cluster node. The method may include:

[0056] S2110: When the data page specified for access is not found in the first-level cache of the first user, determine a cache partition of the second-level cache to be traversed first according to the access mode value of the first user;

[0057] S2120: Prioritize traversing (searching) the secondary cache area corresponding to the access pattern of the first user.

[0058] Here, prioritizing traversal of the cache area of ​​the secondary cache corresponding to the access mode of the first user may include:

[0059] After traversing the cache partition of the secondary cache corresponding to the first user access pattern, traversing other cache partitions in the secondary cache;

[0060] or,

[0061] If the cache partitions traversed by the cache node at one time are less than the total number of cache partitions in the secondary cache and there are multiple cache partitions, the cache partition corresponding to the first user access mode must be included in the first batch of traversed cache partitions.

[0062] The first batch of cache partitions traversed may include: a cache partition corresponding to the first user's access pattern and a predetermined partition, wherein the pattern value interval corresponding to the predetermined partition is adjacent to the interval containing the first user's access pattern value.

[0063] In some embodiments, calculating the access pattern value according to the type of data pages accessed by the first user in the first-level cache and the access frequency of each type of the data pages includes:

[0064] The access mode value is calculated according to the type of data pages accessed by the first user in the first-level cache, the first weight of each type of the data pages, and the access frequency of each type of the data pages.

[0065] In the embodiment of the present disclosure, since different types of data pages store data of different importance, in order to better determine the access patterns of different users, a first weight is introduced for calculation. The first weight here can be a pre-configured weight.

[0066] Exemplarily, the types of data pages include: index page, system page, metadata page and / or content page;

[0067] wherein the first weight of the metadata page is greater than the first weight of the system page;

[0068] The first weight of the system page is greater than the first weight of the index page;

[0069] The first weight of the index page is greater than the first weight of the content page.

[0070] In some embodiments, the system page may store some statistical information generated by the storage system. The metadata page may store metadata of the stored content. The content page is the stored content, for example, the file content of the stored file.

[0071] A user's first-level cache stores one or more types of data pages mentioned above.

[0072] Of course, in other embodiments, the first weight of the metadata page is greater than the first weight of the index page; the first weight of the index page is greater than the first weight of the system page; the first weight of the index page is greater than the first weight of the content page.

[0073] It is worth noting that the first weights of different types of data pages may not be introduced when calculating the access pattern value, that is, the access pattern value may be calculated directly based solely on the access frequency of each type of data page and the total number of visits by the user.

[0074] like Figure 3 As shown, an embodiment of the present disclosure provides a data processing method, which is executed by a cluster node. The method may include:

[0075] S3110: Determine a first cache hit rate and a cache capacity of each cache partition of the secondary cache;

[0076] S3120: Determine a second cache hit ratio of the secondary cache according to the first cache hit ratio and the cache capacity;

[0077] S3130: Adjust the cache partition according to the first cache hit ratio and the second cache hit ratio.

[0078] Exemplarily, the second cache hit rate may be calculated according to the following formula:

[0079] Among them, S i is the first cache hit rate of the i-th cache partition; H i is the capacity of the i-th cache partition; X+1 is the total number of cache partitions contained in the secondary cache.

[0080] It can be the average cache hit ratio of the second-level cache.

[0081] If there is S i Greater than When , it means that the cache hit rate of the i-th cache partition is too high. If S i Less than When , ...

[0082] For example, if If the value of is greater than the first value, the i-th cache partition is scaled down; if If the value is greater than the second value, the i-th cache partition is expanded.

[0083] By scaling down, one or more data pages with the highest access popularity are divided into the cache partition that needs to be expanded, thereby achieving load balancing among different cache partitions.

[0084] Therefore, in some embodiments, adjusting the cache partition according to the first cache hit rate and the second cache hit rate includes:

[0085] When the first cache hit rate of the P1th cache partition is higher than or equal to the second cache hit rate, one or more data pages with the highest access popularity in the P1th cache partition are divided into the P2th cache area, wherein the first cache hit rate of the P2th cache area is lower than the second cache hit rate.

[0086] Wherein, P1 and P2 may both be natural numbers less than or equal to X. That is, P1 is different from P2, and both P1 and P2 are positive integers less than or equal to X; X is the total number of cache partitions included in the secondary cache.

[0087] In some embodiments, the method further comprises:

[0088] Setting a cache queue in the first-level cache;

[0089] Determining access popularity based on access information of each type of data page in the first-level cache and / or predicted access probability within a preset future time period;

[0090] Determining the position of each data page in the cache queue according to the access popularity; wherein the access popularity of the data page at the head of the cache queue is lower than the access popularity of the data page at the tail of the cache queue;

[0091] The step of transferring the data of the data page whose access popularity satisfies a preset condition to a target partition of the secondary cache according to the access pattern of the first user includes:

[0092] When the first-level cache overflows, the data page at the head of the cache queue is transferred to the target partition of the second-level cache at the head of the cache queue according to the access pattern of the first user.

[0093] Build cache queues and cache queue management to transfer low-access data eliminated from the first-level cache to the second-level cache. Figure 4 This is a schematic diagram of a cache queue. In the cache queue, the access popularity decreases from the head to the tail. Data pages with high access popularity are inserted from the tail and discarded from the head or transferred to the secondary cache.

[0094] The access information of the data page may be historical access information before the current moment.

[0095] The historical access information may include at least one of the following information:

[0096] Contains the access frequency per unit time at the current moment;

[0097] Last visited time.

[0098] When determining the access popularity based solely on the access information, the following functional relationship may be used:

[0099] H = (α*(1-R)+β*t)*f; where H is the access popularity; α and β are predetermined coefficients and can be considered as arbitrary integers.

[0100] R is the space occupancy ratio of the data page in the L2 cache.

[0101] t is the initial weight of the data page or the inverse of the time difference between the last access time and the current time; f is the access frequency of the data page.

[0102] If t is the initial weight of the data page, then t can be a preset value. For example, a value determined based on the data content contained in different types of data pages. Specifically, the following relationship can also be satisfied:

[0103] The initial weight of the metadata page is greater than the initial weight of the system page;

[0104] The initial weight of the system page is greater than the initial weight of the index page;

[0105] The initial weight of the index page is greater than the initial weight of the content page.

[0106] Of course, the above is merely an example of determining access popularity based on access information, and the specific implementation is not limited to the above example.

[0107] The predicted access probability within a preset future time period may be determined based on at least one of the following information:

[0108] According to historical access information, preference information of the first user, whether other data pages related to the content in the data page are generated, whether the first user is updating the data page, etc.

[0109] The access probability within a preset time period in the future is positively correlated with the access frequency in the historical access information;

[0110] The probability of access within a preset time period in the future is negatively correlated with the time between the last access time in the historical access information and the current time;

[0111] The access probability within the future preset time period is increased if the content of the data page pointed to by the probability is a data page that the first user likes to visit; otherwise, the access probability within the future preset time period remains unchanged;

[0112] If the first user is updating the data page, the access probability within the future preset time period increases; otherwise, the access probability within the future preset time period remains unchanged.

[0113] Of course, the above is merely an example of determining the access probability within a preset time period in the future, and the specific implementation is not limited to the above example.

[0114] If cache queue storage is used in the first-level cache, the data page with the highest access popularity will be placed at the end of the queue, and the data page with the lowest access popularity will be placed at the head of the queue. In this way, when the first-level cache overflows (that is, there is not enough cache space), the data page can be directly eliminated from the head of the queue to the second-level cache, etc., which makes the operation simpler and improves the efficiency of freeing up space for subsequent highly accessed data.

[0115] In some embodiments, transferring the data page at the head of the cache queue to the target partition of the secondary cache according to the access pattern of the first user includes:

[0116] When the access popularity of the data page at the head of the cache queue is higher than a preset value, the data page at the head of the cache queue is transferred to the target partition of the secondary cache according to the access pattern of the first user.

[0117] Since the access popularity of data pages stored in the L2 cache is lower than that of data pages stored in the L1 cache, but still higher than the preset value, the access popularity of data pages with low access popularity is reduced. Therefore, when a data page is evicted from the L1 cache, the access popularity of the data page transferred to the L2 cache must be higher than the preset value.

[0118] In some embodiments, the method further comprises:

[0119] When the access popularity of the data page at the head of the cache queue is lower than or equal to the preset value, determining whether the data page at the head of the cache queue has been updated;

[0120] When there is an update, the data page at the head of the cache queue is written to the database disk;

[0121] In the absence of update, the data page at the head of the cache queue is discarded.

[0122] If the access popularity of a data page eliminated from the cache queue of the first-level cache is lower than or equal to the preset value, the data page will not be transferred to the second-level cache.

[0123] If the data page is updated according to user operations during the caching process in the first-level cache, for data consistency, the updated data page will be written to the database disk before the first-level cache deletes the data page.

[0124] If the version of the data page stored in the first-level cache is the same as the version stored on the database disk, that is, the data page has not been updated in the first-level cache, the data page is already stored on the database disk, so it can be deleted directly from the first-level cache.

[0125] In some embodiments, the data pages of each partition of the secondary cache can also be stored using a storage queue, and the position of the secondary cache in the storage queue of the corresponding partition can also be determined based on the access heat. The calculation method of the access heat can refer to the access heat of each data page in the first-level cache. Similarly, the data page with high access heat can be placed at the end of the queue, and the data page with low access heat can be placed at the head of the queue. In this way, if the corresponding secondary cache overflows, the data page can also be directly discarded from the head of the queue or the data page with updated content can be written to the database disk.

[0126] like Figure 5 The present disclosure provides a data processing device, including:

[0127] A determination module 110 is configured to determine an access pattern of the first user based on access information of the first user to data pages in a first-level cache, wherein the first-level cache is set in a memory of a service process of the first user;

[0128] The transfer module 120 is used to transfer the data of the data page whose access popularity meets the preset conditions to the target partition of the secondary cache according to the access pattern of the first user; wherein the secondary cache is set on the hard disk; the target partition is the cache partition corresponding to the access pattern of the first user among the multiple cache partitions of the secondary cache, and different access patterns correspond to different cache partitions.

[0129] In some embodiments, the determination module 110 is specifically configured to determine an access pattern value of the first user based on access information of the first user to data pages in the first-level cache; and determine the access pattern of the first user based on the access pattern value. The access information includes at least one of the following: the type of data pages accessed by the first user in the first-level cache; the frequency of access by the first user to each type of data page; and a first weight of each type of data page.

[0130] Exemplarily, the types of data pages include:

[0131] Index pages, system pages, metadata pages and / or content pages;

[0132] wherein the first weight of the metadata page is greater than the first weight of the system page;

[0133] The first weight of the system page is greater than the first weight of the index page;

[0134] The first weight of the index page is greater than the first weight of the content page;

[0135] The determination module 110 can be specifically used to determine the first weight of each type of data page based on the space ratio occupied by each type of data page in the first-level cache and the second-level cache, the total access frequency of each type of data page in the first-level cache and the second-level cache, and a pre-set second weight.

[0136] In some embodiments, the apparatus comprises:

[0137] a cache partition module, configured to determine a first cache hit rate and a cache capacity of each cache partition of the secondary cache;

[0138] a second cache hit rate module, configured to determine a second cache hit rate of the secondary cache according to the first cache hit rate and the cache capacity;

[0139] An adjustment module is configured to adjust the cache partition of the secondary cache according to the first cache hit rate and the second cache hit rate.

[0140] In some embodiments, the adjustment module is specifically configured to, when the first cache hit rate of the P1th cache partition is higher than or equal to the second cache hit rate, allocate one or more most frequently accessed data pages in the P1th cache partition to the P2th cache area, wherein the first cache hit rate of the P2th cache area is lower than the second cache hit rate. P1 is different from P2, and both P1 and P2 are positive integers less than or equal to X; X is the total number of cache partitions included in the secondary cache.

[0141] In some embodiments, the apparatus further comprises:

[0142] A cache module, configured to set a cache queue in the first-level cache;

[0143] an access heat module, configured to determine access heat based on access information of each type of data page in the first-level cache and / or a predicted access probability within a preset future time period;

[0144] The determination module 110 is configured to determine the position of each data page in the cache queue according to the access popularity; wherein the access popularity of the data page at the head of the cache queue is lower than the access popularity of the data page at the tail of the cache queue;

[0145] The transfer module 120 is specifically configured to transfer the data page at the head of the cache queue to the target partition of the second-level cache according to the access pattern of the first user when the first-level cache overflows.

[0146] In some embodiments, the transfer module 120 is specifically used to transfer the data page at the head of the cache queue to the target partition of the secondary cache according to the access pattern of the first user when the access popularity of the data page at the head of the cache queue is higher than a preset value.

[0147] In some embodiments, the apparatus further comprises:

[0148] An update determination module 110, configured to determine whether the data page at the head of the cache queue has been updated when the access popularity of the data page at the head of the cache queue is lower than or equal to the preset value;

[0149] A writing module, configured to write the data page at the head of the cache queue to the database disk when an update is made;

[0150] The discard module is used to discard the data page at the head of the cache queue when there is no update.

[0151] like Figure 6 As shown, an embodiment of the present disclosure provides an electronic device, the electronic device comprising:

[0152] Memory;

[0153] A processor is connected to the memory and is used to implement the data processing method provided by any of the above embodiments by executing computer executable instructions stored in the memory, such as executing Figures 1 to 3 The data processing method shown is as follows.

[0154] The electronic device may be the aforementioned cluster node. Figure 6 As shown, the electronic device may further include a network interface, which can be used to interact with a peer device through a network.

[0155] The present disclosure provides a computer storage medium storing computer executable instructions. After the computer executable instructions are executed by a processor, the data processing method provided by any of the above embodiments can be implemented. For example, Figures 1 to 3 The data processing method shown is as follows.

[0156] The computer storage medium is a non-transitory storage medium.

[0157] The technical solutions described in the embodiments of the present disclosure can be arbitrarily combined without conflict.

[0158] In the several embodiments provided in the present disclosure, it should be understood that the disclosed methods and intelligent devices can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0159] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0160] In addition, all functional units in the embodiments of the present disclosure may be integrated into a second processing unit, or each unit may be separately configured as a unit, or N or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0161] The above description is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the present disclosure, and they should all be covered by the protection scope of the present disclosure.

Claims

1. A data processing method, characterized in that: include: Determining an access pattern of the first user based on access information of the first user to data pages in a first-level cache, wherein the first-level cache is set in a memory of a service process of the first user; Based on the access pattern of the first user, data of a target data page whose access popularity meets a preset condition is transferred to a target partition of the secondary cache; wherein the secondary cache is provided on a hard disk; the target partition is a cache partition corresponding to the access pattern of the first user among multiple cache partitions of the secondary cache, and different access patterns correspond to different cache partitions; Determining the access pattern of the first user based on the access information of the first user to the data pages in the first-level cache includes: determining an access pattern value of the first user based on the access information of the first user to the data pages in the first-level cache; determining the access pattern of the first user based on the access pattern value; wherein the access information includes at least one of the following: the type of data pages accessed by the first user in the first-level cache; the access frequency of the first user to each type of data page; and a first weight of each type of data page; The method also includes: determining a first cache hit rate and a cache capacity of each cache partition of the secondary cache; determining a second cache hit rate of the secondary cache based on the first cache hit rate and the cache capacity; and adjusting the cache partition of the secondary cache based on the first cache hit rate and the second cache hit rate.

2. The method according to claim 1, characterized in that The types of data pages include: Index pages, system pages, metadata pages and / or content pages; wherein the first weight of the metadata page is greater than the first weight of the system page; The first weight of the system page is greater than the first weight of the index page; The first weight of the index page is greater than the first weight of the content page; The method further comprises: The first weight of each type of data page is determined according to the space ratio occupied by each type of data page in the first-level cache and the second-level cache, the total access frequency of each type of data page in the first-level cache and the second-level cache, and the preset second weight.

3. The method according to claim 1, characterized in that The adjusting the cache partition of the secondary cache according to the first cache hit rate and the second cache hit rate includes: When the first cache hit rate of the P1th cache partition is higher than or equal to the second cache hit rate, one or more data pages with the highest access popularity in the P1th cache partition are divided into the P2th cache partition, wherein the first cache hit rate of the P2th cache partition is lower than the second cache hit rate; the P1th cache partition is different from the P2th cache partition, and the P1th cache partition and the P2th cache partition are both less than or equal to A positive integer; The total number of cache partitions included in the secondary cache.

4. The method according to claim 1, wherein The method further comprises: Setting a cache queue in the first-level cache; Determining access popularity based on access information of each type of data page in the first-level cache and / or predicted access probability within a preset future time period; Determining the position of each data page in the cache queue according to the access popularity; wherein the access popularity of the data page at the head of the cache queue is lower than the access popularity of the data page at the tail of the cache queue; The step of transferring the data of the data page whose access popularity satisfies a preset condition to a target partition of the secondary cache according to the access pattern of the first user includes: When the first-level cache overflows, the data page at the head of the cache queue is transferred to the target partition of the second-level cache according to the access pattern of the first user.

5. The method according to claim 4, characterized in that The transferring of the data page at the head of the cache queue to the target partition of the secondary cache according to the access pattern of the first user includes: When the access popularity of the data page at the head of the cache queue is higher than a preset value, the data page at the head of the cache queue is transferred to the target partition of the secondary cache according to the access pattern of the first user.

6. The method according to claim 5, characterized in that The method further comprises: When the access popularity of the data page at the head of the cache queue is lower than or equal to the preset value, determining whether the data page at the head of the cache queue has been updated; When there is an update, the data page at the head of the cache queue is written to the database disk; In the absence of update, the data page at the head of the cache queue is discarded.

7. A data processing device, characterized in that: include: a determination module, configured to determine an access mode value of the first user based on access information of the first user to the data page in the first-level cache; Determining an access pattern of the first user based on the access pattern value; wherein the first-level cache is set in a memory of a service process of the first user; and the access information includes at least one of the following: a type of data page accessed by the first user in the first-level cache; a frequency of access by the first user to each type of data page; and a first weight of each type of data page; a transfer module, configured to transfer data of the data page whose access popularity satisfies a preset condition to a target partition of the secondary cache according to the access pattern of the first user; wherein the secondary cache is provided on a hard disk; and the target partition is a cache partition corresponding to the access pattern of the first user among multiple cache partitions of the secondary cache, wherein different access patterns correspond to different cache partitions; a cache partition module, configured to determine a first cache hit rate and a cache capacity of each cache partition of the secondary cache; a second cache hit rate module, configured to determine a second cache hit rate of the secondary cache according to the first cache hit rate and the cache capacity; An adjustment module is configured to adjust the cache partition of the secondary cache according to the first cache hit rate and the second cache hit rate.

8. An electronic device, characterized in that: The electronic device comprises: Memory; A processor, connected to the memory, is configured to implement the data processing method provided in any one of claims 1 to 6 by executing computer-executable instructions stored in the memory.

9. A computer storage medium, characterized in that The computer storage medium stores computer-executable instructions; after the computer-executable instructions are executed by the processor, the data processing method provided in any one of claims 1 to 6 can be implemented.

Images