Method and system for realizing rapid retrieval of mass data based on multi-level cache

Abstract

The method comprises the steps that one or more Redis instances are deployed, and each Redis instance represents a cache level; dynamically adjusting the cache level of the data according to the data access frequency and importance, which comprises the following steps of: storing the data of which the data access frequency and importance can be judged in advance by using a specified key value and a cluster, and calculating a storage level corresponding to Hash matching through a Hash algorithm; for data which cannot be predicted in advance, the same key value and the scheduling center are used for setting cache clusters and levels. According to the method and the device, the hot data is stored in the cache of a higher level, and the cache level of the data is dynamically adjusted according to the data access frequency and importance, so that efficient data retrieval and quick response time can be realized.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and in particular to a method and system for realizing rapid retrieval of massive data based on multi-level cache. Background Art

[0002] With the rapid development of information technology, the amount of data has exploded, and all walks of life are facing the challenge of processing and retrieving massive amounts of data. In the era of big data, quickly and accurately obtaining the required data has become a key requirement. However, traditional data storage and retrieval methods often seem to be unable to cope with massive amounts of data, and there are problems such as slow data access speed and poor system performance.

[0003] Especially in some application scenarios with high real-time requirements, such as e-commerce, financial transactions, and Internet services, data access speed and system performance directly affect user experience and the normal operation of the business. Traditional single cache systems are difficult to meet these high-performance requirements because they cannot effectively handle fast access to hot data, nor can they flexibly manage caches based on the access frequency and importance of data.

[0004] In addition, as the business continues to develop, the amount of data and access volume will continue to increase, which requires the cache system to have good scalability and be able to dynamically adjust the cache capacity and performance to adapt to the changing business needs. However, traditional cache systems often have many limitations in terms of capacity expansion, such as complex expansion process, high cost, and great impact on system performance.

[0005] Therefore, in order to improve data access speed and system performance and meet the needs of rapid retrieval of massive data, a more efficient, flexible and scalable multi-level cache solution is urgently needed. Summary of the invention

[0006] In order to solve the above technical problems existing in the prior art, the present invention proposes a method and system for realizing rapid retrieval of massive data based on multi-level cache to solve the above technical problems.

[0007] According to a first aspect of the present invention, a method for realizing rapid retrieval of massive data based on multi-level cache is proposed.

[0008] S1: deploy one or more Redis instances, each of which represents a cache level;

[0009] S2: Dynamically adjust the data cache level based on data access frequency and importance, including:

[0010] S21: For data whose access frequency and importance can be determined in advance, use the specified key value and cluster for storage, calculate the hash matching corresponding storage level through the hash algorithm, and read the data from the cluster with high priority of the same level attribute first when accessing the data;

[0011] S22: For data that cannot be predicted in advance, the same key value and scheduling center are used to set the cache cluster and level. Specifically, MD5 is used to calculate the key value of the data storage, and the cluster controller uniformly schedules the storage priority level during subsequent access accumulation. The data is evenly stored in the cache cluster through the calculated MD5 value, and the vertical automatic expansion method is used to achieve cluster expansion. Data access is performed from the cluster with the highest priority level due to the unified scheduling of the cluster controller.

[0012] In some specific embodiments, the initial default setting in S1 is N clusters, the initial value of N is 2, and N+1 automatic expansion is performed in response to parameters exceeding the set thresholds. The automatic expansion includes horizontal and vertical automatic expansion. The horizontal expansion is the expansion of attributes at the same level but with different priorities, and the vertical expansion is the expansion of attributes at different levels.

[0013] In some specific embodiments, in S21, before storing the data in the cache, a hash function is used to generate a hash value with a fixed prefix, and the cache level is determined by a specific prefix of the hash value to quickly locate the cache cluster.

[0014] In some specific embodiments, when the cache frequency and timeliness cannot be met, the instance is dynamically expanded, specifically including:

[0015] For data with a pre-fixed priority level, a method of vertical fixed cluster and horizontal dynamic expansion is adopted. The controller monitors the number of accesses and response time of each record. When one of them reaches the set threshold, it migrates cache data between clusters. When no higher priority cache is available at the same level, the controller automatically expands the new cluster and performs data migration synchronously based on the CPU memory and load conditions of the cluster to meet the conditions for establishing a new cluster.

[0016] For data with unfixed priority levels, a vertical expansion method is adopted. The controller monitors the number of accesses and response time of each record. When one of them reaches the set threshold, it migrates the cache data between clusters. When no higher priority cache is available at different levels, the controller automatically expands the new cluster and performs data migration synchronously based on the CPU memory and load conditions of the cluster to meet the conditions for establishing a new cluster.

[0017] In some specific embodiments, in response to data migration reaching a set access frequency or response time in the cluster controller, the key-value cache is migrated to a higher-level cache. After the data migration is completed, the cluster controller deletes the data of the old cluster. If a problem occurs during the migration process, it is rolled back to the atomic state.

[0018] In some specific embodiments, when cache space is insufficient or memory is insufficient, data is eliminated based on access frequency and importance, specifically including: pre-setting cache space ratio and memory operation status. When one of the two is exceeded, the cluster controller uses the access frequency of key-value cache data as an indicator to perform data deletion operations on the cluster with the lowest priority. The deletion mechanism clears all caches that are less than the access frequency value pre-set by the cluster controller.

[0019] In some specific embodiments, the cluster controller monitors the response time and memory CPU usage of each cluster, and synchronously records the access concurrency of the cluster, automatically calculates the daily concurrency, CPU and memory usage and adjusts the parameters to meet the pre-configured range of the cluster controller, and downloads the daily concurrency minimum time to reallocate and adjust the cluster memory CPU.

[0020] According to a second aspect of the present invention, a computer-readable storage medium is provided, on which one or more computer programs are stored. When the one or more computer programs are executed by a computer processor, the above method is implemented.

[0021] According to a third aspect of the present invention, a system for realizing rapid retrieval of massive data based on multi-level cache is proposed, comprising:

[0022] Deployment unit, configured to deploy one or more Redis instances, each Redis instance represents a cache level;

[0023] A data access unit is configured to dynamically adjust the cache level of data according to the frequency and importance of data access, including: for data whose access frequency and importance can be determined in advance, a specified key value and cluster are used for storage, and a hash algorithm is used to calculate the hash match corresponding to the storage level. When the data is accessed, the data is first read from a high-priority cluster with the same hierarchical attribute; for data that cannot be predicted in advance, the same key value and scheduling center are used to set the cache cluster and level, specifically including using MD5 to calculate the key value of the data storage, and the cluster controller uniformly schedules the storage priority level during subsequent access accumulation, and evenly stores the data in the cache cluster through the calculated MD5 value, and uses a vertical automatic expansion method to achieve cluster expansion. Data access is performed from a cluster with a high priority level due to the unified scheduling of the cluster controller.

[0024] In some specific embodiments, the initial default setting in the deployment unit is N clusters, the initial value of N is 2, and N+1 automatic expansion is performed in response to parameters exceeding the set thresholds. The automatic expansion includes horizontal and vertical automatic expansion. The horizontal expansion is the expansion of attributes at the same level but with different priorities, and the vertical expansion is the expansion of attributes at different levels.

[0025] In some specific embodiments, a dynamic expansion unit is further included, configured to dynamically expand the instance when the cache frequency and timeliness cannot be met, specifically including:

[0026] For data with a pre-fixed priority level, a method of vertical fixed cluster and horizontal dynamic expansion is adopted. The controller monitors the number of accesses and response time of each record. When one of them reaches the set threshold, it migrates cache data between clusters. When no higher priority cache is available at the same level, the controller automatically expands the new cluster and performs data migration synchronously based on the CPU memory and load conditions of the cluster to meet the conditions for establishing a new cluster.

[0027] For data with unfixed priority levels, a vertical expansion method is adopted. The controller monitors the number of accesses and response time of each record. When one of them reaches the set threshold, it migrates the cache data between clusters. When no higher priority cache is available at different levels, the controller automatically expands the new cluster and performs data migration synchronously based on the CPU memory and load conditions of the cluster to meet the conditions for establishing a new cluster.

[0028] In some specific embodiments, a data migration unit is also included, which is configured to migrate the key-value cache to a higher-level cache in response to the data migration reaching a set access frequency or response time in the cluster controller. After the data migration is completed, the cluster controller deletes the data of the old cluster. If a problem occurs during the migration process, it rolls back to the atomic state.

[0029] In some specific embodiments, a cache elimination unit is also included, which is configured to eliminate data according to the access frequency and importance of the data when the cache space is insufficient or the memory is insufficient. Specifically, the cache space ratio and the memory operation status are pre-set. When one of the two is exceeded, the cluster controller uses the access frequency of the key-value cache data as an indicator to perform data deletion operations on the cluster with the lowest priority. The deletion mechanism clears all caches that are less than the access frequency value pre-set by the cluster controller.

[0030] In some specific embodiments, a monitoring and tuning unit is also included, which is configured for the cluster controller to monitor the response time and memory CPU usage of each cluster, and synchronously record the access concurrency of the cluster, automatically calculate the daily concurrency, CPU and memory usage and adjust the parameters to meet the pre-configured range of the cluster controller, and download the daily concurrency minimum time to reallocate and adjust the cluster memory CPU.

[0031] The present invention proposes a method and system for realizing rapid retrieval of massive data based on multi-level cache, including one or more cache levels, each of which uses Redis instances for data storage and management. By storing hot data in a higher-level cache and dynamically adjusting the cache level of the data according to the frequency and importance of data access, efficient data retrieval and fast response time can be achieved. Specifically, the following effects are achieved:

[0032] 1. Improved data access speed and system performance, reducing the pressure on the back-end database;

[0033] 2. By dynamically adjusting the data cache level, efficient data retrieval and fast response time are achieved;

[0034] 3. Ensure data consistency and availability through reasonable data migration and elimination strategies;

[0035] 4. Through the monitoring and tuning mechanism, real-time monitoring and optimization of the cache system are achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate the embodiments and together with the description are used to explain the principles of the present invention. Other embodiments and many expected advantages of the embodiments will be readily appreciated as they become better understood by reference to the following detailed description. Other features, objects and advantages of the present application will become more apparent by reading the detailed description of the non-limiting embodiments made with reference to the following drawings:

[0037] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0038] Figure 2 This is a flow chart of a method for realizing rapid retrieval of massive data based on multi-level cache according to an embodiment of the present application;

[0039] Figure 3 is a schematic diagram of capacity expansion of a specific embodiment of the present application;

[0040] Figure 4It is a data access flow diagram of a specific embodiment of the present application, which can determine the frequency and importance of data access in advance;

[0041] Figure 5 is a schematic diagram of a data access process that cannot be predicted in advance in a specific embodiment of the present application;

[0042] Figure 6 It is a flowchart of a specific embodiment of the present application for dynamically expanding the cache with a pre-fixed priority level;

[0043] Figure 7 It is a flowchart of a specific embodiment of the present application for dynamically expanding the cache with a pre-fixed priority level;

[0044] Figure 8 A specific embodiment of the present application is a flowchart of data migration;

[0045] Fig. 9 This is a system architecture diagram for realizing rapid retrieval of massive data based on multi-level cache according to an embodiment of the present application;

[0046] Fig.10 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application. DETAILED DESCRIPTION

[0047] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0048] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0049] Figure 1 An exemplary system architecture 100 is shown to which the method for implementing rapid retrieval of massive data based on multi-level cache according to an embodiment of the present application can be applied.

[0050] like Figure 1 As shown, system architecture 100 may include data server 101, network 102 and main server 103. Network 102 is used to provide a medium for a communication link between data server 101 and main server 103. Network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0051] The main server 103 may be a server that provides various services, such as a data processing server that processes information uploaded by the data server 101 .

[0052] It should be noted that the method for realizing rapid retrieval of massive data based on multi-level cache provided in the embodiment of the present application is generally executed by the main server 103. Accordingly, the system for realizing rapid retrieval of massive data based on multi-level cache is generally set in the main server 103.

[0053] It should be noted that the data server and the main server can be hardware or software. When it is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When it is software, it can be implemented as multiple software or software modules (such as software or software modules used to provide distributed services), or it can be implemented as a single software or software module.

[0054] It should be understood that Figure 1 The numbers of data servers, networks and main servers in the embodiment are only for illustration purposes. Any number of terminal devices, networks and servers may be provided as required.

[0055] Figure 2 FIG. 1 shows a flow chart of a method for realizing rapid retrieval of massive data based on multi-level cache according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0056] S201: deploy one or more Redis instances, each Redis instance represents a cache level.

[0057] In a specific embodiment, Figure 3 A schematic diagram of capacity expansion of a specific embodiment of the present application is shown, as shown in FIG. Figure 3 As shown, it includes at least two cache levels that can be dynamically expanded, and each level uses Redis instances for data storage and management. The initial default setting is N clusters, and the initial value of N is 2. In response to parameters exceeding the set thresholds, N+1 automatic expansion is performed. Automatic expansion includes horizontal and vertical automatic expansion. Horizontal expansion is the expansion of attributes at the same level but with different priorities, and vertical expansion is the expansion of attributes at different levels. Whether each node is dynamically expanded is uniformly monitored, scheduled and managed by the cluster controller.

[0058] S202: Dynamically adjust the cache level of data according to the data access frequency and importance, specifically including:

[0059] For data whose access frequency and importance can be determined in advance, use the specified key value and cluster for storage, calculate the hash matching corresponding to the storage level through the hash algorithm, and read the data from the cluster with the same hierarchical attribute with high priority first when accessing the data;

[0060] For data that cannot be predicted in advance, the same key value and scheduling center are used to set the cache cluster and level. Specifically, MD5 is used to calculate the key value of the data storage, and the cluster controller uniformly schedules the storage priority level during subsequent access accumulation. The data is evenly stored in the cache cluster through the calculated MD5 value, and the vertical automatic expansion method is used to achieve cluster expansion. Data access is performed from the cluster with the highest priority level due to the unified scheduling of the cluster controller.

[0061] In a specific embodiment, Figure 4 A data access flow diagram of a specific embodiment of the present application is shown, which can determine the frequency and importance of data access in advance. Figure 4 As shown, for data whose access frequency and importance can be determined in advance, the number of vertical clusters can be planned in advance and stored using specific key values ​​and clusters. For data key values ​​that can be predicted in advance, a hash algorithm is used to calculate the hash match corresponding to the storage level. The hash value protects a specific prefix to determine the cache level, which facilitates the rapid location of the cache cluster. Therefore, before storing the data in the cache, we use a hash value with a fixed prefix, which can be implemented using a hash function. The following is a simple Python example that uses the SHA-256 algorithm to generate a hash value that starts with a specific prefix:

[0062] from hashlib import sha256

[0063] def generate_hash_with_prefix(prefix,message):

[0064] #Connect the prefix and the message

[0065] message_with_prefix=prefix+message

[0066] #Calculate the hash value of the message after connection

[0067] hash_object=sha256(message_with_prefix.encode())

[0068] hex_dig=hash_object.hexdigest()

[0069] return hex_dig

[0070] #use

[0071] prefix="first_colony"

[0072] message="message"

[0073] hash_value=generate_hash_with_prefix(prefix,message)

[0074] print(hash_value)

[0075] In a specific embodiment, this method can be used to ensure that the generated hash value contains a specific prefix. Data can be accurately cached in a specific cluster, and it can also ensure whether the data is already cached when it is queried in a specific cluster. When accessing data, data is first read from a high-priority cluster with the same hierarchical attributes. If it does not meet the requirements, a low-level cluster is accessed until all clusters are accessed. For data that needs to be cached synchronously, data is written to the cluster from the lowest level, and a cluster controller is used to perform unified management and scheduling between clusters in the later stage. When the data cluster does not meet the access requirements, the channel scheduler automatically expands the capacity of the new cluster and migrates the data to ensure the timeliness of data access.

[0076] In a specific embodiment, Figure 5 FIG. 4 shows a schematic diagram of a data access process that cannot be predicted in advance according to a specific embodiment of the present application. Figure 5 As shown, data whose priority cannot be predicted in advance uses MD5 to calculate the key value of data storage, and the cluster controller uniformly schedules the storage priority level during the subsequent access accumulation process. The generated MD5 code is as follows:

[0077]

[0078] data = f.read(4096) #Read 4096 bytes each time. Adjusting the buffer size according to the file size can improve efficiency.

[0079] if not

[0080] break

[0081] md5_hash.update(data)#Update hash value

[0082] return md5_hash.hexd igest()#Returns the hash value in hexadecimal string format

[0083] In a specific embodiment, data is evenly stored in a cache cluster through the calculated MD5 value. For unpredictable data, a vertical automatic expansion method is used to achieve cluster expansion. Data access is performed from a cluster with a high priority level. Due to the same scheduling of the cluster controller, the higher the priority level of the cluster, the less data it has, and the cluster controller dynamically allocates memory and CPU to ensure faster access speed.

[0084] In a specific embodiment, the present application also proposes a cache dynamic expansion strategy for dynamically expanding instances when the cache frequency and timeliness cannot be met. The cluster controller plays a decisive role in the entire cluster. The controller monitors the access frequency, number of accesses, and cluster memory cpu load of each record in real time, and makes dynamic adjustments based on the set thresholds. In terms of access frequency and timeliness, any one of the thresholds is taken as the basis, and the key value migration and dynamic expansion of the new cluster are performed based on the CPU memory and cluster load.

[0085] In a specific embodiment, Figure 6 A schematic diagram of a process flow of dynamically expanding cache capacity for a pre-fixed priority level according to a specific embodiment of the present application is shown. Figure 6 As shown in the figure, for data with pre-fixed priority levels, we use the method of vertical fixed cluster and horizontal dynamic expansion. The controller monitors the number of accesses and response time of each record. When one of them reaches the set threshold, it will migrate the cache data between clusters. When there is no higher priority cache available at the same level, the controller will automatically expand the new cluster and migrate the data synchronously based on the CPU memory and load conditions of the cluster to meet the conditions for establishing a new cluster.

[0086] In a specific embodiment, Figure 7 A schematic diagram of a process flow of dynamically expanding cache capacity with unfixed priority levels according to a specific embodiment of the present application is shown. Figure 7 As shown in the figure, for data with unfixed priority levels in advance, a vertical expansion method is adopted. The controller monitors the number of accesses and response time of each record. When one of them reaches the set threshold, it migrates the cache data between clusters. When no higher priority cache is available at different levels, the controller automatically expands the new cluster and performs data migration synchronously according to the CPU memory and load conditions of the cluster to meet the conditions for establishing a new cluster.

[0087] In a specific embodiment, the present application also proposes a data migration strategy for migrating data between different cache levels to maintain data consistency and availability. Figure 8 A specific embodiment of the present application is shown in the flowchart of data migration, such as Figure 8As shown, when data migration reaches the set access frequency or response time in the cluster controller, the key-value cache is migrated to a higher-level cache. When data migration is completed, the cluster controller deletes the data of the old cluster. If a problem occurs during the migration process, it is rolled back to the atomic state to ensure data consistency and availability.

[0088] In a specific embodiment, the present application also proposes a cache elimination strategy, which is used to eliminate data according to the access frequency and importance of data when the cache space is insufficient or the memory is insufficient. The cluster controller is responsible for the execution of the cache elimination strategy. It sets the cache space ratio and the memory operation status in advance. When one of the two is exceeded, the cluster controller performs data deletion operations on the cluster with the lowest priority based on the access frequency of all key-value cache data as an indicator. The deletion mechanism clears all caches that are less than the access frequency value set in advance by the cluster controller.

[0089] In a specific embodiment, the present application also provides a monitoring and tuning mechanism for real-time monitoring of system performance indicators, and adjusting cache configuration and parameters as well as hardware-level adjustments as needed. The cluster controller monitors the response time and memory cpu usage of each cluster, and synchronously records the access concurrency of the cluster, and automatically calculates the daily concurrency, CPU and memory usage and adjustment parameters within the range of the cluster controller configuration in advance, downloads the daily concurrency minimum time to redistribute the cluster memory cpu, and ensures sufficient cluster resources.

[0090] In a specific embodiment, the above method is specifically implemented as follows:

[0091] 1. Deploy one or more Redis instances, each representing a cache level. For example, you can deploy three Redis instances, one for the first-level cache, one for the second-level cache, and one for the third-level cache;

[0092] 2. Assign a unique key to each data item and store it in the appropriate cache level based on its access frequency and importance. For example, hot data can be stored in a high-level cache, while infrequently used data can be stored in a low-level cache;

[0093] 3. When the client requests data, it first searches for the data in the highest level cache. If found, the result is returned directly; otherwise, it searches in the next level cache in turn until the data is found or the lowest level cache is reached;

[0094] 4. Dynamically adjust the cache level of data based on data access frequency and importance. For example, if the access frequency of a data item increases, it can be migrated from a lower level cache to a higher level cache;

[0095] 5. When the cache space is insufficient, data is eliminated based on access frequency and importance. For example, data items with low access frequency can be eliminated first to free up space for more important data items;

[0096] 6. Monitor system performance indicators in real time through monitoring and tuning mechanisms, and adjust cache configuration and parameters as needed. For example, the size and number of caches at all levels can be dynamically adjusted according to system load conditions.

[0097] The method of the present application improves data access speed and system performance, and reduces the pressure on the back-end database; by dynamically adjusting the cache level of data, efficient data retrieval and fast response time are achieved; through reasonable data migration and elimination strategies, data consistency and availability are guaranteed; through monitoring and tuning mechanisms, real-time monitoring and optimization of the cache system are achieved.

[0098] Fig. 9 A system architecture diagram of a multi-level cache based on an embodiment of the present application for realizing rapid retrieval of massive data is shown as follows: Fig. 9 As shown, the system includes a deployment unit 901 and a data access unit 902, wherein the deployment unit 901 is configured to deploy one or more Redis instances, each Redis instance represents a cache level; the data access unit 902 is configured to dynamically adjust the cache level of the data according to the data access frequency and importance, including: for data whose data access frequency and importance can be judged in advance, use the specified key value and cluster for storage, calculate the hash match corresponding to the storage level through the hash algorithm, and when the data is accessed, first read the data from the cluster with high priority attributes of the same level; for data that cannot be predicted in advance, use the same key value and scheduling center to set the cache cluster and level, specifically including using MD5 to calculate the key value of the data storage, and the cluster controller uniformly schedules the storage priority level during the subsequent access accumulation process, and evenly stores the data in the cache cluster through the calculated MD5 value, and adopts the vertical automatic expansion method to achieve cluster expansion, and accesses the data from the cluster with high priority level due to the unified scheduling of the cluster controller.

[0099] In a specific embodiment, it also includes a dynamic expansion unit, which is configured to dynamically expand the instance when the cache frequency and timeliness cannot be met, specifically including: for data with a pre-fixed priority level, a method of vertical fixed cluster horizontal dynamic expansion is adopted, and the controller monitors the number of accesses and response time of each record. When one of them reaches a set threshold, migration between cache data clusters is performed. When no higher priority cache is available at the same level, the controller automatically expands the new cluster according to the CPU memory and load conditions of the cluster to meet the conditions for establishing a new cluster and simultaneously migrates data; for data with an unfixed priority level in advance, a vertical expansion method is adopted, and the controller monitors the number of accesses and response time of each record. When one of them reaches a set threshold, migration between cache data clusters is performed. When no higher priority cache is available at different levels, the controller automatically expands the new cluster according to the CPU memory and load conditions of the cluster to meet the conditions for establishing a new cluster and simultaneously migrates data.

[0100] In a specific embodiment, it also includes a data migration unit, which is configured to migrate the key-value cache to a higher-level cache in response to the data migration reaching a set access frequency or response time in the cluster controller. After the data migration is completed, the cluster controller deletes the data of the old cluster. If a problem occurs during the migration process, it rolls back to the atomic state.

[0101] In a specific embodiment, it also includes a cache elimination unit, which is configured to eliminate data according to the access frequency and importance of the data when the cache space is insufficient or the memory is insufficient. Specifically, it includes: pre-setting the cache space ratio and the memory operation status. When one of the two is exceeded, the cluster controller uses the access frequency of the key-value cache data as an indicator to perform data deletion operations on the cluster with the lowest priority. The deletion mechanism clears all caches that are less than the access frequency value pre-set by the cluster controller.

[0102] In a specific embodiment, it also includes a monitoring and tuning unit, which is configured for the cluster controller to monitor the response time and memory CPU usage of each cluster, and synchronously record the access concurrency of the cluster, automatically calculate the daily concurrency, CPU and memory usage and adjust the parameters to meet the pre-configured range of the cluster controller, and download the daily concurrency minimum time to reallocate and adjust the cluster memory CPU.

[0103] Reference below Fig.10 , which shows a schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application. Fig.10 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0104] like Fig.10As shown, the computer system includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage part 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the system 1000 are also stored. The CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0105] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0106] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the central processing unit (CPU) 1001, the above functions defined in the method of the present application are executed. It should be noted that the computer-readable storage medium of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wireline, optical cable, RF, etc., or any suitable combination of the foregoing.

[0107] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0108] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0109] The modules involved in the embodiments of the present application may be implemented by software or by hardware.

[0110] As another aspect, the present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or it may exist alone and not be assembled into the electronic device. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed by the electronic device, the electronic device: deploys one or more Redis instances, each Redis instance represents a cache level; dynamically adjusts the cache level of the data according to the frequency and importance of data access, including: for data whose access frequency and importance can be judged in advance, use the specified key value and cluster for storage, calculate the hash matching corresponding storage level through the hash algorithm, and read the data from the cluster with high priority attribute at the same level when the data is accessed; for data that cannot be predicted in advance, use the same key value and scheduling center to set the cache cluster and level, specifically including using MD5 to calculate the key value of the data storage, and the cluster controller uniformly schedules the storage priority level in the subsequent access accumulation process, and evenly stores the data in the cache cluster through the calculated MD5 value, and adopts the vertical automatic expansion method to realize the expansion of the cluster, and for data access, it is performed from the cluster with high priority level, due to the unified scheduling of the cluster controller.

[0111] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other.

Claims

1. A method for realizing rapid retrieval of massive data based on multi-level cache, characterized in that: S1: deploy one or more Redis instances, each of which represents a cache level; S2: Dynamically adjusting the cache level of the data according to the data access frequency and importance, including: S21: For data whose access frequency and importance can be determined in advance, use the specified key value and cluster for storage, calculate the hash matching corresponding storage level through the hash algorithm, and read the data from the cluster with high priority of the same level attribute first when accessing the data; S22: For data that cannot be predicted in advance, the same key value and scheduling center are used to set the cache cluster and level. Specifically, MD5 is used to calculate the key value of the data storage, and the cluster controller uniformly schedules the storage priority level during subsequent access accumulation. The data is evenly stored in the cache cluster through the calculated MD5 value, and the vertical automatic expansion method is used to achieve cluster expansion. Data access is performed from the cluster with the highest priority level due to the unified scheduling of the cluster controller.

2. The method for realizing rapid retrieval of massive data based on multi-level cache according to claim 1, characterized in that: The initial default setting in S1 is N clusters, and the initial value of N is 2. In response to parameters exceeding the set thresholds, N+1 automatic expansion is performed. The automatic expansion includes horizontal and vertical automatic expansion. The horizontal expansion is the expansion of attributes at the same level but different priorities, and the vertical expansion is the expansion of attributes at different levels.

3. The method for realizing rapid retrieval of massive data based on multi-level cache according to claim 1, characterized in that: In S21, before storing the data in the cache, a hash function is used to generate a hash value with a fixed prefix, and the cache level is determined by the specific prefix of the hash value to quickly locate the cache cluster.

4. The method for realizing rapid retrieval of massive data based on multi-level cache according to claim 1, characterized in that: When the cache frequency and timeliness cannot be met, the instance is dynamically expanded, including: For data with a pre-fixed priority level, a method of vertical fixed cluster and horizontal dynamic expansion is adopted. The controller monitors the number of accesses and response time of each record. When one of them reaches the set threshold, it migrates cache data between clusters. When no higher priority cache is available at the same level, the controller automatically expands the new cluster and performs data migration synchronously based on the CPU memory and load conditions of the cluster to meet the conditions for establishing a new cluster. For data with unfixed priority levels, a vertical expansion method is adopted. The controller monitors the number of accesses and response time of each record. When one of them reaches the set threshold, it migrates the cache data between clusters. When no higher priority cache is available at different levels, the controller automatically expands the new cluster and performs data migration synchronously based on the CPU memory and load conditions of the cluster to meet the conditions for establishing a new cluster.

5. The method for realizing rapid retrieval of massive data based on multi-level cache according to claim 4, characterized in that: In response to data migration reaching a set access frequency or response time in the cluster controller, the key-value cache is migrated to a higher-level cache. After the data migration is completed, the cluster controller deletes the data of the old cluster. If a problem occurs during the migration process, it is rolled back to the atomic state.

6. The method for realizing rapid retrieval of massive data based on multi-level cache according to claim 1, characterized in that: When the cache space is insufficient or the memory is insufficient, the data is eliminated based on its access frequency and importance, including: pre-setting the cache space ratio and the memory operation status. When one of the two is exceeded, the cluster controller uses the access frequency of the key-value cache data as an indicator to delete data for the cluster with the lowest priority. The deletion mechanism clears all caches that are less than the access frequency value pre-set by the cluster controller.

7. The method for realizing rapid retrieval of massive data based on multi-level cache according to claim 1, characterized in that: The cluster controller monitors the response time and memory CPU usage of each cluster, and synchronously records the access concurrency of the cluster, automatically calculates the daily concurrency, CPU and memory usage and adjusts the parameters to meet the pre-configured range of the cluster controller, and downloads the daily concurrency minimum time to reallocate and adjust the cluster memory CPU.

8. A computer-readable storage medium having one or more computer programs stored thereon, characterized in that: When the one or more computer programs are executed by a computer processor, the method according to any one of claims 1 to 7 is implemented.

9. A system for realizing rapid retrieval of massive data based on multi-level cache, characterized in that: include: A deployment unit, configured to deploy one or more Redis instances, each of which represents a cache level; A data access unit is configured to dynamically adjust the cache level of data according to the frequency and importance of data access, including: for data whose access frequency and importance can be determined in advance, using a specified key value and cluster for storage, calculating the hash match corresponding to the storage level through a hash algorithm, and when the data is accessed, first reading the data from a high-priority cluster with the same hierarchical attribute; for data that cannot be predicted in advance, using the same key value and scheduling center to set the cache cluster and level, specifically including using MD5 to calculate the key value of the data storage, and having the cluster controller uniformly schedule the storage priority level during subsequent access accumulation, evenly storing the data in the cache cluster through the calculated MD5 value, and using a vertical automatic expansion method to achieve cluster expansion, and performing data access from a cluster with a high priority level due to the unified scheduling of the cluster controller.

10. The system for realizing rapid retrieval of massive data based on multi-level cache according to claim 9, characterized in that: The deployment unit is initially set to N clusters by default, with an initial value of N being 2. In response to parameters exceeding various set thresholds, N+1 automatic expansion is performed. The automatic expansion includes horizontal and vertical automatic expansion. The horizontal expansion is expansion of attributes at the same level but with different priorities, and the vertical expansion is expansion of attributes at different levels.

11. The system for realizing rapid retrieval of massive data based on multi-level cache according to claim 9, characterized in that: It also includes a dynamic expansion unit, which is configured to dynamically expand the instance when the cache frequency and timeliness cannot be met, including: For data with a pre-fixed priority level, a method of vertical fixed cluster and horizontal dynamic expansion is adopted. The controller monitors the number of accesses and response time of each record. When one of them reaches the set threshold, it migrates cache data between clusters. When no higher priority cache is available at the same level, the controller automatically expands the new cluster and performs data migration synchronously based on the CPU memory and load conditions of the cluster to meet the conditions for establishing a new cluster. For data with unfixed priority levels, a vertical expansion method is adopted. The controller monitors the number of accesses and response time of each record. When one of them reaches the set threshold, it migrates the cache data between clusters. When no higher priority cache is available at different levels, the controller automatically expands the new cluster and performs data migration synchronously based on the CPU memory and load conditions of the cluster to meet the conditions for establishing a new cluster.

12. The system for realizing rapid retrieval of massive data based on multi-level cache according to claim 11, characterized in that: It also includes a data migration unit, which is configured to migrate the key-value cache to a higher-level cache in response to the data migration reaching a set access frequency or response time in the cluster controller. After the data migration is completed, the cluster controller deletes the data of the old cluster. If a problem occurs during the migration process, it rolls back to the atomic state.

13. The system for realizing rapid retrieval of massive data based on multi-level cache according to claim 9, characterized in that: It also includes a cache elimination unit, which is configured to eliminate data according to the access frequency and importance of the data when the cache space is insufficient or the memory is insufficient. Specifically, it includes: pre-setting the cache space ratio and the memory operation status. When one of the two is exceeded, the cluster controller uses the access frequency of the key-value cache data as an indicator to perform data deletion operations on the cluster with the lowest priority. The deletion mechanism clears all caches that are less than the access frequency value pre-set by the cluster controller.

14. The system for realizing rapid retrieval of massive data based on multi-level cache according to claim 9, characterized in that: It also includes a monitoring and tuning unit, which is configured for the cluster controller to monitor the response time and memory CPU usage of each cluster, and synchronously record the access concurrency of the cluster, automatically calculate the daily concurrency, CPU and memory usage and adjust the parameters to meet the pre-configured range of the cluster controller, and download the daily concurrency minimum time to reallocate and adjust the cluster memory CPU.

Images