Bidirectional expansion block multi-source cache management system and method

Abstract

The invention relates to the technical field of cache management, and discloses a two-way expansion block multi-source cache management system and method, and the cache management system is configured to receive data, store the data into a cache region according to the characteristics of different data, set a corresponding elimination strategy type for the cache region, and perform cache life cycle management on the data. The cache region comprises a structured cache region and an unstructured cache region, the structured cache region comprises a hotspot cache region, and the unstructured cache region comprises a local disk cache region; the cache management system comprises a plurality of cluster nodes, the cache management system checks cache occupied space at intervals, when it is checked that the cache occupied space reaches a specified threshold value, the cluster nodes are used for cluster multi-node expansion and node multi-level expansion, and hierarchical cache management is adopted for the cluster nodes.

Description

Technical Field

[0001] The present invention relates to the technical field of data cache hierarchical management, and particularly to a bidirectionally expandable block-based multi-source cache management system and method. Background Art

[0002] In recent years, with the rapid development of networking technologies and the explosive growth of data volume, higher requirements have been put forward for data storage and usage. Especially in the fields of cloud computing, big data analysis, real-time data processing, etc., efficient data cache management technology has become a key factor in improving system performance. Currently, traditional cache systems have experienced a development process from simple memory caches to distributed caches and then to multi-level caches. For example, Memcached and Redis often accelerate data processing by providing high-speed memory access. However, these systems usually have a single data source for caching, fail to meet business requirements in data classification and management, lack flexibility and scalability, and have low security in the process of caching data transmission.

[0003] Existing cache management systems usually adopt a fixed cache structure and a unidirectional data flow mechanism. The product structures of these systems mainly include cache servers and cache clients. For example, Memcached is a widely used distributed memory object cache system that stores data dispersedly on multiple servers. Redis is a memory database that supports multiple data structures, provides rich data operation commands, and supports data persistence. The working principle of these systems is to copy frequently accessed data from the main storage to the cache. When the data is accessed, it is preferentially read from the cache to reduce the number of accesses to the main storage system, thereby improving data access speed and system performance; if the data is not in the cache, it is read from the main storage and the cache is updated. However, the disadvantages and deficiencies of these existing technologies are mainly reflected in the following aspects:

[0004] 1. Unclear classification and single partitioning: Traditional cache systems have a single partitioning, such as:

[0005] Memory and hard disk; there is no clear classification rule, and only classification is based on data types, such as: strings, hashes, lists, sets. It is impossible to perform hierarchical classification according to the attributes or characteristics of the business system, and it is impossible to perform adaptive caching according to data characteristics.

[0006] 2. Low data security: When traditional cache systems interact with data, they often use plaintext transmission, and security cannot be guaranteed.

[0007] 3. Imperfect lifecycle management: Traditional cache eviction policies are basically implemented based on LRU and LUF, which are only limited to the recent access time and usage frequency, with low flexibility. They involve less in the lifecycle scheduling of data and have no efficient buffering and persistence mechanisms. Once the cached data becomes invalid, it is not stored in the buffer but immediately deleted, resulting in unstable data in memory or the problem of being unable to be stored for a long time.

[0008] 4. Unidirectional and cumbersome expansion: Traditional cache dynamic expansion is the addition of unidirectional horizontal nodes. The expansion process is cumbersome, often requiring manual intervention, with low security, being unable to adapt well to dynamic business requirements and data growth, and being difficult to manage cluster nodes.

[0009] 5. Insufficient multi-source data integration ability: Traditional cache systems are difficult to effectively process and integrate data from different data sources in a multi-source data environment, restricting their application in complex data processing scenarios and resulting in low resource utilization and data integrity. Summary of the Invention

[0010] In order to solve the problems of unclear classification, single partition, low data transmission security, imperfect lifecycle management, unidirectional and cumbersome expansion process, single data source and insufficient data processing ability in traditional cache systems, this application proposes a bidirectionally extended block-based multi-source cache management system and method.

[0011] According to one aspect of the present invention, a bidirectionally extended block-based multi-source cache management system is proposed. The cache management system is configured to receive data, store the data in a cache area according to the characteristics of different data, set corresponding eviction policy types for the cache area, and perform cache lifecycle management on the data. The cache area includes a structured cache area and an unstructured cache area. The structured cache area includes a hot cache area, and the unstructured cache area includes a local disk cache area. The cache management system includes multiple cluster nodes. The cache management system periodically checks the cache occupied space. When it is detected that the cache occupied space reaches a specified threshold, the cluster nodes are used for cluster multi-node expansion and node multi-level expansion. The cluster nodes adopt hierarchical cache management.

[0012] Preferably, setting corresponding eviction policy types specifically includes:

[0013] The cache management system sets the unstructured cache area to an expiration eviction policy type or a priority eviction policy type;

[0014] The cache management system sets the structured cache area to an expiration eviction policy type, a priority eviction policy type, a memory eviction policy type, or a hot eviction policy type.

[0015] Further preferably, the cache life cycle management of the data specifically includes:

[0016] When the cache area is of the expired elimination policy type or the priority elimination policy type, the cache management system directly destroys the data after detecting that the data stored in the cache area triggers the critical condition;

[0017] When the cache area is of the memory elimination policy type or the hot spot elimination policy type, after the cache management system detects that the data stored in the cache area triggers the critical condition, it transfers the data stored in the cache area from their respective cache areas to the buffer area and sets the corresponding expiration duration. When the expiration duration arrives, it then transfers the data transferred to the buffer area to the local disk cache area and sets it to the expired elimination policy type, and destroys the data after triggering the critical condition.

[0018] Further preferably, the cache life cycle management of the data specifically includes:

[0019] The cache management system is configured to automatically switch the cache levels of the data in the hot spot cache area and the local disk cache area, set the condition threshold for each hot spot cache area, and when it detects that the data stored in the hot spot cache area reaches the condition threshold, it transfers the data with the lowest 20% total usage frequency in the hot spot cache area to the buffer area;

[0020] When the cache management system detects that the data stored in the buffer area has a stable usage frequency of 20% or more within a certain period of time, it transfers the data stored in the buffer area to the hot spot cache area; when the cache management system detects that the data stored in the buffer area has a stable usage frequency of less than 10% within a certain period of time, it transfers the data stored in the buffer area to the local disk cache area;

[0021] When the cache management system detects that the data is transferred from the hot spot cache area to the local disk cache area again and the usage frequency reaches 20% or more, it transfers the data from the local disk cache area to the buffer area;

[0022] After the cache management system transfers the data, if the data triggers the elimination policy of the cache area, it is destroyed.

[0023] Based on the configuration policy based on strategies such as memory elimination, expired elimination, priority elimination, and hot spot elimination, the system proposed by the present invention proposes a life cycle scheduling mechanism for cached data, which can perform periodic maintenance of the cache, transfer buffered data, transfer and schedule cold and hot cached data, and destroy data.

[0024] Preferably, the structured data buffer further includes a static data buffer, a component buffer, and a data source buffer; the unstructured data buffer further includes a file system buffer and a page buffer.

[0025] More preferably, the cache management system stores the data in the buffer according to the characteristics of different data, which specifically includes: the cache management system is configured to determine whether the data has specified parameters. If there are specified parameters, the data is stored in the corresponding buffer according to the specified parameters of the data. If there are no specified parameters, the cache management system determines whether the source of the data has attributes. If there are attributes, the cache management system stores the data in the corresponding buffer according to the attributes of the source of the data. If there are no attributes, the cache management system stores the data in the corresponding buffer according to the type of the data.

[0026] Based on this, the present invention classifies and grades according to data size, data type, specified type, etc., realizes the integration, processing, and caching of data from multiple data sources including memory, hard disk, environment variables, data sources, etc., and provides an efficient and flexible caching technology. This technology makes full use of the advantages of multiple data sources to improve the stability and reliability of the business system.

[0027] More preferably, the cache management system uses the cluster nodes for cluster multi-node expansion, which specifically includes: the cache management system uses the defined and unused IPs within the cluster nodes for node expansion, and each node is initially set with a hot cache area and a local disk cache area;

[0028] The cache management system uses the cluster nodes for node multi-level expansion, which specifically includes: performing exposed parameter input on the nodes of the cluster nodes. The exposed parameter input includes input exposed parameter 1 and exposed parameter 2. The exposed parameter 1 specifies the hierarchical type of the node that has not been expanded, that is, the type of the buffer area, and the exposed parameter 2 specifies the unused hierarchical level to be bound to this type, so as to expand the hierarchical type of the node and reallocate the hierarchical levels.

[0029] By automatically horizontally expanding the cluster server nodes and driving the vertical expansion of the cache hierarchy in the nodes, the storage capacity, high availability of the cluster, fault tolerance, and query concurrency and efficiency can be improved.

[0030] Preferably, the cache management system further includes a resource scheduling adapter. The resource scheduling adapter is used to obtain the call requirement configuration of the business system, generate a unique cacheId rule, match the cacheId with the data of the cache management system, find the corresponding data or data source according to the cacheId matching result, and perform processing, merging, and deduplication, and form a corresponding cache data set to be transmitted to the business system.

[0031] Further preferably, the resource scheduling adapter includes a data encryption module, which is configured to obtain the SM2 public key sm2pubk of the service system, block-encrypt the cache data set using the SM4 algorithm to obtain the SM4 ciphertext sm4pric and the key sm4prik of SM4, and encrypt the key sm4prik of SM4 using the SM2 public key sm2pubk, and transmit the encrypted key sm4prik2 of SM4 and the SM4 ciphertext sm4pric to the service system.

[0032] By adopting a hybrid algorithm of symmetric encryption and asymmetric encryption to encrypt the transmitted data, the plaintext of the main body uses a symmetric encryption algorithm with extremely fast encryption speed, and then the key and ciphertext of the symmetric encryption algorithm are encrypted using a relatively secure asymmetric encryption algorithm before transmission, effectively improving the security of data during transmission and preventing the data from being maliciously tampered with.

[0033] According to one aspect of the present invention, a method for bidirectional extended block-based multi-source cache management is proposed, which is applied to a bidirectional extended block-based multi-source cache management system described in the first aspect. The method includes:

[0034] S1, the cache management system receives data and periodically checks the cache occupied space. When it is detected that the cache occupied space reaches a specified threshold, the cluster nodes are used for cluster multi-node expansion and node multi-level expansion, and the cluster nodes adopt hierarchical cache management;

[0035] S2, the cache management system stores the data in the cache area according to the characteristics of different data, sets corresponding elimination strategy types for the cache area, manages the cache life cycle of the data, and transfers or destroys the data.

[0036] According to one aspect of the present invention, a computer program product is proposed, on which a computer program is stored, and the computer program, when executed by a processor, implements the method described in any one of the first aspects.

[0037] Compared with the prior art, the present invention has the following beneficial technical effects:

[0038] 1. High data integration ability and resource utilization rate: The multi-source composite processing technology solves the limitations of data processing and format conversion, inconsistent and incomplete formats, and overcomes the limitations of a single data source, making full use of the advantages of multiple data sources, and improving the resource utilization rate and reliability of the service system.

[0039] 2. High security: The composite dual-key encryption technology effectively improves the security of data during transmission and prevents the data from being maliciously tampered with.

[0040] 3. Efficient Management of Cache Lifecycle: Hierarchical and classified block-based and improved lifecycle scheduling mechanisms are used to efficiently manage cache data, solve the problems of lack of flexibility, personalized eviction strategies, and insufficiently fine-grained cache lifecycle scheduling in conventional cache systems, and enhance data persistence capabilities. Appropriate strategy configurations are made according to the actual usage scenarios. By optimizing the data access path and achieving cache partition and level hits, the data processing speed and response speed are significantly improved, and the access frequency of the database and network transmission overhead are effectively reduced.

[0041] 4. Easier and Smarter Expansion: The two-way cache dynamic expansion technology enables horizontal cluster node addition and vertical flexible hierarchical management. The automatic expansion program simplifies the expansion operation process without manual intervention, solving the complexity and cumbersome problems of traditional cache expansion. It better adapts to dynamic business requirements and data growth, and improves the high availability, fault tolerance, query concurrency, and efficiency of the cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The drawings illustrate the embodiments and, together with the description, are used to explain the principles of the present invention. Other embodiments and many of the expected advantages of the embodiments will be readily appreciated as they become better understood by reference to the following detailed description. The elements of the drawings are not necessarily to scale with each other. The same reference numerals refer to corresponding like parts.

[0043] Figure 1 FIG. shows a system architecture diagram of a two-way extended block-based multi-source cache management according to an embodiment of the present invention;

[0044] Figure 2 FIG. shows a schematic diagram of two-way expansion according to a specific embodiment of the present invention;

[0045] Figure 3 FIG. shows a schematic diagram of cache area eviction strategy setting according to a specific embodiment of the present invention;

[0046] Figure 4 FIG. shows a schematic diagram of automatic switching between hot and cold levels according to a specific embodiment of the present invention;

[0047] Figure 5 FIG. shows a schematic diagram of a resource scheduling adapter obtaining data according to a specific embodiment of the present invention;

[0048] Figure 6 FIG. shows a schematic diagram of data encryption by a data encryption module and decryption by a business system according to a specific embodiment of the present invention;

[0049] Figure 7Shows a flowchart of a method for two-way extended block-based multi-source cache management according to a specific embodiment of the present invention;

[0050] Figure 8 Is a schematic structural diagram of a computer device of an electronic device suitable for implementing the embodiments of the present invention. Specific embodiments

[0051] The following further describes the present application in detail with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only for explaining the related invention, rather than limiting the invention. Additionally, it should be noted that for the convenience of description, only parts related to the relevant invention are shown in the drawings.

[0052] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The following will detail the present application with reference to the drawings and embodiments.

[0053] Figure 1 Shows an architecture diagram of a two-way extended block-based multi-source cache management system according to an embodiment of the present invention.

[0054] In a specific embodiment, the cache management system is a multi-node cluster architecture. Each cluster node is divided into two major cache areas, a structured data cache area and an unstructured data cache area. The types of the structured data cache area include: static data storage cache area, component cache area, hot cache area, data source cache area; the types of the unstructured data cache area include: file system cache area, page cache area, local disk cache area.

[0055] As Figure 2 Shown, during the operation of the cache management system, cluster multi-node expansion is performed. There is a cache occupancy space interval check program in the system. When the cache space reaches a specified threshold, the system will start an automatic expansion program to perform node expansion; use the defined and unused IPs within the cluster for node expansion. After the newly added node registers its identity in the registration center, it officially joins the cluster for use, expanding the storage capacity of the cache, improving the cache query speed and concurrency, and enhancing the fault tolerance of the system.

[0056] At the same time, node multi-level expansion is carried out. The node cache adopts hierarchical cache management. Each node has a different cache level. Each node is initially set with two cache areas, by default, the first-level hot cache area and the seventh-level local disk cache area

[0057] When the cache management system detects that the two-level cache does not meet the cache requirements of the existing application, it uses the driver to expand the hierarchy type and reassign the hierarchy levels. When adding a new type of cache area later, a level will be bound to it, and the levels (1-7) cannot overlap. The system front-end entry triggers the driver, and the type to be expanded can be specified according to business requirements. For the explicit parameter input, explicit parameter 1 specifies the unexpanded hierarchy type, including: static data buffer, component buffer, data source buffer, file system buffer, page buffer. Existing types cannot be selected again, and there is no requirement for the binding order. Explicit parameter 2 specifies the unused hierarchy level to be bound to this type.

[0058] In one embodiment, as Figure 1 shown, the received data comes from the APP business system. When the cache management system receives data, the system will perform cache block mapping according to the characteristics of different data. The characteristics of the data include factors such as the attributes of the source business system, the type of data, the data size, etc., or directly specify the type. The cache management system is configured to determine whether the data has specified parameters. If there are specified parameters, the data is stored in the corresponding cache area according to the specified parameters of the data. If there are no specified parameters, the cache management system determines whether the source of the data has attributes. If there are attributes, the cache management system stores the data in the corresponding cache area according to the attributes of the source of the data. If there are no attributes, the cache management system stores the data in the corresponding cache area according to the type of the data.

[0059] Among them, writing data into the cache area according to the specified parameters is based on the parameter passing when the APP business system transmits data to specify the cache area for writing. This direct specification of the type is relatively flexible.

[0060] When the type is not specified, the cache area is preferentially selected according to the attributes of the called business system first. The cache management system can define the attributes of the business scope of the business system and associate them with different types of cache areas, such as hot query services (corresponding to hot cache areas), visual application construction services (corresponding to component cache areas), static websites (corresponding to static data buffers), multi-type data source services (corresponding to data source cache areas), file management systems (corresponding to file system cache areas), front-end system services (corresponding to page cache areas).

[0061] When the cache management system does not label the attributes of the business system, it judges the data type. The judgment rules are as follows:

[0062] 1. When the data type is an unstructured file, when it meets the conditions that the file suffix is zip, tar, and the data size is greater than 10M, it enters the local disk cache area; otherwise, it enters the file system cache area to store smaller files and files that may be accessed more frequently, such as png, jpg pictures, doc, xls documents, etc.

[0063] When the type is structured data and it meets the json structure, it enters the data source cache area; otherwise, it enters the static data storage buffer.

[0064] When the business system calls the cache system to write data, it makes reasonable use of the above steps and rules, and the cache management system will reasonably select a suitable cache area for data writing.

[0065] Four types of cache eviction strategies are implemented in the cache management system: memory eviction based on the memory reaching the threshold condition, expiration eviction based on the time reaching the expiration period, priority eviction based on the priority level, and hot spot eviction based on the call frequency being lower than the specified frequency. Flexible configuration of strategies is carried out based on these types.

[0066] Among them, the cache management system sets the unstructured cache area to the expiration eviction strategy type or the priority eviction strategy type, and sets the structured cache area to the expiration eviction strategy type, the priority eviction strategy type, the memory eviction strategy type, or the hot spot eviction strategy type.

[0067] After the data is written to the cache area, the cache life cycle management is started, and the configured strategies of the cache come into effect, and periodic maintenance and management are carried out according to the strategy configuration rules and the life cycle scheduling mechanism.

[0068] The cache life cycle management of the data includes two management mechanisms. One is to detect that the cache eviction strategy set by the cache area is triggered, and the other is to detect that after the maximum memory threshold set by the hot spot cache area is reached, the data is switched between the hot and cold levels. The two management mechanisms run in parallel, and the cache management system executes the running mechanism that first meets the trigger conditions.

[0069] As Figure 3 shown, among them, detecting that the cache eviction strategy set by the cache area is triggered specifically includes: when the cache area is of the expiration eviction strategy type or the priority eviction strategy type, the cache management system directly destroys the data after detecting that the data stored in the cache area triggers the critical condition;

[0070] When the cache area is of the memory eviction strategy type or the hot spot eviction strategy type, after the cache management system detects that the data stored in the cache area triggers the critical condition, it transfers the data stored in the cache area to the buffer area from their respective corresponding cache areas, and sets the corresponding expiration duration. When the expiration duration arrives, it then transfers the data transferred to the buffer area to the local disk cache area and sets it to the expiration eviction strategy type, and destroys the data after triggering the critical condition.

[0071] The hot and cold level switching only applies between the hot cache area and the local disk cache area. Hot area data refers to the data stored in the hot cache area, which is usually used for data that is frequently accessed by the system and does not change easily. Cold area data refers to the data stored in the local disk cache area, and automatic cache level switching will occur when the conditions are met.

[0072] Each hot cache area is set with a maximum capacity value, and the cache management system sets the condition threshold for each hot cache area. When the condition threshold is reached, automatic cache level switching occurs, and the switching process does not involve destruction, unless the elimination policy set by the cache itself is triggered.

[0073] In one embodiment, the condition threshold is set to 95%. When the data in the hot cache area reaches the 95% threshold of the maximum capacity, regardless of the elimination policy used by the cache at this time, the cache data with the lowest 20% total usage frequency in the hot area is transferred to the buffer area. When in the buffer area, the original corresponding elimination policy rules are still effective.

[0074] As Figure 4 shown, when the cache management system detects that the data is transferred from the hot cache area to the local disk cache area and the usage frequency reaches 20% or more, the data is transferred from the local disk cache area to the buffer area.

[0075] When the cache management system detects that the data stored in the buffer area has a stable usage frequency of 20% or more within a period of time, the data stored in the buffer area is transferred to the hot cache area; when the cache management system detects that the data stored in the buffer area has a stable usage frequency of less than 10% within a period of time, the data stored in the buffer area is transferred to the local disk cache area.

[0076] It can be understood that the usage frequency and condition threshold corresponding to the cache management system's judgment of data transfer can be flexibly set according to the actual situation, and the specific values designed in the embodiment should not be construed as a limitation of the present invention.

[0077] During the hot and cold level switching process, when the data previously used the hot or memory elimination policy, the policy rules become invalid and will take effect again when transferred to the buffer area or the hot cache area next time.

[0078] When the business system needs to read data, it will send the corresponding call requirement configuration. After being obtained by the resource scheduling adapter of the cache management system, the adapter generates a unique cacheId rule according to the configuration. First, it matches the cacheId from the cache pool. If the corresponding key value is detected to exist, it directly returns the corresponding data; if the data does not exist, it will select a suitable data source according to the call requirement, or adopt the cache hierarchy priority principle, and search sequentially from high to low according to the levels bound by the cache hierarchy (the hot cache has the highest default level) for data source scheduling. Finally, whether data is obtained or not, it will search for data in the buffer area again.

[0079] In one embodiment, the call requirement configuration data structure is as follows: {"cacheKey":"userId:9999","sourceType":"hot,database","level":"1","deal":"dataform(yyyy-MM-dd)","sm2pubk":"123"}.

[0080] After the data is read, the data is processed, merged, and deduplicated. The processing includes data format conversion, normalization, etc., to ensure the consistency and correctness of the data in the system. Finally, the processed data is merged to form a cache data set corresponding to the cacheId, which is put into the cache pool and the data is transmitted to the business system of the calling APP.

[0081] The resource scheduling adapter further includes a data encryption module. When data is exchanged, a hybrid algorithm of symmetric encryption and asymmetric encryption is used. The main body plaintext uses a symmetric encryption algorithm with extremely fast encryption speed, and then the encryption key and ciphertext are encrypted using an asymmetric encryption algorithm with higher security and then transmitted.

[0082] In one embodiment, the data encryption module uses SM4 to encrypt the plaintext of the cache data set, selects a suitable encryption key and encryption mode, divides the plaintext block into multiple small blocks, and performs encryption operations on each small block to obtain the SM4 ciphertext sm4pric.

[0083] Then, the SM2 public key sm2pubk of the business system is obtained, and the SM4 key sm4prik is encrypted using the SM2 public key sm2pubk of the business system. By operating on the SM4 key sm4prik and the public key sm2pubk of the receiving party, the encrypted SM4 key sm4prik2 is generated, and then the encrypted SM4 key sm4prik2 and the SM4 ciphertext sm4pric are transmitted to the business system together.

[0084] When decrypting data, after the business system receives the encrypted data, it uses the unique SM2 private key sm2prik of the system to decrypt and decrypt the SM4 key sm4prik2 to obtain the original SM4 key sm4prik. Then, it uses the original SM4 key sm4prik to perform the decryption operation on the ciphertext sm4pric to obtain the cached data.

[0085] In one embodiment, the returned data structure is as follows: {"sm4prik2":"lcFEVSPRkcQZlgGzln0OMS43pcD0Kl3iH6aRR7Yg8Ki","sm4pric":"ieQY5HdKwitauURtEVh1+dO6qpz1veuu9XLzH6P6G6Z1llSo qM8iuMPswrR9bWVp"}.

[0086] According to one aspect of the present invention, a method for bidirectional extended block-based multi-source cache management is proposed, which is applied to a bidirectional extended block-based multi-source cache management system described in the first aspect, as Figure 7 shown. The method includes:

[0087] S1, the cache management system receives data and periodically checks the occupied space of the cache. When it is detected that the occupied space of the cache reaches a specified threshold, the cluster nodes are used for cluster multi-node expansion and node multi-level expansion, and the cluster nodes adopt hierarchical cache management;

[0088] S2, the cache management system stores the data in the cache area according to the characteristics of different data, sets corresponding elimination policy types for the cache area, performs cache life cycle management on the data, and transfers or destroys the data.

[0089] Next, refer to Figure 8 , which shows a schematic structural diagram of a computer system 800 of an electronic device suitable for implementing the embodiments of the present application. Figure 8 The shown electronic device is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present application.

[0090] As Figure 8 shown, the computer system 800 includes a central processing unit (CPU) 801, which performs various appropriate actions and processes according to the program stored in the read-only memory (ROM) 802 or the program loaded from the storage section 809 into the random access memory (RAM) 804. In the RAM 804, various programs and data required for the operation of the system 800 are also stored. The CPU 801, ROM 802, ROM 803, and RAM 804 are connected to each other through a bus 805. The input / output (I / O) interface 806 is also connected to the bus 805.

[0091] The following components are connected to the I / O interface 806: an input section 807 including a keyboard, a mouse, etc.; an output section 808 including, for example, a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 809 including a hard disk, etc.; and a communication section 810 including a network interface card such as a LAN card, a modem, etc. The communication section 810 performs communication processing via a network such as the Internet. A drive 811 is also connected to the I / O interface 806 as needed. A removable medium 812 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 811 as needed so that a computer program read therefrom is installed into the storage section 809 as needed.

[0092] Specifically, according to an embodiment of the present disclosure, the processes described above with reference to the flowchart are implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product that includes a computer program carried on a computer-readable storage medium, and the computer program includes program codes for performing the methods shown in the flowchart. In such an embodiment, the computer program is downloaded and installed from a network via the communication section 810, and / or installed from the removable medium 812. When the computer program is executed by a central processing unit (CPU) 801, the above-described functions defined in the method of the present application are executed.

[0093] It should be noted that the computer-readable storage medium of the present application is a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. The computer-readable storage medium is, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable storage medium is any tangible medium that contains or stores a program, and the program is used by or in conjunction with an instruction execution system, apparatus, or device. In the present application, the computer-readable signal medium includes a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal takes various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium is also any computer-readable storage medium other than the computer-readable storage medium, and the computer-readable storage medium sends, propagates, or transmits a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium is transmitted by any appropriate medium, including but not limited to: wireless, wire, optical fiber cable, RF, etc., or any suitable combination of the above.

[0094] Computer program code for performing the operations of the present application is written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code is executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer is connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or is connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0095] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagram represents a module, a segment of a program, or a portion of code that contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown actually execute substantially in parallel, and they sometimes execute in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as combinations of blocks in the block diagram and / or flowchart, are implemented by a dedicated hardware-based system for performing the specified functions or operations, or by a combination of dedicated hardware and computer instructions.

[0096] The modules described in the embodiments of the present application are implemented in software and also in hardware.

[0097] As another aspect, the present application also provides a computer-readable storage medium, which is included in the electronic device described in the above embodiments; it also exists separately without being assembled into the electronic device. The above computer-readable storage medium carries one or more programs. When the above one or more programs are executed by the electronic device, the electronic device performs: S1, the cache management system receives data, periodically checks the cache occupied space. When it is detected that the cache occupied space reaches the specified threshold, the cluster nodes are used for cluster multi-node expansion and node multi-level expansion, and the cluster nodes adopt hierarchical cache management; S2, the cache management system stores the data in the buffer area according to the characteristics of different data, sets corresponding eviction policy types for the buffer area, manages the cache life cycle of the data, and transfers or destroys the data.

[0098] The above description is only the preferred embodiments of the present application and the description of the applied technical principles. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solutions formed by the specific combination of the above technical features, but also covers other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the technical solutions formed by mutually replacing the above features with the (but not limited to) technical features with similar functions disclosed in the present application.

Claims

1. A bidirectionally scalable block-based multi-source cache management system, characterized in that: The cache management system is configured to receive data, store the data into a cache area according to the characteristics of different data, set a corresponding elimination policy type for the cache area, and perform cache life cycle management on the data. The cache area includes a structured cache area and an unstructured cache area. The structured cache area includes a hot spot cache area, and the unstructured cache area includes a local disk cache area. The cache management system includes multiple cluster nodes. The cache management system performs cache occupied space checks at intervals. When it is checked that the cache occupied space reaches a specified threshold, the cluster nodes are used to perform cluster multi-node expansion and node multi-level expansion. The cluster nodes adopt hierarchical cache management.

2. A bidirectionally scalable block-based multi-source cache management system according to claim 1, characterized in that: in, Set the corresponding elimination strategy type, including: The cache management system sets the unstructured cache area to an expired elimination strategy type or a priority elimination strategy type; The cache management system sets the structured cache area to an expired elimination policy type, a priority elimination policy type, a memory elimination policy type or a hot spot elimination policy type.

3. A bidirectionally scalable block-based multi-source cache management system according to claim 2, characterized in that: in, The data is cached for lifecycle management, specifically including: When the cache area is of an expired elimination policy type or a priority elimination policy type, the cache management system directly destroys the data stored in the cache area after detecting that the data triggers a critical condition; When the cache area is of a memory elimination policy type or a hot spot elimination policy type, after the cache management system detects that the data stored in the cache area triggers a critical condition, the data stored in the cache area is transferred from the corresponding cache area to the buffer area, and a corresponding expiration time is set. When the expiration time is reached, the data transferred to the buffer area is transferred to the local disk cache area, and set to the expiration elimination policy type. The data is destroyed after the critical condition is triggered.

4. A bidirectionally scalable block-based multi-source cache management system according to claim 2, characterized in that: in, The data is cached for lifecycle management, specifically including: The cache management system is configured to automatically switch the cache level of the data in the hotspot cache area and the local disk cache area, set a condition threshold for each hotspot cache area, and when it is detected that the data stored in the hotspot cache area reaches the condition threshold, transfer the data in the hotspot cache area that is ranked last in terms of total frequency of use to the buffer area; When the cache management system detects that the usage frequency of the data stored in the buffer is stably maintained at 20% or above for a period of time, the data stored in the buffer is transferred to the hotspot cache area; when the cache management system detects that the usage frequency of the data stored in the buffer is stably maintained at less than 10% for a period of time, the data stored in the buffer is transferred to the local disk cache area; When the cache management system detects that the data is transferred from the hotspot cache area to the local disk cache area and the usage frequency reaches 20% or more, the data is transferred from the local disk cache area to the buffer area; After transferring the data, the cache management system destroys the data if the data triggers an elimination strategy of the cache area.

5. The bidirectionally scalable block-based multi-source cache management system according to claim 2, characterized in that: The structured data cache area also includes a static data cache area, a component cache area, and a data source cache area; the unstructured data cache area also includes a file system cache area and a page cache area.

6. A bidirectionally scalable block-based multi-source cache management system according to claim 5, characterized in that: The cache management system stores the data into the cache area according to the characteristics of different data, specifically including: the cache management system is configured to determine whether the data has specified parameters, if so, store the data into the corresponding cache area according to the specified parameters of the data, if not, the cache management system determines whether the source of the data has attributes, if so, the cache management system stores the data into the corresponding cache area according to the attributes of the source of the data, if not, the cache management system stores the data into the corresponding cache area according to the type of the data.

7. The bidirectionally scalable block-based multi-source cache management system according to claim 5, characterized in that: The cache management system uses the cluster node to perform cluster multi-node expansion, specifically including: the cache management system uses the defined and unused IP in the cluster node to perform node expansion, and each node is initially set with a hotspot cache area and a local disk cache area; The cache management system uses the cluster nodes to perform multi-level node expansion, specifically including: inputting exposed parameters of the nodes of the cluster nodes, the exposed parameter input includes inputting exposed parameter 1 and exposed parameter 2, the exposed parameter 1 specifies the level type of the node that has not been expanded, that is, the type of the cache area, and the exposed parameter 2 specifies the unused level to be bound to the type, and expands the level type of the node and reallocates the level.

8. The bidirectionally scalable block-based multi-source cache management system according to claim 1, characterized in that: The cache management system also includes a resource scheduling adapter, which is used to obtain the call requirement configuration of the external business system, generate a unique cacheId rule, match the cacheId with the data of the cache management system, find the corresponding data or data source according to the cacheId matching result, and process, merge and deduplicate it to form a corresponding cache data set to be transmitted to the external business system.

9. A bidirectionally scalable block-based multi-source cache management system according to claim 8, characterized in that: The resource scheduling adapter includes a data encryption module, which is configured to obtain the SM2 public key sm2pubk of the external business system, use the SM4 algorithm to perform block encryption on the cached data set, obtain the SM4 ciphertext sm4pric and the SM4 key sm4prik, and use the SM2 public key sm2pubk to encrypt the SM4 key sm4prik, and transmit the encrypted SM4 key sm4prik2 and the SM4 ciphertext sm4pric to the external business system.

10. A bidirectionally scalable block multi-source cache management method, characterized in that: A bidirectionally extended block-based multi-source cache management system according to any one of claims 1 to 9 is applied, the method comprising: S1, the cache management system receives data and checks the cache occupied space at intervals. When it is checked that the cache occupied space reaches a specified threshold, the cluster node is used to perform cluster multi-node expansion and node multi-level expansion. The cluster node adopts hierarchical cache management; S2, the cache management system stores the data into a cache area according to the characteristics of different data, sets a corresponding elimination strategy type for the cache area, performs cache life cycle management on the data, and transfers or destroys the data.

11. A computer program product having a computer program stored thereon, which, when executed by a processor, implements the method according to claim 10.

Images