Cache system dynamic expansion method, device, equipment, medium and program product

Abstract

The present disclosure provides a cache system dynamic expansion method, relates to the technical field of cloud computing, and can be applied to the technical field of finance. The method comprises the following steps: in response to a data write request of an upstream application, a monitoring thread is set to acquire a remaining capacity of a first target cluster in real time; a second target cluster is created according to the remaining capacity of the first target cluster and an amount of data to be written of the upstream application; after the deployment of the second target cluster is completed, configuration information of the second target cluster is updated to a routing layer, wherein the routing layer is used for storing cluster Internet protocol addresses and forwarding of data to be written; and a data synchronization thread is started to synchronize inventory data of the first target cluster to the second target cluster. The present disclosure further provides a cache system dynamic expansion device, equipment, a storage medium and a program product.

Description

Technical Field

[0001] This disclosure relates to the field of cloud computing technology, specifically to the field of cache system expansion technology, and more specifically to a method, apparatus, device, medium, and program product for dynamic expansion of a cache system. Background Technology

[0002] To improve user experience, the financial industry typically stores frequently used data in caching systems. Compared to file systems and database systems, caching systems can respond to service requests much faster. Taking Redis Cluster as an example, the common scaling solution for caching systems is to use the native scaling mechanism. Caching systems typically have the following use cases: caching parameter data, caching session data, and batch writing of various data to the caching system, such as customer data and product data. In financial industry applications, insufficient storage may occur when writing data in batches. If the native Redis Cluster scaling solution is used, it can affect online transactions.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] In view of the above problems, this disclosure provides a method, apparatus, device, storage medium and program product for dynamic expansion of a cache system.

[0005] According to a first aspect of this disclosure, a method for dynamically expanding a cache system is provided, the method comprising:

[0006] In response to data write requests from upstream applications, a monitoring thread is set up to obtain the remaining capacity of the first target cluster in real time.

[0007] Create and deploy a second target cluster based on the remaining capacity of the first target cluster and the amount of data to be written by the upstream application;

[0008] After the second target cluster is deployed, its configuration information is updated to the routing layer, whereby the routing layer stores the cluster's Internet Protocol address and the forwarding of data to be written; and

[0009] Start the data synchronization thread to synchronize the existing data of the first target cluster to the second target cluster.

[0010] According to embodiments of this disclosure, creating and deploying a second target cluster based on the remaining capacity of the first target cluster and the amount of data to be written by the upstream application includes:

[0011] When it is determined that the remaining capacity of the first target cluster is less than the first preset threshold, a query request for the amount of data to be written is sent to the upstream application.

[0012] The cache system expansion requirements are determined based on the amount of data to be written returned by the upstream application, the remaining capacity of the first target cluster, and the total capacity of the first target cluster; and

[0013] When it is determined that the caching system needs to be expanded, a second target cluster is created and deployed based on the caching system expansion requirement information.

[0014] According to embodiments of this disclosure, the step of creating and deploying a second target cluster based on the remaining capacity of the first target cluster and the amount of data to be written by the upstream application further includes:

[0015] When it is determined that the remaining capacity of the first target cluster is less than the second preset threshold, a query request for the amount of data to be written is sent to the upstream application.

[0016] The total capacity of the second target cluster is determined based on the amount of data to be written returned by the upstream application, the remaining capacity of the first target cluster, and the total capacity of the first target cluster.

[0017] Create and deploy the second target cluster and update the configuration information of the second target cluster to the routing layer; and

[0018] Start the data synchronization thread to synchronize the existing data of the first target cluster to the second target cluster.

[0019] According to embodiments of this disclosure, determining the cache system expansion requirement information based on the amount of data to be written returned by the upstream application, the remaining capacity of the first target cluster, and the total capacity of the first target cluster includes:

[0020] Calculate the difference between the remaining capacity of the first target cluster and the amount of data to be written returned by the upstream application;

[0021] If the difference is less than or equal to a third preset threshold, it is determined that the cache system needs to be expanded, wherein the third preset threshold is related to the total capacity of the first target cluster;

[0022] The total capacity of the second target cluster is calculated based on the amount of data to be written returned by the upstream application, the remaining capacity of the first target cluster, and the total capacity of the first target cluster; and

[0023] If the difference is greater than the third preset threshold, then it is determined that the cache system does not need to be expanded.

[0024] According to embodiments of this disclosure, after initiating the data synchronization thread, the method further includes:

[0025] Record the offset position information of the message middleware for consuming messages in the first target cluster; and

[0026] After data synchronization is completed, the second target cluster continues to consume the data to be written based on the offset position information of the message middleware.

[0027] According to embodiments of this disclosure, after data synchronization is completed, the method further includes:

[0028] The machine resources of the first target cluster are reclaimed based on preset rules.

[0029] According to embodiments of this disclosure, the step of reclaiming machine resources of the first target cluster based on preset rules includes:

[0030] Once it is confirmed that the read / write status of the second target cluster is normal and the business data transaction status of the second target cluster is normal, the machine resources of the first target cluster are reclaimed according to the preset reclamation time.

[0031] A second aspect of this disclosure provides a dynamic expansion device for a cache system, the device comprising:

[0032] The cluster capacity monitoring module is used to respond to data write requests from upstream applications and set up a monitoring thread to obtain the remaining capacity of the first target cluster in real time.

[0033] The capacity expansion module is used to create and deploy a second target cluster based on the remaining capacity of the first target cluster and the amount of data to be written by the upstream application.

[0034] The cluster configuration information update module is used to update the configuration information of the second target cluster to the routing layer after the second target cluster is deployed, wherein the routing layer is used to store the cluster's Internet Protocol address and the forwarding of data to be written; and

[0035] The data synchronization module is used to start a data synchronization thread to synchronize the existing data of the first target cluster to the second target cluster.

[0036] According to embodiments of this disclosure, the capacity expansion module includes: a first determining submodule, a second determining submodule, and a target cluster first deployment submodule.

[0037] The first determining submodule is used to send a query request for the amount of data to be written to the upstream application when it is determined that the remaining capacity of the first target cluster is less than the first preset threshold.

[0038] The second determining submodule is used to determine the cache system expansion requirement information based on the amount of data to be written returned by the upstream application, the remaining capacity of the first target cluster, and the total capacity of the first target cluster; and

[0039] The first deployment submodule of the target cluster is used to create and deploy a second target cluster based on the cache system expansion requirement information when it is determined that the cache system needs to be expanded.

[0040] According to embodiments of this disclosure, the capacity expansion module further includes: a third determination submodule, a fourth determination submodule, a target cluster second deployment submodule, and a data synchronization submodule.

[0041] The third determining submodule is used to send a query request for the amount of data to be written to the upstream application when it is determined that the remaining capacity of the first target cluster is less than the second preset threshold.

[0042] The fourth determination submodule is used to determine the total capacity of the second target cluster based on the amount of data to be written returned by the upstream application, the remaining capacity of the first target cluster, and the total capacity of the first target cluster.

[0043] The second deployment submodule for the target cluster is used to create and deploy the second target cluster and update the configuration information of the second target cluster to the routing layer; and

[0044] The data synchronization submodule is used to start a data synchronization thread to synchronize the existing data of the first target cluster to the second target cluster.

[0045] According to embodiments of this disclosure, the second determining submodule includes: a first calculation unit, a first determining unit, a second calculation unit, and a second determining unit.

[0046] The first computing unit is used to calculate the difference between the remaining capacity of the first target cluster and the amount of data to be written returned by the upstream application.

[0047] The first determining unit is configured to determine that the cache system needs to be expanded if the difference is less than or equal to a third preset threshold, wherein the third preset threshold is related to the total capacity of the first target cluster.

[0048] The second calculation unit is used to calculate the total capacity of the second target cluster based on the amount of data to be written returned by the upstream application, the remaining capacity of the first target cluster, and the total capacity of the first target cluster; and

[0049] The second determining unit is used to determine that the cache system does not need to be expanded if the difference is greater than a third preset threshold.

[0050] According to embodiments of this disclosure, it further includes a recording module and a data writing module.

[0051] The recording module is used to record the offset position information of the message middleware for consuming messages in the first target cluster; and

[0052] The data writing module is used so that after data synchronization is completed, the second target cluster continues to consume the data to be written according to the offset position information of the message middleware.

[0053] According to embodiments of this disclosure, it further includes a resource recycling module.

[0054] The resource recycling module is used to recycle the machine resources of the first target cluster based on preset rules.

[0055] According to an embodiment of this disclosure, the resource recycling module includes a resource recycling submodule.

[0056] The resource reclamation submodule is used to reclaim the machine resources of the first target cluster according to a preset reclamation time after confirming that the read / write status of the second target cluster is normal and the business data transaction status of the second target cluster is normal.

[0057] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the above-described dynamic expansion method for a cache system.

[0058] A fourth aspect of this disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the above-described method for dynamically expanding the cache system.

[0059] The fifth aspect of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method for dynamically expanding the cache system.

[0060] This disclosure provides a dynamic scaling method for a caching system. A monitoring thread is set up to obtain the remaining capacity of a first target cluster in real time. Based on the remaining capacity of the first target cluster and the amount of data to be written by the upstream application, a second target cluster is created and deployed. After the second target cluster is deployed, its configuration information is updated to the routing layer. The routing layer stores the cluster's Internet Protocol address and forwards the data to be written. A data synchronization thread is started to synchronize the existing data of the first target cluster to the second target cluster. Compared to related technologies, the dynamic scaling method for a caching system provided by this disclosure is based on a routing layer mechanism, ensuring that incremental data can be quickly switched to the new cluster, online transactions are not affected during scaling, and for existing data, a data synchronization thread is started to synchronize the data. Attached Figure Description

[0061] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0062] Figure 1 The diagram illustrates a method for scaling up a native Redis cluster in related technologies.

[0063] Figure 2a This schematically illustrates a system architecture diagram of a dynamic scaling device for a cache system according to an embodiment of the present disclosure;

[0064] Figure 2b The illustration schematically depicts an application scenario of a dynamic expansion method, apparatus, device, storage medium, and program product for a cache system according to embodiments of the present disclosure.

[0065] Figure 3 The flowchart illustrating a method for dynamically expanding a cache system according to an embodiment of the present disclosure is shown in the illustration.

[0066] Figure 4 A flowchart illustrating a method for dynamically expanding a cache system according to another embodiment of this disclosure is shown schematically.

[0067] Figure 5 A flowchart illustrating a method for dynamically expanding a cache system according to yet another embodiment of this disclosure is shown.

[0068] Figure 6 A flowchart illustrating a method for dynamically expanding a cache system according to another embodiment of the present disclosure is shown.

[0069] Figure 7 A flowchart illustrating a method for dynamically expanding a cache system according to another embodiment of the present disclosure is shown.

[0070] Figure 8 This schematically illustrates a structural block diagram of a dynamic expansion device for a cache system according to an embodiment of the present disclosure; and

[0071] Figure 9 A block diagram schematically illustrates an electronic device suitable for implementing a dynamic expansion method for a cache system according to an embodiment of the present disclosure. Detailed Implementation

[0072] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0073] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0074] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0075] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0076] Commonly used caching systems include Redis Cluster, and this disclosure uses Redis Cluster as an example. To better understand the technical issues addressed in this disclosure, we will first... Figure 1 This section introduces the scaling mechanisms related to Redis Cluster. Figure 1 This diagram illustrates a method for scaling up a native Redis cluster in related technologies. For example... Figure 1As shown, if a native Redis Cluster is scaled up, a portion of slots and data needs to be migrated to a new node. Each master node migrates a portion of its slots and data to the new node04. The slot migration process is as follows: 1. Assume there were originally 3 masters, each responsible for 16384 / 3, approximately 5461 slots. 2. After adding a new master, each master is responsible for 16384 / 4 = 4096 slots. 3. After determining the migration plan, each master migrates the portion exceeding 4096 slots to the new master, and then begins the migration in units of slots. Because the migration process is synchronous, the main thread of the original node is blocked between the execution of the restore command on the target node and the deletion of the key on the original node, until the key is successfully deleted. If a network failure occurs during the migration process, and the entire slot migration is only half completed, the two nodes will still be marked as intermediate filtering states, i.e., "migrating" and "importing". The migration will continue the next time the migration tool reconnects. During migration, if the content of each key is small, the migration process is fast and will not affect normal client access. However, if the content of each key is large, the migration process for a single key is blocking, which will cause both the original and target nodes to lag, affecting the stability of the cluster. Therefore, the native scaling solution for Redis clusters may experience blocking and lag issues, affecting online transactions.

[0077] Based on the above-mentioned technical problems, embodiments of this disclosure provide a method for dynamically expanding a caching system. The method includes: responding to a data write request from an upstream application, setting a monitoring thread to obtain the remaining capacity of a first target cluster in real time; creating and deploying a second target cluster based on the remaining capacity of the first target cluster and the amount of data to be written by the upstream application; after the second target cluster is deployed, updating the configuration information of the second target cluster to the routing layer, wherein the routing layer is used to store the cluster's Internet Protocol address and the forwarding of the data to be written; and starting a data synchronization thread to synchronize the existing data of the first target cluster to the second target cluster.

[0078] Figure 2a The diagram schematically illustrates the system architecture of the dynamic scaling device for the caching system according to an embodiment of this disclosure. Figure 2aAs shown in the embodiments of this disclosure, the caching system architecture includes: a message middleware Kafka, a routing layer, and a Redis cluster. When an upstream application needs to write data to the Redis cluster, the data is first written to the message middleware. The routing layer consumes the data and forwards it to the Redis cluster according to pre-stored cluster configuration information. The Redis cluster starts a monitoring thread to monitor the cluster capacity usage in real time. When preset conditions are met, it queries the upstream application for the amount of data to be written and determines whether to perform a scaling operation based on the amount of data to be written and the cluster capacity usage. After scaling is determined, a new cluster is deployed, the configuration information of the new cluster is updated to the routing layer, and data synchronization is completed.

[0079] Figure 2b The illustration shows an application scenario of a method, apparatus, device, storage medium, and program product for dynamically expanding a cache system according to embodiments of the present disclosure.

[0080] like Figure 2b As shown, application scenario 200 according to this embodiment may include a scenario of dynamic expansion of the cache system. Network 204 is used as a medium to provide a communication link between terminal devices 201, 202, 203 and server 205. Network 204 may include various connection types, such as wired, wireless communication links or fiber optic cables, etc.

[0081] Users can use terminal devices 201, 202, and 203 to interact with server 205 via network 204 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 201, 202, and 203, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0082] Terminal devices 201, 202, and 203 can be various electronic devices with displays that support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0083] Server 205 can be a dynamic scaling server for a caching system. This server can execute the dynamic scaling method for a caching system provided in this embodiment of the present disclosure, set up a monitoring thread to obtain the remaining capacity of the first target cluster in real time; create and deploy a second target cluster based on the remaining capacity of the first target cluster and the amount of data to be written by the upstream application; after the deployment of the second target cluster is completed, update the configuration information of the second target cluster to the routing layer, wherein the routing layer is used to store the cluster Internet Protocol address and the forwarding of the data to be written; and start a data synchronization thread to synchronize the existing data of the first target cluster to the second target cluster.

[0084] It should be noted that the dynamic scaling method for the cache system provided in this embodiment can generally be executed by server 205. Correspondingly, the dynamic scaling device for the cache system provided in this embodiment can generally be located in server 205. The dynamic scaling method for the cache system provided in this embodiment can also be executed by a server or server cluster that is different from server 205 and capable of communicating with terminal devices 201, 202, 203 and / or server 205. Correspondingly, the dynamic scaling device for the cache system provided in this embodiment can also be located in a server or server cluster that is different from server 205 and capable of communicating with terminal devices 201, 202, 203 and / or server 205.

[0085] It should be understood that Figure 2b The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0086] It should be noted that the dynamic scaling method and apparatus for a cache system determined in the embodiments of this disclosure can be used in the field of cloud computing technology, the field of financial technology, and any field other than the financial field. The application field of the dynamic scaling method and apparatus for a cache system determined in the embodiments of this disclosure is not limited.

[0087] The following will be based on Figure 2a The system architecture described and Figure 2b The described application scenarios, through Figures 3 to 7 The method for dynamically expanding the cache system according to embodiments of this disclosure will be described in detail.

[0088] Figure 3 The illustration shows a flowchart of a dynamic expansion method for a cache system according to an embodiment of the present disclosure. Figure 3 As shown, the dynamic expansion method of the cache system in this embodiment includes operations S210 to S220, which can be executed by a server or other computing device.

[0089] When operating S210, in response to data write requests from upstream applications, a monitoring thread is set up to obtain the remaining capacity of the first target cluster in real time.

[0090] In operation S220, a second target cluster is created and deployed based on the remaining capacity of the first target cluster and the amount of data to be written by the upstream application.

[0091] In operation S230, after the second target cluster is deployed, the configuration information of the second target cluster is updated to the routing layer.

[0092] In operation S240, a data synchronization thread is started to synchronize the existing data of the first target cluster to the second target cluster.

[0093] According to embodiments of this disclosure, the routing layer is used to store cluster Internet Protocol addresses and forward data to be written.

[0094] In one example, when an upstream application writes data, the data is first written to a message broker, such as Kafka. Between the upstream application and the Redis cluster is a routing layer. This routing layer stores the Redis cluster information, and the application accesses the Redis cluster through this layer. The routing layer consumes data from Kafka and writes it to the Redis cluster. The routing layer in this disclosure is equivalent to adding a proxy node to the Redis cluster, primarily used for storing the cluster's Internet Protocol address and forwarding data to be written. When the downstream Redis cluster IP changes, the routing layer updates the Redis cluster's configuration information, without the upstream application being aware of it.

[0095] In one example, when an upstream application writes data to the caching system, a monitoring thread is set up to obtain the remaining capacity of the first target cluster in real time. Based on the remaining capacity of the first target cluster and the amount of data to be written by the upstream application, a specific scaling strategy is determined, i.e., whether a second target cluster needs to be created and deployed. For details on the scaling process, please refer to [link to documentation]. Figures 4 to 6 The operation steps are as follows: After the second target cluster is deployed, the configuration information of the second target cluster is updated to the routing layer, and the routing layer disconnects the first target cluster. At this time, the cluster that the upstream application does not know about has been switched from the first target cluster to the second target cluster.

[0096] In one example, after creating the second target cluster, a data synchronization thread is started to synchronize the existing data of the first target cluster to the second target cluster. This completes the dynamic expansion of the caching system without the user's awareness, and the expansion process does not affect online transactions.

[0097] This disclosure provides a dynamic scaling method for a caching system. A monitoring thread is set up to obtain the remaining capacity of a first target cluster in real time. Based on the remaining capacity of the first target cluster and the amount of data to be written by the upstream application, a second target cluster is created and deployed. After the second target cluster is deployed, its configuration information is updated to the routing layer. The routing layer stores the cluster's Internet Protocol address and forwards the data to be written. A data synchronization thread is started to synchronize the existing data of the first target cluster to the second target cluster. Compared to related technologies, the dynamic scaling method for a caching system provided by this disclosure is based on a routing layer mechanism, ensuring that incremental data can be quickly switched to the new cluster, online transactions are not affected during scaling, and for existing data, a data synchronization thread is started to synchronize the data.

[0098] The following will combineFigures 4 to 6 This invention describes the judgment logic for dynamic expansion of the cache system in the embodiments of this disclosure. Figure 4 The flowchart illustrating a method for dynamically expanding a cache system according to another embodiment of the present disclosure is shown. Figure 5 The flowchart illustrating a method for dynamically expanding a cache system according to yet another embodiment of the present disclosure is shown. Figure 6 The flowchart illustrates a method for dynamically expanding a cache system according to another embodiment of the present disclosure.

[0099] like Figure 4 As shown, operation S220 includes operations S310 to S330.

[0100] In operation S310, when it is determined that the remaining capacity of the first target cluster is less than the first preset threshold, a query request for the amount of data to be written is sent to the upstream application.

[0101] In operation S320, the cache system expansion requirement information is determined based on the amount of data to be written returned by the upstream application, the remaining capacity of the first target cluster, and the total capacity of the first target cluster.

[0102] When operating S330, if it is determined that the cache system needs to be expanded, a second target cluster is created and deployed based on the cache system expansion requirement information.

[0103] In one example, when the used capacity of the first target cluster is greater than 50%, it is determined that the remaining capacity of the first target cluster is less than the first preset threshold. In this embodiment of the disclosure, the first preset threshold can be 50%. The monitoring thread will send a data query request to the upstream application that writes data to obtain how much data is left to be written, and compare it with the obtained amount of data to be written x1, the remaining capacity of the first target cluster x2, and the total capacity of the first target cluster x3.

[0104] like Figure 5 As shown, operation S320 includes operations S321 to S324.

[0105] In operation S321, the difference between the remaining capacity of the first target cluster and the amount of data to be written returned by the upstream application is calculated.

[0106] In operation S322, if the difference is less than or equal to a third preset threshold, it is determined that the cache system needs to be expanded, wherein the third preset threshold is related to the total capacity of the first target cluster.

[0107] In operation S323, the total capacity of the second target cluster is calculated based on the amount of data to be written returned by the upstream application, the remaining capacity of the first target cluster, and the total capacity of the first target cluster.

[0108] In operation S324, if the difference is greater than the third preset threshold, it is determined that the cache system does not need to be expanded.

[0109] In one example, the difference x2-x1 between the remaining capacity of the first target cluster and the amount of data to be written returned by the upstream application is calculated. If this difference is less than or equal to a third preset threshold, it indicates that the remaining capacity of the first target cluster is insufficient to meet the amount of data to be written, and it is determined that the caching system needs to be expanded, requiring the construction of a new cluster. The third preset threshold is related to the total capacity x3 of the first target cluster, and in this embodiment, the third preset threshold may be, for example, x3*10%. Based on the amount of data to be written returned by the upstream application, the remaining capacity of the first target cluster, and the total capacity of the first target cluster, the total capacity y*90%≥x1+(x3-x2) of the second target cluster is determined. A deployment cluster is created based on the total capacity of the second target cluster.

[0110] In another feasible embodiment, such as Figure 6 As shown, operation S220 also includes operations S410 to S440.

[0111] In operation S410, when it is determined that the remaining capacity of the first target cluster is less than the second preset threshold, a query request for the amount of data to be written is sent to the upstream application.

[0112] In operation S420, the total capacity of the second target cluster is determined based on the amount of data to be written returned by the upstream application, the remaining capacity of the first target cluster, and the total capacity of the first target cluster.

[0113] In operation S430, a second target cluster is created and deployed, and the configuration information of the second target cluster is updated to the routing layer.

[0114] In operation S440, a data synchronization thread is started to synchronize the existing data of the first target cluster to the second target cluster.

[0115] In one example, when it is determined that the remaining capacity of the first target cluster is less than a second preset threshold, such as 10%, an expansion operation is immediately performed. A query request for the amount of data to be written is sent to the upstream application, and the total capacity of the second target cluster is determined based on the amount of data to be written returned by the upstream application, the remaining capacity of the first target cluster, and the total capacity of the first target cluster.

[0116] Figure 7 The flowchart illustrates a method for dynamically expanding a cache system according to another embodiment of the present disclosure.

[0117] After starting the data synchronization thread, operations S250 to S270 are included after operation S240.

[0118] During operation S250, the offset position information of the message middleware for consuming messages in the first target cluster is recorded.

[0119] During operation S260, after data synchronization is completed, the second target cluster continues to consume the data to be written based on the message middleware offset position information.

[0120] In operation S270, the machine resources of the first target cluster are reclaimed based on preset rules.

[0121] According to the embodiments of this disclosure, after confirming that the read / write status of the second target cluster is normal and the business data transaction status of the second target cluster is normal, the machine resources of the first target cluster are reclaimed according to a preset reclamation time.

[0122] In one example, after setting up and deploying a new cluster, some data in the first target cluster needs to be synchronized to the second target cluster for aggregation. A separate data synchronization thread is started to consume data from the first target cluster and transfer it to the second target cluster. The Kafka offset position information consumed by the first target cluster is recorded. After the data synchronization is complete, the second target cluster continues to consume data from Kafka based on the recorded Kafka offset position information, ensuring no data loss.

[0123] In one example, after the second target cluster updates its routing layer configuration information, and provided that the second target cluster is functioning correctly and its data is error-free, the machine resources of the first target cluster will be reclaimed to ensure device reuse. The reclamation time can be manually configured. Once it is confirmed that the read / write status of the second target cluster is normal and the business data transaction status of the second target cluster is normal, the machine resources of the first target cluster will be reclaimed according to the preset reclamation time.

[0124] Based on the above-described method for dynamically expanding a cache system, this disclosure also provides a device for dynamically expanding a cache system. The following will be combined with... Figure 8 The device is described in detail.

[0125] Figure 8 A schematic block diagram illustrating a dynamic expansion device for a cache system according to an embodiment of the present disclosure is shown. Figure 8 As shown, the dynamic expansion device 800 of the cache system in this embodiment includes a cluster capacity monitoring module 810, a capacity expansion module 820, a cluster configuration information update module 830, and a data synchronization module 840.

[0126] The cluster capacity monitoring module 810 is used to respond to data write requests from upstream applications and set up a monitoring thread to obtain the remaining capacity of the first target cluster in real time. In one embodiment, the cluster capacity monitoring module 810 is used to perform the operation S210 described above, which will not be repeated here.

[0127] The capacity expansion module 820 is used to create and deploy a second target cluster based on the remaining capacity of the first target cluster and the amount of data to be written by the upstream application. In one embodiment, the capacity expansion module 820 can be used to perform the operation S220 described above, which will not be repeated here.

[0128] The cluster configuration information update module 830 is used to update the configuration information of the second target cluster to the routing layer after the second target cluster is deployed. The routing layer stores the cluster's Internet Protocol address and the forwarding information for the data to be written. In one embodiment, the cluster configuration information update module 830 can be used to perform the operation S2230 described above, which will not be repeated here.

[0129] The data synchronization module 840 is used to start a data synchronization thread to synchronize the existing data of the first target cluster to the second target cluster. In one embodiment, the data synchronization module 840 can be used to perform the operation S240 described above, which will not be repeated here.

[0130] According to embodiments of this disclosure, the capacity expansion module includes: a first determining submodule, a second determining submodule, and a target cluster first deployment submodule.

[0131] The first determining submodule is used to send a query request for the amount of data to be written to the upstream application when it is determined that the remaining capacity of the first target cluster is less than a first preset threshold. In one embodiment, the first determining submodule can be used to perform the operation S310 described above, which will not be repeated here.

[0132] The second determining submodule is used to determine the cache system expansion requirement information based on the amount of data to be written returned by the upstream application, the remaining capacity of the first target cluster, and the total capacity of the first target cluster. In one embodiment, the second determining submodule can be used to perform the operation S320 described above, which will not be repeated here.

[0133] The first deployment submodule of the target cluster is used to create and deploy a second target cluster based on the cache system expansion requirement information when it is determined that the cache system needs to be expanded. In one embodiment, the first deployment submodule of the target cluster can be used to perform the operation S330 described above, which will not be repeated here.

[0134] According to embodiments of this disclosure, the capacity expansion module further includes: a third determination submodule, a fourth determination submodule, a target cluster second deployment submodule, and a data synchronization submodule.

[0135] The third determining submodule is used to send a query request for the amount of data to be written to the upstream application when it is determined that the remaining capacity of the first target cluster is less than a second preset threshold. In one embodiment, the third determining submodule can be used to perform the operation S410 described above, which will not be repeated here.

[0136] The fourth determining submodule is used to determine the total capacity of the second target cluster based on the amount of data to be written returned by the upstream application, the remaining capacity of the first target cluster, and the total capacity of the first target cluster. In one embodiment, the fourth determining submodule can be used to perform the operation S420 described above, which will not be repeated here.

[0137] The second deployment submodule for the target cluster is used to create and deploy the second target cluster and update the configuration information of the second target cluster to the routing layer. In one embodiment, the second deployment submodule for the target cluster can be used to perform the operation S430 described above, which will not be repeated here.

[0138] The data synchronization submodule is used to start a data synchronization thread to synchronize the existing data of the first target cluster to the second target cluster. In one embodiment, the data synchronization submodule can be used to perform the operation S440 described above, which will not be repeated here.

[0139] According to embodiments of this disclosure, the second determining submodule includes: a first calculation unit, a first determining unit, a second calculation unit, and a second determining unit.

[0140] The first computing unit is used to calculate the difference between the remaining capacity of the first target cluster and the amount of data to be written returned by the upstream application. In one embodiment, the first computing unit can be used to perform the operation S321 described above, which will not be repeated here.

[0141] The first determining unit is configured to determine that the cache system needs to be expanded if the difference is less than or equal to a third preset threshold, wherein the third preset threshold is related to the total capacity of the first target cluster. In one embodiment, the first determining unit may be used to perform the operation S322 described above, which will not be repeated here.

[0142] The second calculation unit is used to calculate the total capacity of the second target cluster based on the amount of data to be written returned by the upstream application, the remaining capacity of the first target cluster, and the total capacity of the first target cluster. In one embodiment, the second calculation unit can be used to perform the operation S323 described above, which will not be repeated here.

[0143] The second determining unit is configured to determine that the cache system does not need to be expanded if the difference is greater than a third preset threshold. In one embodiment, the second determining unit may be used to perform the operation S324 described above, which will not be repeated here.

[0144] According to embodiments of this disclosure, it further includes a recording module and a data writing module.

[0145] The recording module is used to record the offset position information of the message middleware consumed by the first target cluster. In one embodiment, the recording module can be used to perform the operation S250 described above, which will not be repeated here.

[0146] The data writing module is used so that, after data synchronization is completed, the second target cluster continues to consume the data to be written based on the offset position information of the message middleware. In one embodiment, the data writing module can be used to perform the operation S260 described above, which will not be repeated here.

[0147] According to embodiments of this disclosure, it further includes a resource recycling module.

[0148] The resource reclamation module is used to reclaim machine resources of the first target cluster based on preset rules. In one embodiment, the resource reclamation module can be used to perform the operation S270 described above, which will not be repeated here.

[0149] According to an embodiment of this disclosure, the resource recycling module includes a resource recycling submodule.

[0150] The resource reclamation submodule is used to reclaim the machine resources of the first target cluster according to a preset reclamation time after confirming that the read / write status of the second target cluster is normal and the business data transaction status of the second target cluster is normal. In one embodiment, the resource reclamation submodule can be used to execute the operation S270 described above, which will not be repeated here.

[0151] According to embodiments of this disclosure, any multiple modules among the cluster capacity monitoring module 810, capacity expansion module 820, cluster configuration information update module 830, and data synchronization module 840 can be merged into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the cluster capacity monitoring module 810, capacity expansion module 820, cluster configuration information update module 830, and data synchronization module 840 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in a suitable combination of any of these. Alternatively, at least one of the cluster capacity monitoring module 810, capacity expansion module 820, cluster configuration information update module 830, and data synchronization module 840 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0152] Figure 9 A block diagram schematically illustrates an electronic device suitable for implementing a dynamic expansion method for a cache system according to an embodiment of the present disclosure.

[0153] like Figure 9 As shown, an electronic device 900 according to an embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0154] RAM 903 stores various programs and data required for the operation of electronic device 900. Processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Processor 901 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 902 and / or RAM 903. It should be noted that the programs may also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0155] According to embodiments of this disclosure, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to a bus 904. The electronic device 900 may also include one or more of the following components connected to the I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 909 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 909 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0156] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the dynamic expansion method of the cache system according to the embodiments of this disclosure.

[0157] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 902 and / or RAM 903 and / or one or more memories other than ROM 902 and RAM 903 described above.

[0158] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the dynamic expansion method for the cache system provided in the embodiments of this disclosure.

[0159] When the computer program is executed by the processor 901, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0160] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 909, and / or installed from a removable medium 911. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0161] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0162] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0163] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0164] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0165] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A method for dynamically expanding a cache system, characterized in that, The method includes: In response to data write requests from upstream applications, a monitoring thread is set up to obtain the remaining capacity of the first target cluster in real time. Create and deploy a second target cluster based on the remaining capacity of the first target cluster and the amount of data to be written by the upstream application; After the second target cluster is deployed, its configuration information is updated to the routing layer, whereby the routing layer stores the cluster's Internet Protocol address and the forwarding of data to be written; and Start the data synchronization thread to synchronize the existing data of the first target cluster to the second target cluster.

2. The method according to claim 1, characterized in that, The step of creating and deploying a second target cluster based on the remaining capacity of the first target cluster and the amount of data to be written by the upstream application includes: When it is determined that the remaining capacity of the first target cluster is less than the first preset threshold, a query request for the amount of data to be written is sent to the upstream application. The cache system expansion requirements are determined based on the amount of data to be written returned by the upstream application, the remaining capacity of the first target cluster, and the total capacity of the first target cluster; and When it is determined that the caching system needs to be expanded, a second target cluster is created and deployed based on the caching system expansion requirement information.

3. The method according to claim 1, characterized in that, The step of creating and deploying the second target cluster based on the remaining capacity of the first target cluster and the amount of data to be written by the upstream application also includes: When it is determined that the remaining capacity of the first target cluster is less than the second preset threshold, a query request for the amount of data to be written is sent to the upstream application. The total capacity of the second target cluster is determined based on the amount of data to be written returned by the upstream application, the remaining capacity of the first target cluster, and the total capacity of the first target cluster. Create and deploy the second target cluster and update the configuration information of the second target cluster to the routing layer; and Start the data synchronization thread to synchronize the existing data of the first target cluster to the second target cluster.

4. The method according to claim 2, characterized in that, The process of determining the cache system expansion requirement information based on the amount of data to be written returned by the upstream application, the remaining capacity of the first target cluster, and the total capacity of the first target cluster includes: Calculate the difference between the remaining capacity of the first target cluster and the amount of data to be written returned by the upstream application; If the difference is less than or equal to a third preset threshold, it is determined that the cache system needs to be expanded, wherein the third preset threshold is related to the total capacity of the first target cluster; The total capacity of the second target cluster is calculated based on the amount of data to be written returned by the upstream application, the remaining capacity of the first target cluster, and the total capacity of the first target cluster; and If the difference is greater than the third preset threshold, then it is determined that the cache system does not need to be expanded.

5. The method according to any one of claims 1 to 4, characterized in that, After starting the data synchronization thread, the following is also included: Record the offset position information of the message middleware for consuming messages in the first target cluster; and After data synchronization is completed, the second target cluster continues to consume the data to be written based on the offset position information of the message middleware.

6. The method according to claim 5, characterized in that, After data synchronization is complete, the following is also included: The machine resources of the first target cluster are reclaimed based on preset rules.

7. The method according to claim 6, characterized in that, The step of reclaiming machine resources of the first target cluster based on preset rules includes: Once it is confirmed that the read / write status of the second target cluster is normal and the business data transaction status of the second target cluster is normal, the machine resources of the first target cluster are reclaimed according to the preset reclamation time.

8. A dynamic expansion device for a cache system, characterized in that, The device includes: The cluster capacity monitoring module is used to respond to data write requests from upstream applications and set up a monitoring thread to obtain the remaining capacity of the first target cluster in real time. The capacity expansion module is used to create and deploy a second target cluster based on the remaining capacity of the first target cluster and the amount of data to be written by the upstream application. The cluster configuration information update module is used to update the configuration information of the second target cluster to the routing layer after the second target cluster is deployed, wherein the routing layer is used to store the cluster's Internet Protocol address and the forwarding of data to be written; and The data synchronization module is used to start a data synchronization thread to synchronize the existing data of the first target cluster to the second target cluster.

9. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors execute the cache system dynamic expansion method according to any one of claims 1 to 7.

10. A computer-readable storage medium having stored executable instructions thereon, which, when executed by a processor, cause the processor to perform the cache system dynamic expansion method according to any one of claims 1 to 7.

Images