Method and system for automatically arranging cache service

Abstract

The invention belongs to the technical field of data processing, and discloses a method for automatically arranging cache services, which comprises the following specific steps: step 1, a user applies for required capacity according to requirements, fills in related data and submits the next process; according to the method, high-reliability service capability is provided for middleware Redis service automatic arrangement in a production environment, a standard business application process is established, so that the process is standardized, application and configuration are standardized, safety, reliability and stability are achieved, the problems of configuration disorder and the like are avoided, the Redis versions are uniform and are 5.6, and the Redis versions are uniform and are 5.2. 6. Meanwhile, based on the work order process, butt joint with existing platform standardized services can be supported, pod is automatically created, services are started, clusters are established, template configuration, different capacities, specifications and copies of Redis services are generated in a customized mode according to different business scenes, services are provided, cost reduction and efficiency improvement are achieved, the server utilization rate is effectively increased, resources are reasonably used, and the system can be used immediately after being opened.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular relates to a method and system for automatic arrangement of cache services. Background Art

[0002] With the development of Internet technology, massive storage of big data, and explosive growth of business, a well-designed system needs to consider many factors or scenarios, such as business reliability, security, scalability, extensibility, customizability, maintainability, stability, and user experience.

[0003] In order to support business development more efficiently, more and more companies have chosen to build platforms based on the concept of cloud native (Kubernetes) to improve standardization and resource utilization and deployment efficiency. The emergence of Kubernetes makes the management and change of container applications and resources easy. Efficient resource scheduling, flexible scaling and orchestration, automatic service discovery, and automatic fault recovery make Kubernetes almost the absolute protagonist of the container technology ecosystem. Kubernetes provides an extension mechanism that allows you to define your own resource types and run a pod on the cluster to monitor various types of custom resources and manage other resources based on it. When orchestrating cache services (Redis) based on Operators, it is difficult for existing native Operators to achieve 7x24 uninterrupted user requests, improve service stability, automatic service discovery, automatic fault recovery and other difficult technical problems. It is difficult to achieve 99.99% availability indicators for containerized services. Based on the defects of native Operators, improve Operator processes, methods, and scalability to ensure the stable operation of the business. This is the problem to be solved by automatic orchestration services. Summary of the invention

[0004] The purpose of the present invention is to provide a method and system for automatic arrangement of cache services to solve the problems raised in the above background technology.

[0005] In order to achieve the above object, the present invention provides the following technical solution: a method for automatic arrangement of cache services, the specific steps are as follows:

[0006] Step 1:

[0007] Users apply for the required capacity based on demand, fill in relevant data, and submit to the next process;

[0008] Step 2:

[0009] After receiving the process, the superior will evaluate the rationality of the resource and combine it with the business budget assessment. If it is rejected, it can enter the next process;

[0010] Step 3:

[0011] The administrator confirms whether the resources meet the business requirements based on the capacity submitted by the user. The administrator can reject the request or trigger the build process.

[0012] Step 4:

[0013] After entering the build process, you will enter the entire service orchestration process;

[0014] Step 5:

[0015] Create LB and domain name;

[0016] Step 6:

[0017] Since then, the Redis service orchestration has been completed and delivered to users;

[0018] Step 7:

[0019] Redis Operator will scale the capacity of the cluster without any user awareness.

[0020] Preferably, the specific steps in step 4 include:

[0021] Step 4.1, the work order calls the escort redis interface, which then calls the k8s api interface to create;

[0022] Step 4.2, directly call the k8s api to create instances according to the work order parameters;

[0023] Step 4.3, the Operator create function will be configured according to the work order parameters;

[0024] Step 4.4, Operator obtains the image address and selects the corresponding version of the image according to the template;

[0025] Step 4.5, Operator generates a file that can be parsed by the k8s StatefulSet yaml standard;

[0026] Step 4.6, Operator calls built-in methods to create redis pod and predixy;

[0027] Step 4.7: If the creation fails, an exception message will be returned, indicating the relevant error and solving it;

[0028] Step 4.8. If the creation is successful, the Operator starts the pod according to the statefulset yaml configuration;

[0029] Step 4.9. After the Pod is started, a rediscluster set will be automatically created according to the redis cluster command; the group consists of two nodes in the same group, the master / salve, and all node IPs and ports are written to the predixy configuration file.

[0030] Preferably, the specific steps in step 5 include:

[0031] Step 5.1, call the 4-layer LB interface, create the corresponding LB instance, and configure the source station to be the IP and port of the container predixy;

[0032] Step 5.2: After LB is created successfully, call the DNS interface to create the corresponding domain name resolution.

[0033] Preferably, after submitting the application in step 1, the user needs to check the data filled in by the user and verify the key fields to ensure that the data is accurate, including capacity, configuration requirements, etc.

[0034] Preferably, as described in step 4, the k8s API interface needs to be monitored and tested before building to ensure the stability and availability of the k8s API interface, and the configuration of the Operator needs to be strictly reviewed and tested before building to ensure its accuracy, avoid service problems caused by configuration errors, and enable the building process to proceed smoothly.

[0035] Preferably, when performing capacity scaling operations on Redis Operator as described in step 7, it is necessary to perform capacity scaling operations during non-peak hours to minimize the impact on users. At the same time, when Redis Operator performs capacity scaling operations, operators are required to monitor the capacity scaling process to ensure that the capacity scaling operations do not affect existing services, thereby ensuring the stability of the services.

[0036] A system for automatic cache service orchestration, comprising:

[0037] Load module: connects the entire cluster to the operation and maintenance LB load;

[0038] Proxy module: Rediscluster cluster uses Predixy as a proxy;

[0039] Deployment module: RedisCluster cluster is deployed using multiple Statsfulset+Headless Service.

[0040] Preferably, the proxy module is deployed using Deployment, with 3 replicas by default, where all pods have fixed IPs, and the default configuration value is 2-core cpu / 2g memory, while supporting rolling upgrade configuration and rapid expansion of the number of replicas.

[0041] Preferably, each group of redisclusternode in the proxy module is a group of Statsfulset+Headless, which contains two copies, namely two pods, namely master and slave, wherein anti-affinity is set between each group of pods to ensure that they are not on the same node, but different nodes groups (Statsfulset+Headless) are allowed to be on the same node, and all pods must mount storage, and / usr / local / redis-cluster / data is persisted, which stores cluster node information and rdb files, and the IPs of all pods are fixed IPs. After a pod exception, the IP of a new pod remains unchanged after it comes up, and the persistent storage information of the original pod can be read, and all pods can adjust the configuration information in a rolling manner.

[0042] The beneficial effects of the present invention are as follows:

[0043] The present invention mainly solves the problem of providing highly reliable service capabilities for the automatic orchestration of middleware Redis services in a production environment. By establishing a standard business application process, the process is standardized, and the application and configuration are standardized to achieve safety, reliability, stability, and avoid configuration errors and other problems. The redis version is unified, all of which are 5.0.6. At the same time, it can support docking with standardized services of existing platforms based on the work order process, automatically create pods and start services, form clusters, and customize the template configuration of Redis services according to different business scenarios. Different capacities, specifications, and copies provide services to achieve cost reduction and efficiency improvement, effectively improve server utilization, use resources reasonably, and use them out of the box, reduce the application, deployment, and delivery of business process resources, and use resources reasonably through containerization capabilities to reduce the increase in resource costs, ensure stable service operation, self-healing of faults, safety and reliability, and ensure the availability and reliability of services in the event of user network failures. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a schematic diagram of the business process of the present invention;

[0045] Figure 2 It is a schematic diagram of the architecture design of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] like Figure 1 to Figure 2 As shown, the embodiment of the present invention provides a method for automatic arrangement of cache services, and the specific steps are as follows:

[0048] Step 1:

[0049] Users apply for the required capacity based on demand, fill in relevant data, and submit to the next process;

[0050] Step 2:

[0051] After receiving the process, the superior will evaluate the rationality of the resource and combine it with the business budget assessment. If it is rejected, it can enter the next process;

[0052] Step 3:

[0053] The administrator confirms whether the resources meet the business requirements based on the capacity submitted by the user. The administrator can reject the request or trigger the build process.

[0054] Step 4:

[0055] After entering the build process, you will enter the entire service orchestration process;

[0056] Step 5:

[0057] Create LB and domain name;

[0058] Step 6:

[0059] Since then, the Redis service orchestration has been completed and delivered to users;

[0060] Step 7:

[0061] Redis Operator will scale the capacity of the cluster without any user awareness.

[0062] By establishing a standard business application process, using internal work orders to apply based on resource evaluation and submitting relevant configurations, the superior needs to approve whether the resource is reasonable, evaluate the business, and ensure that the business resources are reasonably applied. After approval, the administrator needs to approve whether the work order resource exists, and then select the corresponding template configuration to trigger the call to the k8s api interface for rapid construction, delivery in seconds, and ready to use out of the box, reducing the long process of user resources, reducing the cost of operation and maintenance deployment, and quickly completing the delivery of Redis services to meet business needs.

[0063] The specific steps in step 4 include:

[0064] Step 4.1, the work order calls the escort redis interface, which then calls the k8s api interface to create;

[0065] Step 4.2, directly call the k8s api to create instances according to the work order parameters;

[0066] Step 4.3, the Operator create function will be configured according to the work order parameters;

[0067] Step 4.4, Operator obtains the image address and selects the corresponding version of the image according to the template;

[0068] Step 4.5, Operator generates a file that can be parsed by the k8s StatefulSet yaml standard;

[0069] Step 4.6, Operator calls built-in methods to create redis pod and predixy;

[0070] Step 4.7: If the creation fails, an exception message will be returned, indicating the relevant error and solving it;

[0071] Step 4.8. If the creation is successful, the Operator starts the pod according to the statefulset yaml configuration;

[0072] Step 4.9. After the Pod is started, a rediscluster set will be automatically created according to the redis cluster command; the group consists of two nodes in the same group, the master / salve, and all node IPs and ports are written to the predixy configuration file.

[0073] The work order system calls the Escort Redis interface, and then further calls the Kubernetes API to create resources. The entire process does not require manual configuration one by one, which improves deployment efficiency. The Operator generates a standardized StatefulSet YAML file based on the work order parameters to ensure the consistency and predictability of each deployment.

[0074] The specific steps in step five include:

[0075] Step 5.1, call the 4-layer LB interface, create the corresponding LB instance, and configure the source station to be the IP and port of the container predixy;

[0076] Step 5.2: After LB is created successfully, call the DNS interface to create the corresponding domain name resolution.

[0077] By creating a 4-layer LB instance and configuring the source station as the IP and port of the container Predixy, high availability and load balancing of the Redis service are achieved. LB can automatically distribute user requests to multiple Predixy instances, improving the system's throughput and response speed and reducing the risk of single point failure.

[0078] In step 1, after submitting the application, the user needs to check the data filled in by the user and verify the key fields to ensure that the data is accurate, including capacity, configuration requirements, etc.

[0079] By having users fill out applications, deployment is carried out according to customer needs, ensuring that service deployment meets the actual needs of users, improving the pertinence and efficiency of resource use, and at the same time standardizing processes, applications and configurations to ensure safety, reliability and stability, thereby avoiding problems such as configuration errors.

[0080] Among them, in step 4, the k8s API interface needs to be monitored and tested before building to ensure the stability and availability of the k8s API interface, and the configuration of the Operator needs to be strictly reviewed and tested before building to ensure its accuracy and avoid service problems caused by configuration errors, so that the building process can proceed smoothly.

[0081] By testing the k8s API interface and strictly reviewing and testing the Operator configuration before building, we can ensure that the build process can proceed smoothly and avoid errors in subsequent processes.

[0082] In step 7, when performing capacity scaling operations on the Redis Operator, it is necessary to perform capacity scaling operations during non-peak hours to minimize the impact on users. At the same time, when the Redis Operator performs capacity scaling operations, the operator is required to monitor the capacity scaling process to ensure that the capacity scaling operations do not affect existing services, thereby ensuring service stability.

[0083] Through automated capacity scaling, the performance and resource utilization of the Redis service are ensured, while manual intervention and operation and maintenance costs are reduced.

[0084] A system for automatic cache service orchestration, comprising:

[0085] Load module: connects the entire cluster to the operation and maintenance LB load;

[0086] Proxy module: Rediscluster cluster uses Predixy as a proxy;

[0087] Deployment module: RedisCluster cluster is deployed using multiple Statsfulset+Headless Service.

[0088] As the proxy layer of Redis, Predixy can effectively process access requests from clients and automatically distribute these requests evenly to each node of the Redis cluster, thereby improving the overall system throughput and response speed.

[0089] The proxy module is deployed using Deployment, with three replicas by default. All pods have fixed IP addresses, and the default configuration value is 2-core CPU / 2G memory. It also supports rolling upgrade configuration and rapid expansion of the number of replicas.

[0090] Deployment supports rolling upgrades, which allows you to update the configuration or version of Predixy without interrupting service. At the same time, by adjusting the number of Deployment replicas, you can quickly expand the number of Predixy instances to cope with traffic growth.

[0091] Among them, each group of redisclusternode in the proxy module is a group of Statsfulset+Headless, which contains two copies, namely two pods, namely master and slave. Anti-affinity is set between each group of pods to ensure that they are not on the same node, but different nodes groups (Statsfulset+Headless) are allowed to be on the same node, and all pods must mount storage and persist / usr / local / redis-cluster / data, which stores cluster node information and rdb files. The IPs of all pods are fixed IPs, and after the pod is abnormal, the IP of the new pod remains unchanged after it comes up, and the persistent storage information of the original pod can be read. All pods can adjust the configuration information in a rolling manner.

[0092] Each Pod is mounted with persistent storage to ensure that its data can be retained and restored even if the Pod fails or is rescheduled. At the same time, high availability and redundant backup of data are achieved by configuring master-slave replicas.

[0093] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0094] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for automatic orchestration of cache services, characterized in that, The specific steps are as follows: Step 1: Users apply for the required capacity based on their needs, fill in the relevant data, and submit to the next step. Step 2: After receiving the process, the superior leader assesses the rationality of the resource, taking into account the business budget. If the request is rejected, the process can proceed to the next step. Step 3: The administrator can determine whether the resources meet the business requirements based on the capacity submitted by the user. The administrator can reject the submission or initiate the build process. Step 4: Once the build process begins, the entire service orchestration process will commence. Step 5: Create a load balancer and domain name; Step 6: This completes the Redis service orchestration and delivery to users. Step 7: Redis Operator will perform capacity scaling operations on the cluster without the user's awareness.

2. The method for automatic orchestration of cache services according to claim 1, characterized in that: The specific steps described in step four include: Step 4.1: The work order calls the escort Redis interface, which in turn calls the Kubernetes API interface to create the application; Step 4.2: Create instances by directly calling the Kubernetes API based on the work order parameters; Step 4.3: The Operator create function will be configured according to the work order parameters; Step 4.4: The Operator obtains the image address and selects the corresponding version of the image based on the template; Step 4.5: The Operator generates a Kubernetes StatefulSet YAML standard parsable file; Step 4.6: The Operator calls built-in methods to create the Redis pod and Predixy; Step 4.7: If creation fails, an exception message will be returned, indicating the relevant error. You will need to resolve this issue. Step 4.8: If creation is successful, the Operator starts the pod based on the StatefulSet YAML configuration; Step 4.9: After the Pod starts, it will automatically create a rediscluster based on the redis cluster command. The two nodes in the same group form a master / salve, and write the IPs and ports of all nodes into the predixy configuration file.

3. The method for automatic orchestration of cache services according to claim 1, characterized in that: The specific steps described in step five include: Step 5.1: Call the Layer 4 Load Balancer (LB) interface to create the corresponding LB instance and configure the origin server to the IP address and port of the container Predixy; Step 5.2: After the Load Balancer (LB) is successfully created, call the DNS interface to create the corresponding domain name resolution.

4. The method for automatic orchestration of cache services according to claim 1, characterized in that: After submitting the application, the user in step 1 needs to verify the data they filled in and validate key fields to ensure the data is accurate, including capacity, configuration requirements, etc.

5. The method for automatic orchestration of cache services according to claim 1, characterized in that: As mentioned in step 4, the Kubernetes API interface needs to be monitored and tested before building to ensure its stability and availability. The Operator configuration also needs to be strictly reviewed and tested before building to ensure its accuracy, avoid service problems caused by configuration errors, and ensure that the building process can proceed smoothly.

6. The method for automatic orchestration of cache services according to claim 1, characterized in that: As described in step 7, when performing capacity scaling operations on Redis Operator, these operations should be conducted during off-peak hours to minimize the impact on users. Furthermore, operators must monitor the capacity scaling process to ensure it does not affect existing services, thereby ensuring service stability.

7. A system for automatically orchestrating cache services, characterized in that, include: Load balancing module: Connects the entire cluster to the operation and maintenance load balancer. Proxy module: Rediscluster clusters use Predixy as a proxy; Deployment module: The RedisCluster cluster is deployed using multiple StatsfulSets + Headless Services.

8. The system for automatic orchestration of cache services according to claim 7, characterized in that: The agent module is deployed using Deployment, with 3 replicas by default. All pods have fixed IPs and are configured with a default value of 2 CPU cores / 2GB memory. It also supports rolling upgrades of configuration and rapid expansion of the number of replicas.

9. The system for automatic orchestration of cache services according to claim 7, characterized in that: In the proxy module, each group of redisclusternodes is a Statsfulset+Headless group, containing two replicas, i.e., two pods, namely master and slave. Anti-affinity is set between pods in each group to ensure that they are not on the same node. However, different groups of nodes (Statsfulset+Headless) are allowed to be on the same node. All pods must mount storage and persist / usr / local / redis-cluster / data, which stores cluster node information and RDB files. All pods have static IPs, and after a pod fails, the IP of the new pod remains unchanged and can read the persistent storage information of the original pod. All pods can have their configuration information adjusted in a rolling manner.

Images