Data service operation environment isolation system based on cloud native and implementation method

Abstract

The invention provides a data service operation environment isolation system and implementation method based on cloud native, and the system comprises a cluster management module which is used for obtaining registered cluster information, and carrying out the management of the cluster information; the resource group management module is used for acquiring the registered resource group information and managing the resource group information; and the operation environment management module is used for selecting corresponding resource group information and cluster information according to requirements, creating a corresponding operation environment for the API on the container platform based on the selected resource group information and cluster information, and managing the operation environment.

Description

Technical Field

[0001] The present invention relates to the field of cloud native data service technology, and in particular to a cloud native-based data service operating environment isolation system and an implementation method. Background Art

[0002] Data service is a serverless architecture. Users do not need to pay attention to how to deploy the server, how to configure the database, how to build the cache and other middleware, etc. They only need to configure the API information on the interface. As the use of data services increases, the stability of data services is becoming more and more important. However, since all APIs run on the same operating environment, it is inevitable that APIs of different users will affect each other, which will seriously affect the operating stability of data services. For example: some APIs have a large number of visits, resulting in high CPU usage, which will affect the access efficiency of other APIs; some APIs return a large amount of data, resulting in high memory usage, which will also affect the access efficiency of other APIs; some APIs have a large amount of logs due to a large number of visits, which in turn affects the normal operation of the container, and thus also affects the access efficiency of other APIs. Therefore, the isolation of data service APIs is an urgent problem to be solved.

[0003] Patent document CN117931442A (application number: 202410097614.X) discloses a data service system based on Kubernetes. The technical problem it solves is how to use Kubernetes and cloud native technologies to improve the flexibility, maintainability and reliability of the data service system. The technical solution is: API interface service: Each published API interface corresponds to a Kubernetes Service. The Kubernetes Service consists of multiple copies of Pods, and each Pod runs the code of the backend storage engine; Service Gateway: Envoy is used as the service gateway. Envoy load balances HTTP requests to multiple Pods of the Kubernetes Service; IngressController: IngressController monitors the Pod changes of each Service in Kubernetes and dynamically writes the IP information of the Pod back to Envoy; Front-end access layer: The front-end APP, Web or business system is connected to Envoy through a 4-layer load balancer to direct traffic to the back-end API interface service. The load balancer is used to process external requests and distribute traffic evenly to multiple Envoy instances. Compared with the patent, the system described in the patent can solve the problem of mutual influence between data service APIs. However, this system has the following limitations: (1) The system corresponds to a Kubernetes Service for each published API interface. For a big data platform, the number of APIs may be in the tens of thousands. If each API interface corresponds to a Kubernetes Service, tens of thousands of Kubernetes Services will be required, which places great demands on the use of resources. (2) The main function of the system is to create an operating environment, and it lacks some management operations for the operating environment, such as modifying the configuration of the operating environment CPU, memory, number of replicas, image adjustment, start, restart, stop, etc. (3) The system also lacks some configuration management functions for the data services of the operating environment. Summary of the invention

[0004] In view of the defects in the prior art, the object of the present invention is to provide a cloud-native based data service operating environment isolation system and implementation method.

[0005] A cloud-native-based data service operating environment isolation system provided by the present invention includes:

[0006] The cluster management module is used to obtain the registered cluster information and manage the cluster information;

[0007] The resource group management module is used to obtain the registered resource group information and manage the resource group information;

[0008] The operating environment management module is used to select the corresponding resource group information and cluster information according to the needs, create a corresponding operating environment for the API on the container platform based on the selected resource group information and cluster information, and manage the operating environment.

[0009] Preferably, in the cluster management module, the registered cluster information obtained includes: the English name of the cluster and the Chinese name of the cluster;

[0010] In the resource group management module, the registered resource group information obtained includes: cluster, resource group name, resource group description, project name, memory limit, CPU limit and replica number limit.

[0011] Preferably, in the operating environment management module, creating a corresponding operating environment for the API on the container platform includes: creating a corresponding operating environment and importing the corresponding operating environment;

[0012] The creation of the corresponding operating environment includes creating a Deployment object and configuring the parameters of the corresponding operating environment, including: operating environment name, description, workspace, number of CPU cores, memory, number of replicas, resource group, mirror, whether to enable log collection, whether to enable health check, and configuration template;

[0013] The importing of the corresponding operating environment includes: importing a Deployment object, configuring the parameters of the corresponding operating environment, including: resource group, application name, environment, operating environment name, description, workspace, and configuration template; based on the configured resource group and application name parameters, querying the corresponding operating environment according to the resource group and application name.

[0014] Preferably, the operating environment management module includes: modifying the operating environment, starting the operating environment or restarting the operating environment, stopping the operating environment, mirroring adjustment, and configuring the operating environment;

[0015] The modification of the operating environment is achieved by modifying the YAML or JSON configuration file of the Deployment, including modifying at least one of the operating environment name, description, workspace, number of CPU cores, memory, number of replicas, resource group, image, whether to enable log collection, whether to enable health check, and configuration template;

[0016] The starting operating environment or restarting the operating environment is to start the created or imported operating environment;

[0017] Stopping the running environment is achieved by adjusting the number of Deployment replicas to 0;

[0018] The image adjustment is achieved by updating the image version in the Deployment configuration, including: viewing the current image and modifying it to the target image; restarting the operating environment after the image adjustment;

[0019] The configuration operating environment is the maximum number of returned records and the dedicated engine of the configuration operating environment.

[0020] Preferably, the operating environment corresponding to the API is queried through query parameters; wherein the query parameters include: at least one parameter of the operating environment name, the workspace to which it belongs, the resource group, and the image name;

[0021] After the query is completed, the information displayed includes: operating environment name, description, workspace, occupied resources, resource group, image name, service status, whether log collection is enabled, whether health check is enabled, and access address.

[0022] A cloud-native-based data service operating environment isolation method provided by the present invention includes:

[0023] Step S1: Obtain registered cluster information and manage the cluster information;

[0024] Step S2: Obtain the registered resource group information and manage the resource group information;

[0025] Step S3: Select corresponding resource group information and cluster information according to requirements, create a corresponding operating environment for the API on the container platform based on the selected resource group information and cluster information, and manage the operating environment.

[0026] Preferably, the obtained registered cluster information includes: the English name of the cluster and the Chinese name of the cluster;

[0027] The resource group information obtained and registered includes: cluster, resource group name, resource group description, project name, memory limit, CPU limit and replica number limit.

[0028] Preferably, creating a corresponding operating environment for the API on the container platform includes: creating a corresponding operating environment and importing the corresponding operating environment;

[0029] The creation of the corresponding operating environment includes: creating a Deployment object, configuring the parameters of the corresponding operating environment, including: operating environment name, description, workspace, number of CPU cores, memory, number of replicas, resource group, mirror, whether to enable log collection, whether to enable health check, and configuration template;

[0030] The importing of the corresponding operating environment includes: importing a Deployment object, configuring the parameters of the corresponding operating environment, including: resource group, application name, environment, operating environment name, description, workspace, and configuration template; based on the configured resource group and application name parameters, querying the corresponding operating environment according to the resource group and application name.

[0031] Preferably, the management of the operating environment includes: modifying the operating environment, starting the operating environment or restarting the operating environment, stopping the operating environment, mirroring adjustment, and configuring the operating environment;

[0032] The modification of the operating environment is achieved by modifying the YAML or JSON configuration file of the Deployment, including modifying at least one of the operating environment name, description, workspace, number of CPU cores, memory, number of replicas, resource group, image, whether to enable log collection, whether to enable health check, and configuration template;

[0033] The starting operating environment or restarting the operating environment is to start the created or imported operating environment;

[0034] Stopping the running environment is achieved by adjusting the number of Deployment replicas to 0;

[0035] The image adjustment is achieved by updating the image version in the Deployment configuration, including: viewing the current image and modifying it to the target image; restarting the operating environment after the image adjustment;

[0036] The configuration operating environment is the maximum number of returned records and the dedicated engine of the configuration operating environment.

[0037] Preferably, the operating environment corresponding to the API is queried through query parameters; wherein the query parameters include: at least one parameter of the operating environment name, the workspace to which it belongs, the resource group, and the image name;

[0038] After the query is completed, the information displayed includes: operating environment name, description, workspace, occupied resources, resource group, image name, service status, whether log collection is enabled, whether health check is enabled, and access address.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] 1. A data service operating environment isolation system was implemented based on cloud native, which solved the problem of data service APIs affecting each other and causing unstable data service operation;

[0041] 2. The data service operating environment isolation system implements the version management function of the operating environment through the image adjustment function. When a new version of the data service is launched, it will only be launched in the default operating environment, and other operating environments will not be upgraded. Users can choose whether to upgrade the data service version or maintain the current stable data service version. The image adjustment function solves the problem that when a new version of the data service is launched, the function change will affect all users and affect the system that has been launched and running normally.

[0042] 3. The data service operating environment isolation system implements a complete set of operating environment management functions including creation, import, modification, start, restart, stop, image adjustment, configuration, query, etc., making the management of the operating environment more convenient and flexible;

[0043] 4. The data service operating environment isolation system supports data service configuration. Each operating environment is configured independently, making it more convenient and flexible to manage data service configuration;

[0044] 5. The data service operating environment isolation system configures a workspace for the operating environment. The API belongs to the workspace, so the API can specify the operating environment, thereby achieving a more flexible and free API operating environment isolation function. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0046] Figure 1 This is a system framework diagram for isolating the cloud-native data service operating environment.

[0047] Figure 2 Provides a state diagram for the operating environment.

[0048] Figure 3 Manage swimlane diagrams for operational environments. DETAILED DESCRIPTION

[0049] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0050] Example 1

[0051] According to a cloud-native-based data service operating environment isolation system and implementation method provided by the present invention, based on cloud-native container technology, a set of data service operating environment isolation functions is implemented, which solves the problem that APIs in the data service operating environment interfere with each other, resulting in unstable operation of the data service. A set of data service operating environment isolation functions includes: cluster management, resource group management, and operating environment management. The operating environment management functions include creating an operating environment, importing an operating environment, modifying an operating environment, starting an operating environment, restarting an operating environment, stopping an operating environment, adjusting an image, configuring an operating environment, querying an operating environment, etc. Users can use this set of operating environment isolation functions to create an independent operating environment for their own APIs. After specifying an operating environment for the API, the API will run in the independent operating environment and will no longer be affected by other APIs.

[0052] The cloud-native-based data service operating environment isolation system, such as Figure 1 , 3 As shown, including:

[0053] Module M1: Register cluster information.

[0054] Specifically, the cluster information registered in the module M1 includes the English name of the cluster and the Chinese name of the cluster, and only the operation and maintenance manager has the authority to register the cluster information.

[0055] Module M2: Register resource group information.

[0056] Specifically, the resource group information registered in step S2 includes: cluster, resource group name, resource group description, project name, memory limit, CPU limit, and replica number limit. Only the operation and maintenance manager has the authority to register cluster information.

[0057] Module M3: Select corresponding resource group information and cluster information according to requirements, create a corresponding operating environment for the API on the container platform based on the selected resource group information and cluster information, and manage the operating environment;

[0058] In this embodiment, the container platform includes: a K8S cluster.

[0059] This embodiment implements a data service operating environment isolation system based on cloud-native container technology. In managing the operating environment, the operating environment corresponds to a Deployment in Kubernetes, and one Deployment corresponds to one operating environment. In Kubernetes, a Deployment is a controller used to declaratively update applications and services.

[0060] Creating a running environment means creating a Deployment object.

[0061] Modifying the running environment is achieved by modifying the Deployment's YAML or JSON configuration file. The Deployment's YAML or JSON configuration file can define the number of application replicas, the image used, environment variables, etc.

[0062] Starting the runtime environment is starting the Deployment, which will automatically create and manage a group of Pod instances based on its configuration. These Pod instances are the running carriers of containerized applications.

[0063] Stopping the running environment is achieved by adjusting the number of replicas of the Deployment to 0, which causes Kubernetes to gradually stop and delete all Pods managed by the Deployment.

[0064] Deleting a runtime environment directly deletes the Deployment object, which will also delete all the Pods it manages, thereby completely removing the runtime environment.

[0065] Image adjustment is achieved by updating the image version in the Deployment configuration. When the image field of the Deployment changes, Kubernetes will automatically roll out all related Pods to use the new image version.

[0066] Ingress is an API object in Kubernetes that is used to route external HTTP(S) traffic to services within the cluster. Through Ingress, you can define routing rules based on domain names and paths, so that different operating environments (even if they share the same domain name) can be accessed through different context paths. Therefore, each operating environment has a unique access address through Ingress technology.

[0067] In the big data platform, registering APIs as shared services allows developers and other systems to discover and call these APIs through a unified interface. Through the operating environment specified by the API, the system can obtain the correct access address and save this information in the shared service information of the API. When the API is called, the system will forward the request to the correct operating environment for execution based on the API name and the operating environment access address stored in the shared service information. This mechanism implements dynamic routing based on the API name and operating environment, enhancing the flexibility and scalability of the system.

[0068] More specifically, creating a corresponding operating environment for the API on the container platform includes: creating an operating environment or importing an operating environment;

[0069] The parameters for creating the operating environment include: operating environment name, description, workspace, number of CPU cores, memory (GB), number of replicas, resource group, mirror, whether to enable log collection, whether to enable health check, and configuration template.

[0070] The parameters for importing the operating environment include: resource group, application name, environment, operating environment name, description, workspace, and configuration template. After entering the resource group and application name, the corresponding environment will be automatically queried based on the resource group and application name. Select the environment to be imported from the drop-down environment.

[0071] Specifically, the management of the operating environment includes:

[0072] Modify the operating environment;

[0073] Specifically, modifying the operating environment includes: operating environment name, description, workspace, number of CPU cores, memory (GB), number of replicas, resource group, image, whether to enable log collection, whether to enable health check, and configuration template.

[0074] Start the operating environment or restart the operating environment;

[0075] Specifically, starting the operating environment means starting the created or imported operating environment. For the created operating environment, the corresponding operating environment will be created on the container platform first, and then the operating environment will be started. For the imported operating environment, the operating environment will be started directly.

[0076] Stop the running environment;

[0077] Specifically, stopping the running environment means stopping the running running environment. After the running environment is stopped, the API will not be able to run in the stopped running environment. Figure 2 As shown in the figure, the runtime environment is in the unstarted state when it is just created. When the start button is clicked, the runtime environment will go through intermediate states such as waiting for resources, starting to queue, and starting. When the startup is successful, the runtime environment will be converted to the normal service state. When the stop button is clicked, the runtime environment will go through the intermediate state of shutting down. After the shutdown is successful, the runtime environment will be converted to the normal shutdown state. If the startup fails, the stop is abnormal, the runtime is abnormal, etc., the runtime environment will be converted to abnormal states such as being killed, service unavailable, platform failure, etc.

[0078] Mirror image adjustment;

[0079] Specifically, the image adjustment is to modify the image of the operating environment, and its parameters include: current image, target image. After the image adjustment, the operating environment is restarted, and the operating environment will update the image.

[0080] Configure the operating environment;

[0081] Specifically, the parameters for configuring the operating environment include: maximum number of returned records, dedicated engine. After the maximum number of returned records is configured, the amount of data returned by the API call cannot exceed this limit in this operating environment. The dedicated engine is the database engine that the operating environment can run. For example, if the dedicated engine is configured as IOTDB, then only the API of the IOTDB database can run in this operating environment, and the APIs of other databases cannot run in this operating environment. After the configuration is completed, it will take effect without restarting the operating environment.

[0082] In addition, the present invention also includes querying the operating environment;

[0083] Specifically, the query parameters for querying the operating environment include: operating environment name, workspace, resource group, and image name; the information displayed after the query is completed includes: operating environment name, description, workspace, occupied resources (core / memory*number of copies), resource group, image name, service status, whether log collection is enabled, whether health check is enabled, and access address.

[0084] Configure the corresponding operating environment for the API;

[0085] Specifically, the API specifies an operating environment that can be selected when the API is released online. The optional operating environments include: a default operating environment and an operating environment allocated to a workspace.

[0086] Calling the API;

[0087] Calling an API is to use POSTMAN or other clients to call the API in a shared service manner. When the API is released online, it will be registered in the shared service, and the API call address will also be saved in the shared service information. When calling through a shared service, the API call address will be obtained according to the API name. The call address is the call address of different operating environments of the data service, so the API will be forwarded to the corresponding operating environment for operation.

[0088] The present invention also provides a cloud-native based data service operating environment isolation system, which can be implemented by executing the process steps of the cloud-native based data service operating environment isolation method, that is, technical personnel in this field can understand the cloud-native based data service operating environment isolation method as a preferred implementation of the cloud-native based data service operating environment isolation system.

[0089] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0090] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A cloud-native data service operating environment isolation system, characterized in that: include: The cluster management module is used to obtain the registered cluster information and manage the cluster information; The resource group management module is used to obtain the registered resource group information and manage the resource group information; The operating environment management module is used to select the corresponding resource group information and cluster information according to the needs, create a corresponding operating environment for the API on the container platform based on the selected resource group information and cluster information, and manage the operating environment.

2. The cloud-native data service operating environment isolation system according to claim 1 is characterized in that: In the cluster management module, the registered cluster information is obtained including: the English name of the cluster and the Chinese name of the cluster; In the resource group management module, the registered resource group information obtained includes: cluster, resource group name, resource group description, project name, memory limit, CPU limit and replica number limit.

3. The cloud-native-based data service operating environment isolation system according to claim 1, characterized in that: In the operating environment management module, creating a corresponding operating environment for the API on the container platform includes: creating a corresponding operating environment and importing the corresponding operating environment; The creation of the corresponding operating environment includes creating a Deployment object and configuring the parameters of the corresponding operating environment, including: operating environment name, description, workspace, number of CPU cores, memory, number of replicas, resource group, mirror, whether to enable log collection, whether to enable health check, and configuration template; The importing of the corresponding operating environment includes: importing a Deployment object, configuring the parameters of the corresponding operating environment, including: resource group, application name, environment, operating environment name, description, workspace, and configuration template; based on the configured resource group and application name parameters, querying the corresponding operating environment according to the resource group and application name.

4. The cloud-native-based data service operating environment isolation system according to claim 1, characterized in that: The operating environment management module includes: modifying the operating environment, starting the operating environment or restarting the operating environment, stopping the operating environment, mirroring adjustment, and configuring the operating environment; The modification of the operating environment is achieved by modifying the YAML or JSON configuration file of the Deployment, including modifying at least one of the operating environment name, description, workspace, number of CPU cores, memory, number of replicas, resource group, image, whether to enable log collection, whether to enable health check, and configuration template; The starting operating environment or restarting the operating environment is to start the created or imported operating environment; Stopping the running environment is achieved by adjusting the number of Deployment replicas to 0; The image adjustment is achieved by updating the image version in the Deployment configuration, including: viewing the current image and modifying it to the target image; restarting the operating environment after the image adjustment; The configuration operating environment is the maximum number of returned records and the dedicated engine of the configuration operating environment.

5. The cloud-native-based data service operating environment isolation system according to claim 1, characterized in that: Query the operating environment corresponding to the API through query parameters; wherein the query parameters include: at least one parameter of the operating environment name, the workspace to which it belongs, the resource group, and the image name; After the query is completed, the information displayed includes: operating environment name, description, workspace, occupied resources, resource group, image name, service status, whether log collection is enabled, whether health check is enabled, and access address.

6. A cloud-native data service operating environment isolation method, characterized in that: include: Step S1: Obtain registered cluster information and manage the cluster information; Step S2: Obtain the registered resource group information and manage the resource group information; Step S3: Select corresponding resource group information and cluster information according to requirements, create a corresponding operating environment for the API on the container platform based on the selected resource group information and cluster information, and manage the operating environment.

7. The cloud-native-based data service operating environment isolation method according to claim 6, characterized in that: The cluster information obtained and registered includes: the English name of the cluster and the Chinese name of the cluster; The resource group information obtained and registered includes: cluster, resource group name, resource group description, project name, memory limit, CPU limit and replica number limit.

8. The cloud-native-based data service operating environment isolation method according to claim 6, characterized in that: The creating a corresponding operating environment for the API on the container platform includes: creating a corresponding operating environment and importing the corresponding operating environment; The creation of the corresponding operating environment includes: creating a Deployment object, configuring the parameters of the corresponding operating environment, including: operating environment name, description, workspace, number of CPU cores, memory, number of replicas, resource group, mirror, whether to enable log collection, whether to enable health check, and configuration template; The importing of the corresponding operating environment includes: importing a Deployment object, configuring the parameters of the corresponding operating environment, including: resource group, application name, environment, operating environment name, description, workspace, and configuration template; based on the configured resource group and application name parameters, querying the corresponding operating environment according to the resource group and application name.

9. The cloud-native-based data service operating environment isolation method according to claim 6, characterized in that: The management of the operating environment includes: modifying the operating environment, starting the operating environment or restarting the operating environment, stopping the operating environment, mirroring adjustment, and configuring the operating environment; The modification of the operating environment is achieved by modifying the YAML or JSON configuration file of the Deployment, including modifying at least one of the operating environment name, description, workspace, number of CPU cores, memory, number of replicas, resource group, image, whether to enable log collection, whether to enable health check, and configuration template; The starting operating environment or restarting the operating environment is to start the created or imported operating environment; Stopping the running environment is achieved by adjusting the number of Deployment replicas to 0; The image adjustment is achieved by updating the image version in the Deployment configuration, including: viewing the current image and modifying it to the target image; restarting the operating environment after the image adjustment; The configuration operating environment is the maximum number of returned records and the dedicated engine of the configuration operating environment.

10. The cloud-native-based data service operating environment isolation method according to claim 6, characterized in that: Query the operating environment corresponding to the API through query parameters; wherein the query parameters include: at least one parameter of the operating environment name, the workspace to which it belongs, the resource group, and the image name; After the query is completed, the information displayed includes: operating environment name, description, workspace, occupied resources, resource group, image name, service status, whether log collection is enabled, whether health check is enabled, and access address.

Images