Connector operation system, method and device for multi-tenant data service

Abstract

The invention is suitable for the technical field of cloud servers, and provides a multi-tenant data service-oriented connector operation system, method and device, and the system comprises a tenant management module, a strategy-driven scheduling module, a connector pool management module, an operation environment management module and a connector task management module. The method has the advantages that through tenant context injection and lightweight sandbox isolation, tenant-level fine-grained hard isolation is realized, and the potential safety hazard of traditional soft isolation is solved; the strategy-driven scheduling module is combined with a dynamic capacity expansion and contraction mechanism, so that the resource scheduling flexibility and fairness in a multi-tenant scene are improved, and resource waste and system bottleneck are avoided; and through full-process auditing and state monitoring, the observability and the security compliance capability of the system are enhanced, and a credible operation guarantee is provided for a multi-tenant environment.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of cloud servers, and particularly relates to a connector running system, method and device for multi-tenant data services. BACKGROUND

[0002] Under the background of rapid development of cloud computing and microservice architecture, more and more enterprises and organizations adopt containerization technology to build flexible and scalable application service platforms. Containerization technology is a virtualization technology that encapsulates application programs and their dependencies, such as libraries, configuration files, etc., in a standardized "container", realizing consistent running of application programs in different environments.

[0003] There are many technical difficulties in the field of containerization technology, such as the construction and isolation of multi-tenant environment, dynamic scheduling, etc. Among them, multi-tenant environment is a software architecture or technology deployment mode, which means that a single software instance or hardware / infrastructure resources simultaneously provide services for multiple independent tenants, and the data and operations of each tenant are isolated and do not interfere with each other.

[0004] In the multi-tenant environment, how to ensure the data security of each tenant and the efficient management of computing tasks is still a technical problem to be solved. Especially in the use scenario of data connectors, the existing solutions often focus on the basic transmission and processing functions of data, but lack comprehensive support for multi-tenant isolation, connector dynamic arrangement and flexible scheduling.

[0005] For example, the multi-tenant isolation in the prior art mainly relies on the soft isolation mechanism in the Kubernetes system. This method only provides basic resource limitation function and does not completely solve the problem of insufficient isolation of different tenants in computing, storage and network. In some high-sensitive scenarios, soft isolation cannot effectively prevent potential resource leakage or mutual interference. Moreover, most of the multi-tenant connector systems based on the original Kubernetes system still use static configuration, fixed allocation rules or simple scheduling strategies, and lack intelligent resource scheduling mechanisms for different tenant characteristics and business needs. This "static" arrangement logic is difficult to dynamically adjust resources according to real-time load, custom priority or task life cycle, which is easy to cause resource waste or system bottleneck.

[0006] Therefore, it can be known that the existing Kubernetes system still has the problems of fixation and static in isolation security policy and resource allocation, which needs to be improved. SUMMARY

[0007] The embodiment of the present application aims to provide a connector running system for multi-tenant data services, aiming to solve the problems of existing Kubernetes systems in isolation security policy and resource allocation, such as fixation and static.

[0008] The embodiment of the present application is implemented by providing a connector running system for multi-tenant data services, which comprises: a tenant management module, a policy-driven scheduling module, a connector pool management module, a running environment management module, and a connector task management module; The tenant management module is configured to receive a task request of a data connection task of a tenant, inject tenant identity and permission information, and record an audit of a task process; The policy-driven scheduling module is configured to change a running node of a connector or expand the running node based on system state information and task characteristics of the data connection task; The connector pool management module is configured to manage registration, caching, and reuse of the connector, and schedule idle instances in the pool to reduce cold start overhead; The running environment management module is configured to allocate an isolated security sandbox for each connector task; The connector task management module is configured to deploy and execute connector task logic in the security sandbox based on a scheduling instruction of the policy-driven scheduling module, and complete data processing and transmission.

[0009] Another object of the embodiment of the present application is to provide a connector running method for multi-tenant data services, which is processed based on the system described above, and the method comprises: After receiving a task request of a data connection of a tenant, tenant identity and permission information are obtained, and a tenant priority is obtained based on the tenant identity information; Based on the tenant priority and real-time load of each running node, a scheduling decision is generated to allocate the task to a running node for processing; Before the task is issued to the target node, it is checked whether there is a connector instance with load lower than a load threshold: if there is, the connector instance with load lower than the load threshold is directly reused; if there is not, a new connector instance is created to execute the task; After the task is allocated to the target node and matched with the connector instance, an independent sandbox environment is created for performance isolation, so that the execution of the task does not affect the tasks of other tenants.

[0010] Another object of the embodiment of the present application is to provide a connector running device for multi-tenant data services, which comprises: An information acquisition unit is configured to acquire tenant identity and permission information when a task request of a data connection of a tenant is received, and acquire a tenant priority based on the tenant identity information; A decision generation unit is configured to generate a scheduling decision based on the tenant priority and real-time load of each running node, so as to allocate the task to the running node for processing; A load judgment unit is configured to check whether there is a connector instance with a load lower than a load threshold before the task is issued to the target node, and if there is, reuse the connector instance with the load lower than the load threshold, and if not, create a new connector instance to execute the task. A sandbox creation unit is configured to create an independent sandbox environment for performance isolation after the task is allocated to the target node and matched with the connector instance, so that the execution of the task does not affect the tasks of other tenants.

[0011] The connector running system for multi-tenant data services provided by the embodiments of the present application has the advantages that the tenant-level fine-grained hard isolation is achieved through tenant context injection and lightweight sandbox isolation, the security risks of traditional soft isolation are solved, the resource scheduling flexibility and fairness in the multi-tenant scenario are improved by the policy-driven scheduling module combined with the dynamic expansion and contraction mechanism, resource waste and system bottlenecks are avoided, the task full-process audit and state monitoring are realized through the setting of the tenant management module, the system observability and security compliance capability are enhanced, and the trusted operation guarantee is provided for the multi-tenant environment. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 An application environment diagram of the connector running system for multi-tenant data services provided by the embodiments of the present application is shown; Figure 2 A module diagram of the connector running system for multi-tenant data services provided by the embodiments of the present application is shown; Figure 3 A step diagram of the connector running method for multi-tenant data services provided by the embodiments of the present application is shown; Figure 4 A module diagram of the connector running device for multi-tenant data services provided by the embodiments of the present application is shown; Figure 5 An internal structure block diagram of the computer device in one embodiment is shown. DETAILED DESCRIPTION

[0013] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0014] It can be understood that the terms "first", "second" and the like used in the present application can be used herein to describe various elements, but unless specifically stated, these elements are not limited by these terms. These terms are only used to distinguish a first unit or module from another unit or module. For example, without departing from the scope of the present application, a first script can be referred to as a second script, and similarly, a second script can be referred to as a first script.

[0015] Figure 1 The application environment diagram of the connector running system for multi-tenant data service provided by the embodiments of the present application is shown in FIG. 1. Figure 1 As shown in the application environment, it includes a tenant terminal 110 and a cloud computer device 120.

[0016] The cloud computer device 120 can be a stand-alone physical server or terminal, or a server cluster composed of multiple physical servers, and can be a cloud server providing cloud server, cloud database, cloud storage, CDN and other basic cloud computing services.

[0017] The tenant terminal 110 can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc., but is not limited thereto. The tenant terminal 110 and the cloud computer device 120 can be connected through a network, which is not limited herein.

[0018] In the embodiments of the present application, the connector refers to a functional component, which can be a lightweight execution unit specially used for realizing data connection, conversion and transmission between heterogeneous systems. The unit runs in a sandbox and is used to provide a micro-service data pipeline, which can be used for data transmission and interaction between different systems in a multi-tenant scenario. Multiple different connectors can perform the same task together or perform different tasks respectively.

[0019] As shown in FIG. 2, in one embodiment, a connector running system for multi-tenant data service is proposed. Figure 2 The embodiments of the present application mainly take the cloud computer device 120 in the above Figure 1 as an example to illustrate. The connector running system for multi-tenant data service can specifically include the following modules: The tenant management module, the policy-driven scheduling module, the connector pool management module, the running environment management module, and the connector task management module; the tenant management module is used for receiving a task request of a data connection task of a tenant, injecting tenant identity and permission information, and performing audit recording on a task process; the policy-driven scheduling module is used for changing a running node of a connector or expanding the running node based on system state information and a task feature of the data connection task; the connector pool management module is used for managing registration, caching, and reuse of the connector, and scheduling idle instances in a pool to reduce cold start overhead; the running environment management module is used for allocating an isolated secure sandbox for each connector task; and the connector task management module is used for deploying and executing connector task logic in the secure sandbox based on a scheduling instruction of the policy-driven scheduling module, and completing data processing and transmission.

[0020] In the embodiment, the tenant management module is an entrance of the system, and is used for receiving a data connection task request submitted by a tenant. Core responsibilities are to identify tenant identity, and inject permission information corresponding to the tenant, such as an accessible data source and operation permission. Meanwhile, a whole life cycle of a task is recorded for audit and tracking, and traceability is ensured.

[0021] The policy-driven scheduling module is used for monitoring system states such as CPU, memory, network, and node load, and a feature of a currently running or to-be-run task, the feature can refer to priority, complexity, data volume, timeliness requirement, and the like, and then making an intelligent scheduling decision based on the information. That is, changing a connector running node, migrating a connector task in running from one node to another node, or expanding the running node, when node load is too high, automatically increasing resources in the node or cluster to process more tasks.

[0022] The connector pool management module is responsible for management of connector instances. Functions thereof are to register, control caching, or reuse when a new connector type or version is added to the system. For example, when a task is completed, a connector instance used by the task is not immediately destroyed, but is placed in a pool and marked as idle. When a new task requests a connector of the same type / configuration, an idle instance is preferentially allocated from the pool, and an initialization process is skipped, so that task startup speed is improved. The module actively manages and allocates idle instances in a scheduling pool to new tasks.

[0023] The running environment management module allocates an independent and isolated runtime environment for each connector task. The sandbox ensures that task execution cannot access task data or resources of other tenants, cannot interfere with a host node or other tasks, and resource use is limited and isolated.

[0024] The connector task management module is an execution engine of the task, configured to receive an instruction of the policy-driven scheduling module on which node to run a control task, obtain or create a connector instance from the connector pool management module, and then deploy the connector instance in a secure sandbox dedicated to the task and execute the business logic of the connector instance, i.e., actual data connection, reading, conversion, writing and other operations, to achieve the goal of data processing and transmission.

[0025] In the embodiment, compared with the existing conventional system, the application has the following advantages. Through tenant context injection and lightweight sandbox isolation, fine-grained hard isolation at the tenant level is achieved, and the security risks of traditional soft isolation are solved. The policy-driven scheduling module combines a dynamic expansion and contraction mechanism to improve the flexibility and fairness of resource scheduling in a multi-tenant scenario, and avoid resource waste and system bottlenecks. Through the setting of the tenant management module, task full-process auditing and state monitoring are achieved, the observability and security compliance capability of the system are enhanced, and reliable operation guarantee is provided for the multi-tenant environment.

[0026] In one embodiment, the tenant management module injects tenant identity and permission information based on the Kubernetes Admission Webhook. The tenant management module is also provided with a Fluent Bit unit for auditing and recording the task flow of the tenant.

[0027] In the embodiment, the system in the application combines the Kubernetes Admission Webhook mechanism to intercept requests and execute custom logic at the "admission control" stage when the APIServer processes the request. Here, the webhook triggered by the tenant management module automatically injects the identity information and permission information such as access tokens, keys, environment variables of the tenant into the configuration file of the Pod before the connector task corresponding to the Pod is created. In this way, the connector running in the Pod can naturally perceive its own tenant context. The module is also integrated with a Fluent Bit log collection processor, which is specially used to collect logs and events related to the task flow of the tenant, such as task submission time, allocated node, start execution time, end time, state, resource consumption, etc. These logs are structured and sent to a centralized log storage and analysis system for persistent storage and audit analysis.

[0028] The embodiment has the advantages that the tenant information injection automation is realized based on a Webhook, manual configuration errors are avoided, and it is ensured that all connector task Pods follow consistent tenant context injection standards. Sensitive information can be safely injected through a Kubernetes Secret, avoiding exposure in plaintext. Through a lightweight and low-overhead log collection scheme provided by Fluent Bit, tenant context can be conveniently attached, greatly simplifying the complexity of task auditing and problem troubleshooting by tenants.

[0029] In one embodiment, the strategy-driven scheduling module expands the running node or changes the running node of the connector based on system state information and task characteristics of the data connection task, and the method is: obtaining system state information, the system state information at least including node load; when the node load of the running node is higher than the node threshold load, expanding the running node once; if the node load of the running node is still higher than the node threshold load for two consecutive periods after expansion, replacing the connector with the highest occupied resource in the running node to a low-load running node.

[0030] In the embodiment, first, system node load state information is obtained, and the scheduling module continuously obtains real-time load indicators of all running nodes from a cluster monitoring system, such as CPU usage and memory usage. When it is monitored that the load of a running node is higher than a preset node threshold load, the scheduling module immediately triggers an expansion operation for the node, and schedules more workload replicas on the node or in the cluster to share the load. If the expansion operation is completed, and the load of the node is still higher than the threshold load after observation for two consecutive monitoring periods, it is indicated that simply expanding the node may not be effective, or there is a hot task. At this time, the scheduling module takes more aggressive measures to identify and migrate the connector task with the highest occupied resource. The system first analyzes all running connector tasks on the node to find the task instance with the largest resource consumption. Then, the heaviest task is migrated to a currently low-load running node for execution to relieve the pressure of the original node.

[0031] The embodiment has the advantages that the relatively simple expansion is preferentially used to relieve the pressure, and the migration operation with higher cost is only taken when the expansion effect is poor, the strategy is more intelligent, and the resource overhead is more reasonable. The first-level expansion operation can quickly cope with load growth, and the second-level migration operation is used to solve the problem of local overload caused by a hot task, preventing single-point failure or performance bottleneck. Through migration of the high-load task, the load distribution of the cluster as a whole is actively optimized, and the resource utilization and system throughput are improved.

[0032] In one embodiment, the method for expanding the running node is: The task weight of the task running in the acquisition node is obtained, and the calculation method of the task weight is:

[0033] wherein, respectively represent the priority feature of the task tenant and the weight thereof, respectively represent the task complexity feature and the weight thereof, respectively represent the task timeliness feature and the weight thereof; The system state information further comprises a load index L, and the running node is expanded based on the task weight and the load index of the running node.

[0034] In the embodiment, the policy-driven scheduling module not only considers the node load when making scheduling decisions, especially expansion decisions, but also considers the importance of the task itself, and the quantitative index is the task weight. The load index is a numerical value of the load degree of the current node, which can comprehensively consider multiple resource usage situations, and the value range thereof is [0, 1], close to 1 indicates that it is closer to the carrying capacity, and close to 0 indicates that there are more idle resources available. is a pre-configured constant, which is used to adjust the influence degree of different features on the maximum weight K. For example, if the system pays more attention to the service experience of the VIP tenant, can be set to be larger, and so on.

[0035] The advantage of the embodiment is that the scheduling decision is upgraded from being based on the load alone to considering the load and the multi-dimensional features of the task itself, so that the decision is more intelligent and more in line with business requirements; high-priority tenants and high-timeliness tasks can obtain necessary resources to meet their service level agreements; different weights are given to tasks with different features, so that more reasonable priority sorting and allocation can be performed when resources are scarce, and key tasks are prevented from being delayed due to simple polling or random allocation.

[0036] In one embodiment, the method for expanding the running node based on the task weight and the load index of the running node is: The load index L of the node is obtained:

[0037] wherein, L represents the load index, represents the Cpu usage rate, represents the memory usage rate, represents the hard disk usage rate, represents the network usage rate, are four proportional coefficients; The newly added capacity value Q satisfies:

[0038] wherein Q represents an added capacity value; the added capacity value Q is rounded up to obtain the number of instances for expanding the running node.

[0039] In the embodiment, the load index L comprehensively considers CPU, memory, disk, and network, and is more capable of reflecting the real bottleneck than a single index. The rounding up refers to taking the minimum integer greater than Q, for example, Q = 3.2, which is rounded up to 4, indicating that 4 connector instance Pods need to be added. The embodiment provides a calculation method of expansion scale, which is used to avoid the problems of insufficient expansion or over-expansion caused by experience or simple threshold. The above scheme simultaneously considers the importance of the task itself and the current load pressure of the node, so that the expansion decision is more accurate. A high-weight task on a high-load node will trigger a larger expansion; the expansion scale Q will change in real time according to K and L, and adapt to different tasks and different node states.

[0040] In one embodiment, the running environment management module is constructed based on a Kubernetes sandbox management platform; The environment management module is integrated with a Seccomp module, an AppArmor module, and a SELinux module to realize permission control and resource isolation of the sandbox.

[0041] In the embodiment, the running environment management module directly utilizes Kubernetes as its underlying infrastructure to create and manage the secure sandbox. Then, the Kubernetes Pod Security Context is defined for the connector task, and the Seccomp Profile is configured; the AppArmor Profile customized for the connector task is loaded on the node, and the use of the AppArmor Profile is specified in the PodAnnotation; finally, the SELinux is enabled in the Kubernetes cluster, and the appropriate SELinux context is configured for the connector task Pod or its namespace, realizing the integration of the above modules.

[0042] The embodiment has the advantage that the triple protection module is complexly integrated, a safer protection boundary is provided, and container escape and privilege escalation attacks are effectively prevented, thereby protecting the host and other tenant tasks on the same node. The above configuration can enable each connector task sandbox to only have the minimum permissions and resource access capabilities required to perform its specific functions, thereby greatly reducing the attack surface.

[0043] In one embodiment, the connector pool management module uses Redis as the index storage of the connector, and uses the SHA-256 algorithm to generate a unique signature of the connector instance, realizing the rapid retrieval and reuse judgment of the connector. The connector pool management module is integrated with a containerd container to implement life cycle management of the connector, and to drive the connector to dynamically switch between an active state, an idle state and a recycling state, thereby optimizing resource utilization.

[0044] In the present embodiment, Redis is used to store metadata indexes of the connector pool, and the index content can include connector type, configuration parameter hash, idle / active state, node, creation time, last use time, etc. When a new connector configuration is registered or an instance is created, the module splices the key configuration information of the connector into a string, calculates a unique, fixed-length hash value signature using a SHA-256 hash algorithm, and the signature uniquely identifies the connector instance of the specific configuration. The system calculates the SHA-256 signature of the connector configuration required by the task. Then, the signature corresponding record in the Redis index is directly queried. If the record is found, and the record indicates that there is a matching instance in the idle state, the instance can be quickly located and reused. The above method is much faster than traversing all instances or comparing all configuration parameters.

[0045] The connector pool management module is also integrated with a containerd container, and the connector pool management module directly interacts with the containerd to manage the actual life cycle of the connector instance, and the connector instance itself runs in the container.

[0046] In one embodiment, the running steps of the connector pool management module are as follows: 1. Open the connector pool management service. 2. When the connector is registered, serialize its configuration into a string, calculate SHA-256 Hash H, and store the metadata of the connector type into Redis using H as the Key. 3. When the connector instance is created, start the corresponding container through the containerd API, and record the association between the instance ID and Hash H. When a task request arrives, calculate the Hash H_req of the required configuration, and query Redis. 4. If H_req exists and the idleInstances list is not empty, take an instance ID from the list, change its state to active, and assign it to the task. If there is no idle instance, create a new instance through containerd, and update Redis. 5. When the task ends, change the instance state back to idle, add it to the idle list, reset the timer, and the background thread monitors the idle instance timer. If it is timed out, containerd is called to delete the container, and the instance record is removed from Redis.

[0047] The embodiment has the advantages that the idle instances are multiplexed, the time-consuming of connector initialization is completely avoided, and the task response speed is greatly improved; the idle instances occupy memory but release CPU; the recycling mechanism ensures that the instances that are no longer needed are timely destroyed, and resources are prevented from being wasted for a long time. Balance is achieved between memory and cold start time; SHA-256 hash provides fast and accurate configuration matching, ensuring that the configuration of the multiplexed instance completely meets the requirements of the new task; Redis provides high-performance index query and state management. Containerd provides stable and reliable container runtime support.

[0048] In one embodiment, as shown in Figure 3 A connector running method for a multi-tenant data service is provided, which is implemented based on the system described above, and the method comprises the following steps: S10: After receiving the task request of the data connection of a tenant, tenant identity and permission information are obtained, and the priority of the tenant is obtained based on the tenant identity information; S20: Based on the tenant priority and the real-time load of each running node, a scheduling decision is generated to allocate the task to a running node for processing; S30: Before the task is issued to the target node, it is checked whether there is a connector instance with a load lower than a load threshold: if there is, the connector instance with a load lower than the load threshold is directly multiplexed; if there is not, a new connector instance is created to execute the task; S40: After the task is allocated to the target node and matched with the connector instance, an independent sandbox environment is created for performance isolation, so that the execution of the task will not affect the tasks of other tenants.

[0049] In this embodiment, the system first receives a request and identity authentication. The identity information and permission information of the tenant are obtained through the tenant management module. Then, based on the tenant identity information, the priority of the tenant is obtained from the tenant management module. The strategy-driven scheduling module comprehensively considers the tenant priority and the real-time load of all running nodes, and generates a scheduling decision using a preset strategy. The core of the decision is to determine which specific running node to allocate the task to for processing. Before the task is about to be issued to the target node, the connector pool management module performs a key check to determine whether there is an idle connector instance that matches the current task requirements and has a current load lower than a certain load threshold on the target node or in the global pool. If there is, the idle connector instance can be directly multiplexed to avoid cold start. If there is not, a connector instance that meets the requirements needs to be newly created on the target node. When the task is allocated to the target node and successfully matched, multiplexed, or a new connector instance is created, the running environment management module creates an independent sandbox environment for the task. The connector instance is deployed and runs in this exclusive sandbox. The sandbox provides performance isolation and security isolation, ensuring that the execution process of the task will not affect the tasks of other tenants in terms of resource consumption and secure access.

[0050] The outstanding advantage of the method provided in this embodiment is that, through tenant context injection and lightweight sandbox isolation, fine-grained hard isolation at the tenant level is achieved, solving the security risks of traditional soft isolation; the policy-driven scheduling module is combined with the dynamic expansion and contraction mechanism to improve the flexibility and fairness of resource scheduling in multi-tenant scenarios, avoiding resource waste and system bottlenecks; through the setting of the tenant management module, full-process auditing and status monitoring of tasks are achieved, enhancing the system's observability and security compliance capabilities, and providing trusted operation guarantees for multi-tenant environments.

[0051] In one embodiment, the method further comprises: During the execution of the task in the sandbox environment, monitor the task activity in real time: If the task is in a highly active state, maintain or increase the supply of system resources; If the task enters the idle state, the system resources occupied by the task node are automatically released; If the task is completed, the system resources are reclaimed.

[0052] In this embodiment, tasks running in the sandbox environment are also continuously monitored. For the description of the above method, please refer to the corresponding device and will not be repeated here. This method reduces performance waste by recycling resources in a timely manner.

[0053] In one embodiment, Figure 4 As shown, a connector operation device for multi-tenant data services is provided. The connector operation device for multi-tenant data services can be integrated into the above-mentioned cloud computer device 120, and specifically may include: The information acquisition unit 510 is configured to, upon receiving a data connection task request from a tenant, acquire the tenant's identity and permission information, and acquire the tenant's priority based on the tenant's identity information; A decision generation unit 520 is used to generate a scheduling decision based on the tenant priority and the current real-time load of each running node to assign tasks to the running nodes for processing; The load judgment unit 530 is used to check whether there is a connector instance with a load lower than the load threshold before sending the task to the target node: if so, directly reuse the connector instance with a load lower than the load threshold; if not, create a new connector instance to execute the task; The sandbox creation unit 540 is used to create an independent sandbox environment for performance isolation after the task is assigned to the target node and matched with the connector instance, so that the execution of the task will not affect the tasks of other tenants.

[0054] In the embodiments of the present application, the explanation and description of the above-mentioned connector running apparatus for multi-tenant data service can refer to the explanation and description of the corresponding method, and the description of the connector running method for multi-tenant data service can refer to the foregoing, which will not be repeated here.

[0055] In the embodiments of the present application, the device has the advantages that, by means of tenant context injection and lightweight sandbox isolation, fine-grained hard isolation at the tenant level is achieved, and the security risks of traditional soft isolation are solved; the policy-driven scheduling module combined with the dynamic expansion and contraction mechanism improves the flexibility and fairness of resource scheduling in the multi-tenant scenario, avoids resource waste and system bottlenecks; the whole-process auditing and state monitoring enhance the observability and security compliance capability of the system, and provide reliable operation guarantee for the multi-tenant environment.

[0056] Figure 5 An internal structure diagram of a computer device in an embodiment is shown. The computer device can be specifically a cloud computer device 120 in Figure 1 As shown in Figure 5 , the computer device includes a processor, a memory, a network interface, an input device and a display screen connected through a system bus. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system, and can also store a computer program, which, when executed by the processor, can enable the processor to implement the connector running method for multi-tenant data service. The internal memory can also store a computer program, which, when executed by the processor, can enable the processor to execute the connector running method for multi-tenant data service.

[0057] In one embodiment, the connector running apparatus for multi-tenant data service provided by the present application can be implemented in the form of a computer program, which can run on the device as shown in Figure 5 . The memory of the device can store various program modules constituting the connector running apparatus for multi-tenant data service. The computer program constituted by the various program modules enables the processor to execute the steps in the connector running method for multi-tenant data service of each embodiment of the present application described in the specification.

[0058] It should be understood that, although each step in the flowcharts of the embodiments of the present application is shown in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless explicitly stated herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least a part of the steps in each embodiment can include a plurality of sub-steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least a part of other steps or sub-steps or stages of other steps.

[0059] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing relevant hardware through a computer program, and the program can be stored in a non-volatile computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments of each method.

[0060] The above-described embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the patent scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A connector operation system for multi-tenant data services, characterized in that: The system comprises: Tenant management module, policy-driven scheduling module, connector pool management module, operating environment management module, and connector task management module; The tenant management module is used to receive the task request of the tenant's data connection task, inject the tenant's identity and permission information, and audit and record the task process; The policy-driven scheduling module is used to change the connector's running node or expand the running node based on system status information and task characteristics of the data connection task; The connector pool management module is used to manage the registration, caching and reuse of connectors, and schedule idle instances in the pool to reduce cold start overhead; The operating environment management module is used to allocate an isolated security sandbox to each connector task; The connector task management module is used to deploy and execute the connector task logic in the security sandbox based on the scheduling instructions of the policy-driven scheduling module to complete data processing and transmission; Among them, the method for expanding the capacity of the running nodes is: Get the task weight of the task running in the node. The task weight is calculated as follows: in, Represent the priority characteristics and weights of task tenants respectively, Represent the task complexity characteristics and their weights respectively, Represent the timeliness characteristics of tasks and their weights respectively; The system status information also includes a load index L. Based on the task weight and the load index of the running node, the running node is expanded. The specific method is as follows: Get the load index L of the node: Where L represents the load index, Indicates CPU usage. Indicates memory usage, Indicates the hard disk usage. Indicates the network usage rate, There are 4 proportional coefficients; For the newly added capacity value Q, the following conditions are satisfied: Among them, Q represents the newly added capacity value; Round up the newly added capacity value Q to get the number of instances for expanding the running node.

2. The system according to claim 1, wherein: The tenant management module injects tenant identity and permission information based on the KubernetesAdmission Webhook; The tenant management module is also provided with a Fluent Bit unit for auditing and recording the tenant's task process.

3. The system according to claim 1, wherein: The policy-driven scheduling module uses the following methods to expand the capacity of running nodes or change the running nodes of connectors based on system status information and task characteristics of data connection tasks: Acquiring system status information, wherein the system status information includes at least node load; When the node load of the running node is higher than the node threshold load, the running node is expanded; If the node load of the running node is still higher than the node threshold load for two consecutive cycles after the expansion, the connector with the highest resource usage in the running node is replaced with a running node with a lower load.

4. The system according to claim 1, wherein: The operating environment management module is built based on the Kubernetes sandbox management platform; The environment management module integrates the Seccomp module, the AppArmor module, and the SELinux module to implement sandbox permission control and resource isolation.

5. The system according to claim 1, wherein: The connector pool management module uses Redis as the index storage for connectors and generates a unique signature for connector instances based on the SHA-256 algorithm to achieve fast retrieval and reuse judgment of connectors. The connector pool management module is integrated with the containerd container to implement the life cycle management of the connector, drive the connector to dynamically switch between the activation state, idle state and recycling state, and optimize resource utilization.

6. A connector operation method for multi-tenant data services, characterized in that: The method is implemented based on the system according to any one of claims 1 to 5, and the method includes: After receiving the task request for the tenant's data connection, obtain the tenant's identity and permission information, and obtain the tenant priority based on the tenant's identity information; Generate scheduling decisions based on tenant priorities and the current real-time load of each running node to assign tasks to running nodes for processing; Before sending the task to the target node, check whether there is a connector instance with a load lower than the load threshold. If so, directly reuse the connector instance with a load lower than the load threshold. If not, create a new connector instance to execute the task. After the task is assigned to the target node and matched with the connector instance, an independent sandbox environment is created for performance isolation so that the execution of the task does not affect the tasks of other tenants.

7. The method according to claim 6, characterized in that The method further comprises: During the execution of the task in the sandbox environment, monitor the task activity in real time: If the task is in a highly active state, maintain or increase the supply of system resources; If the task enters the idle state, the system resources occupied by the task node are automatically released; If the task is completed, the system resources are reclaimed.

8. A connector operation device for multi-tenant data services, characterized in that: The device is implemented based on the system according to any one of claims 1 to 5, and the device includes: An information acquisition unit, configured to, upon receiving a task request for a data connection from a tenant, acquire the tenant's identity and permission information, and acquire the tenant's priority based on the tenant's identity information; The decision generation unit is used to generate scheduling decisions based on tenant priorities and the current real-time load of each running node to allocate tasks to the running nodes for processing; The load judgment unit is used to check whether there is a connector instance with a load lower than the load threshold before sending the task to the target node: if so, the connector instance with a load lower than the load threshold is directly reused; if not, a new connector instance is created to execute the task; The sandbox creation unit is used to create an independent sandbox environment for performance isolation after the task is assigned to the target node and matched with the connector instance, so that the execution of the task does not affect the tasks of other tenants.

Images