A connector operating system, method, and apparatus for multi-tenant data services

Abstract

This invention relates to the field of cloud server technology and provides a connector operating system, method, and apparatus for multi-tenant data services. The system includes: a tenant management module, a policy-driven scheduling module, a connector pool management module, a runtime environment management module, and a connector task management module. The advantages of this application are: by using tenant context injection and lightweight sandbox isolation, fine-grained hard isolation at the tenant level is achieved, solving the security vulnerabilities of traditional soft isolation; the policy-driven scheduling module, combined with a dynamic scaling mechanism, improves the flexibility and fairness of resource scheduling in multi-tenant scenarios, avoiding resource waste and system bottlenecks; and full-process auditing and status monitoring enhance system observability and security compliance capabilities, providing reliable operational assurance for multi-tenant environments.

Description

Technical Field

[0001] This invention belongs to the field of cloud server technology, and in particular relates to a connector operating system, method and apparatus for multi-tenant data services. Background Technology

[0002] With the rapid development of cloud computing and microservice architectures, more and more enterprises and organizations are adopting containerization technology to build flexible and scalable application service platforms. Containerization is a virtualization technology that enables applications to run consistently in different environments by encapsulating applications and their dependencies, such as libraries and configuration files, in a standardized "container."

[0003] Containerization technology presents numerous technical challenges, such as the construction and isolation of multi-tenant environments and dynamic scheduling. A multi-tenant environment is a software architecture or technology deployment model where a single software instance or hardware / infrastructure resource simultaneously provides services to multiple independent tenants, with each tenant's data and operations isolated and uninterrupted.

[0004] In multi-tenant environments, ensuring data security for each tenant and efficient management of computing tasks remains a pressing technical challenge. Especially in data connector applications, existing solutions often focus on basic data transmission and processing functions, but lack comprehensive support for multi-tenant isolation, dynamic connector orchestration, and flexible scheduling.

[0005] For example, existing multi-tenant isolation technologies primarily rely on the soft isolation mechanism in Kubernetes systems. This approach only provides basic resource limiting functionality and does not fully address the insufficient isolation between different tenants in terms of compute, storage, and network. In certain highly sensitive scenarios, soft isolation cannot effectively prevent potential resource leaks or mutual interference. Furthermore, most native multi-tenant connector systems based on Kubernetes still employ static configurations, fixed allocation rules, or simple scheduling strategies, lacking intelligent resource scheduling mechanisms tailored to the characteristics and business needs of different tenants. This "static" orchestration logic makes it difficult to dynamically adjust resources based on real-time load, custom priorities, or task lifecycles, easily leading to resource waste or system bottlenecks.

[0006] Therefore, it can be seen that the existing Kubernetes system still has problems with fixed and static isolation security policies and resource allocation, which need to be improved. Summary of the Invention

[0007] The purpose of this application is to provide a connector operating system for multi-tenant data services, which aims to solve the problems of fixed and static isolation security policies and resource allocation in existing Kubernetes systems.

[0008] This application embodiment is implemented as follows: a connector operating system for multi-tenant data services is provided, the system comprising:

[0009] The module includes a tenant management module, a policy-driven scheduling module, a connector pool management module, a runtime environment management module, and a connector task management module.

[0010] The tenant management module is used to receive task requests for data connection tasks from tenants, inject tenant identity and permission information, and audit and record the task process.

[0011] The strategy-driven scheduling module is used to change the running node of the connector or expand the running node based on system status information and the task characteristics of data connection tasks.

[0012] The connector pool management module is used to manage the registration, caching and reuse of connectors, and schedule idle instances in the pool to reduce cold start overhead;

[0013] The runtime environment management module is used to assign an isolated security sandbox to each connector task;

[0014] The connector task management module is used to deploy and execute connector task logic in a security sandbox based on the scheduling instructions of the policy-driven scheduling module, and complete data processing and transmission.

[0015] Another objective of this application is to provide a connector operation method for multi-tenant data services, the method being based on the system described above, and the method comprising:

[0016] Upon receiving a data connection request from a tenant, obtain the tenant's identity and permission information, and determine the tenant's priority based on the tenant's identity information;

[0017] Based on tenant priority and the real-time load of each running node, a scheduling decision is generated to allocate tasks to the running nodes for processing.

[0018] Before the task is sent to the target node, check if there is a connector instance with a load below the load threshold: if it exists, reuse the connector instance with a load below the load threshold directly; if it does not exist, create a new connector instance to execute the task.

[0019] After a task is assigned to a target node and matched with a connector instance, an independent sandbox environment is created for performance isolation, so that the execution of the task will not affect the tasks of other tenants.

[0020] Another objective of this application is to provide a connector operating device for multi-tenant data services, the device comprising:

[0021] The information acquisition unit is used to acquire the tenant's identity and permission information after receiving a task request for a data connection from a tenant, and to acquire the tenant's priority based on the tenant's identity information.

[0022] The decision generation unit is used to generate scheduling decisions based on tenant priority and the real-time load of each running node, so as to allocate tasks to running nodes for processing;

[0023] The load judgment unit is used to check whether there is a connector instance with a load below the load threshold before the task is sent to the target node: if it exists, the connector instance with a load below the load threshold is reused directly; if it does not exist, a new connector instance is created to execute the task.

[0024] The sandbox creation unit is used to create an independent sandbox environment for performance isolation after a task is assigned to a target node and matched with a connector instance, so that the execution of the task will not affect the tasks of other tenants.

[0025] The connector operation system for multi-tenant data services provided in this application has the following key advantages: it achieves fine-grained hard isolation at the tenant level through tenant context injection and lightweight sandbox isolation, solving the security risks of traditional soft isolation; the policy-driven scheduling module, combined with a dynamic scaling mechanism, improves the flexibility and fairness of resource scheduling in multi-tenant scenarios, avoiding resource waste and system bottlenecks; and through the tenant management module, it enables full-process task auditing and status monitoring, enhancing system observability and security compliance capabilities, and providing reliable operation assurance for multi-tenant environments. Attached Figure Description

[0026] Figure 1 An application environment diagram of a connector operating system for multi-tenant data services provided in this application embodiment;

[0027] Figure 2 This application provides a module diagram of a connector operating system for multi-tenant data services;

[0028] Figure 3 This application provides a flowchart of a connector operation method for multi-tenant data services.

[0029] Figure 4This application provides a module diagram of a connector operating device for multi-tenant data services;

[0030] Figure 5 This is a block diagram of the internal structure of a computer device in one embodiment. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0032] It is understood that the terms "first," "second," etc., used in this application may be used herein to describe various elements, but unless otherwise stated, these elements are not limited by these terms. These terms are used only to distinguish the first unit or module from another unit or module. For example, without departing from the scope of this application, the first script may be referred to as the second script, and similarly, the second script may be referred to as the first script.

[0033] Figure 1 An application environment diagram of the connector operating system for multi-tenant data services provided in this application embodiment is shown below. Figure 1 As shown, this application environment includes a tenant terminal 110 and a cloud computing device 120.

[0034] The cloud computing device 120 can be an independent physical server or terminal, or a server cluster consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud servers, cloud databases, cloud storage, and CDN.

[0035] Tenant terminal 110 may be a smartphone, tablet, laptop, desktop computer, etc., but is not limited to these. Tenant terminal 110 and cloud computing device 120 can be connected via a network, which is not limited herein.

[0036] In this embodiment, a connector refers to a functional component, specifically a lightweight execution unit dedicated to enabling data connection, transformation, and transmission between heterogeneous systems. This unit runs in a sandbox and provides a microservice-based data pipeline, which can be used for data transmission and interaction between different systems in a multi-tenant scenario. Multiple different connectors can jointly perform the same task or each perform different tasks.

[0037] like Figure 2 As shown, in one embodiment, a connector operating system for multi-tenant data services is proposed. This embodiment mainly applies this method to the above-mentioned... Figure 1Taking cloud computing device 120 as an example, a connector operating system for multi-tenant data services may specifically include the following modules:

[0038] The system comprises a tenant management module, a policy-driven scheduling module, a connector pool management module, a runtime environment management module, and a connector task management module. The tenant management module receives task requests for data connection tasks from tenants, injects tenant identity and permission information, and audits and records the task flow. The policy-driven scheduling module changes the running node of a connector or expands the running node based on system status information and the task characteristics of the data connection task. The connector pool management module manages connector registration, caching, and reuse, and schedules idle instances in the pool to reduce cold start overhead. The runtime environment management module allocates an isolated security sandbox for each connector task. The connector task management module deploys and executes connector task logic within the security sandbox based on the scheduling instructions from the policy-driven scheduling module, completing data processing and transmission.

[0039] In this embodiment, the tenant management module serves as the system's entry point, receiving data connection task requests submitted by tenants. Its core responsibility is to identify tenants and inject their corresponding permission information, such as accessible data sources and operational permissions. Simultaneously, it records the entire lifecycle of the task for auditing and tracking, ensuring traceability.

[0040] The policy-driven scheduling module monitors system status such as CPU, memory, network, and node load, as well as the characteristics of currently running or pending tasks. These characteristics can include priority, complexity, data volume, and timeliness requirements. Based on this information, it makes intelligent scheduling decisions. This includes changing the connector's running node, migrating running connector tasks from one node to another, or scaling up the running node. When a node's load is too high, it automatically increases the resources on that node or in the cluster to handle more tasks.

[0041] The connector pool management module is responsible for managing connector instances. Its functions include registering, controlling caching, and reusing new connector types or versions when they are added to the system. For example, when a task completes, the connector instance it used is not immediately destroyed but placed in the pool and marked as idle. When a new task requests a connector of the same type / configuration, an idle instance is preferentially allocated from the pool, skipping the initialization process and improving task startup speed. The module actively manages and allocates idle instances in the scheduling pool to new tasks.

[0042] The runtime environment management module allocates an independent, isolated runtime environment for each connector task. This sandbox ensures that the task cannot access other tenants' task data or resources during execution, nor can it interfere with the host node or other tasks, and resource usage is restricted and isolated.

[0043] The connector task management module is the task execution engine. It receives instructions from the policy-driven scheduling module to control which node the task runs on, obtains or creates connector instances from the connector pool management module, and then deploys the connector instances in the task's dedicated security sandbox and executes their business logic, namely the actual data connection, reading, transformation, writing and other operations, to complete the data processing and transmission goals.

[0044] In this embodiment, compared with existing traditional systems, the outstanding advantages of this application are that, through tenant context injection and lightweight sandbox isolation, fine-grained hard isolation at the tenant level is achieved, solving the security risks of traditional soft isolation; the policy-driven scheduling module, combined with a dynamic scaling mechanism, improves the flexibility and fairness of resource scheduling in multi-tenant scenarios, avoiding resource waste and system bottlenecks; and through the setting of the tenant management module, full-process task auditing and status monitoring are realized, enhancing system observability and security compliance capabilities, and providing reliable operation assurance for multi-tenant environments.

[0045] In one embodiment, the tenant management module injects tenant identity and permission information based on the Kubernetes Admission Webhook;

[0046] The tenant management module also includes a Fluent Bit unit for auditing and recording tenant task processes.

[0047] In this embodiment, the system utilizes the Kubernetes Admission Webhook mechanism to intercept requests and execute custom logic during the "admission control" phase when the APIServer processes them. Specifically, before the Pod corresponding to the connector task is created, a Webhook triggered by the tenant management module automatically injects the tenant's identity information and permission information such as access tokens, keys, and environment variables into the Pod's configuration file. This allows the connector running within the Pod to naturally recognize its tenant context. This module also integrates a Fluent Bit log collection processor, specifically designed to collect logs and events related to the tenant task process, such as task submission time, assigned node, start time, end time, status, and resource consumption. These logs are structured and sent to a centralized log storage and analysis system for persistence and audit analysis.

[0048] The advantage of this embodiment lies in automating tenant information injection based on Webhooks, avoiding manual configuration errors, and ensuring that all connector task Pods adhere to a consistent tenant context injection standard. Sensitive information can be securely injected via Kubernetes Secrets, avoiding plaintext exposure. The lightweight, low-overhead log collection solution provided by Fluent Bit allows for easy attachment of tenant contexts, greatly simplifying the complexity of task auditing and troubleshooting by tenant.

[0049] In one embodiment, the strategy-driven scheduling module expands or changes the running nodes of connectors based on system status information and the task characteristics of data connection tasks, as follows:

[0050] Obtain system status information, which includes at least node load; when the node load of the running node is higher than the node threshold load, expand the capacity of the running node; if the node load of the running node is still higher than the node threshold load for two consecutive cycles after expansion, replace the connector with the highest resource consumption in the running node to a running node with a lower load.

[0051] In this embodiment, the system first acquires the load status information of the nodes. The scheduling module continuously obtains real-time load metrics of all running nodes from the cluster monitoring system, such as CPU utilization and memory utilization. When the load of a running node is detected to be higher than a preset node threshold load, the scheduling module immediately triggers a scaling operation for that node, scheduling more workload replicas on that node or in the cluster to distribute the load. If, after two consecutive monitoring cycles following the scaling operation, the node's load is still higher than the threshold load, it indicates that simply scaling up the node may not be effective, or there may be hot tasks. In this case, the scheduling module takes a more aggressive approach, identifying and migrating the connector task that consumes the most resources. The system first analyzes all running connector tasks on that node to find the task instance with the highest resource consumption. Then, this heaviest task is migrated to a running node with a lower current load to alleviate the pressure on the original node.

[0052] The advantage of this embodiment lies in prioritizing relatively simple scaling to alleviate pressure, and only resorting to more costly migration operations when scaling is ineffective. This strategy is more intelligent and resource consumption is more reasonable. The first-level scaling operation can quickly cope with load growth, while the second-level migration operation is used to solve local overload problems caused by hot tasks, preventing single points of failure or performance bottlenecks. By migrating high-load tasks, the overall load distribution of the cluster is proactively optimized, improving resource utilization and system throughput.

[0053] In one embodiment, the method for expanding the capacity of running nodes is as follows:

[0054] Obtain the task weights of the tasks running in the node. The calculation method for the task weights is as follows:

[0055]

[0056] in, These represent the priority characteristics and weights of the task tenants, respectively. These represent the task complexity features and their weights, respectively. These represent the timeliness characteristics of the task and their weights, respectively.

[0057] The system status information also includes a load index L, which is used to expand the capacity of the running nodes based on the task weights and the load index of the running nodes.

[0058] In this embodiment, when making scheduling decisions, especially scaling decisions, the policy-driven scheduling module considers not only node load but also the importance of the task itself; the quantifiable metric is the task weight. The load index is a numerical value representing the current node's load level, which can comprehensively consider the usage of multiple resources. Its value range is [0,1], where a value closer to 1 indicates that it is closer to its carrying capacity, and a value closer to 0 indicates that there are more idle resources available. `K` is a pre-configured constant used to adjust the degree of influence of different features on the final weight `K`. For example, if the system places more emphasis on the service experience of VIP tenants, It can be set to a larger value, and so on.

[0059] The advantage of this embodiment is that it upgrades the scheduling decision from being based solely on load to considering the multi-dimensional characteristics of both load and the task itself, making the decision more intelligent and more in line with business needs; ensuring that high-priority tenants and time-sensitive tasks obtain the necessary resources to meet their service level agreements; and assigning different weights to tasks with different characteristics, enabling more reasonable priority sorting and allocation when resources are scarce, avoiding delays in critical tasks caused by simple polling or random allocation.

[0060] In one embodiment, the method for scaling up the running nodes based on the task weight and the load index of the running nodes is as follows:

[0061] Get the node's load index L:

[0062]

[0063] Where L represents the load index. Indicates CPU utilization. Indicates memory usage. Indicates hard drive usage. Indicates network usage rate. There are 4 scaling factors;

[0064] For the new capacity value Q, the following conditions must be met:

[0065]

[0066] Where Q represents the new capacity value; rounding up the new capacity value Q gives the number of instances that expand the running node.

[0067] In this embodiment, the load index L comprehensively considers CPU, memory, disk, and network, reflecting the true bottleneck more accurately than a single indicator. Rounding up refers to taking the smallest integer greater than Q. For example, Q = 3.2, rounded up to 4, indicates that 4 new connector instance Pods are needed. This embodiment provides a method for calculating the scaling scale, avoiding under- or over-scaling issues caused by relying on experience or simple thresholds. The above scheme considers both the importance of the task itself and the current load pressure of the node, making scaling decisions more accurate. High-weight tasks on high-load nodes will trigger larger scaling; the scaling scale Q changes in real time according to K and L, adapting to different tasks and node states.

[0068] In one embodiment, the runtime environment management module is built on the Kubernetes sandbox management platform;

[0069] The environment management module integrates the Seccomp module, AppArmor module, and SELinux module to achieve access control and resource isolation within the sandbox.

[0070] In this embodiment, the runtime environment management module directly utilizes Kubernetes as its underlying infrastructure to create and manage security sandboxes. Then, it defines a Kubernetes Pod Security Context for the connector task and configures a Seccomp Profile; loads the AppArmor Profile customized for the connector task on the node and specifies its use in the PodAnnotation; finally, it enables SELinux on the Kubernetes cluster and configures an appropriate SELinux context for the connector task Pod or its namespace, thus integrating the above modules.

[0071] The advantage of this embodiment lies in its triple-protection module integration, which provides a more secure protection boundary, effectively defending against container escape and privilege escalation attacks, and protecting the host and other tenant tasks on the same node. This configuration ensures that each connector task sandbox has only the minimum permissions and resource access capabilities required to perform its specific function, greatly reducing the attack surface.

[0072] In one embodiment, the connector pool management module uses Redis as the index storage for connectors and generates unique signatures for connector instances based on the SHA-256 algorithm to achieve fast retrieval and reuse judgment of connectors.

[0073] The connector pool management module integrates a containerd container to realize connector lifecycle management, drive connectors to dynamically switch between active, idle and recycled states, and optimize resource utilization.

[0074] In this embodiment, Redis is used to store the metadata index of the connector pool. The index content may include connector type, configuration parameter hash, idle / active status, node, creation time, last use time, etc. When a new connector configuration is registered or an instance is created, the module concatenates the key configuration information of the connector into a string, calculates a unique, fixed-length hash signature using the SHA-256 hash algorithm, and this signature uniquely identifies the connector instance of that specific configuration. The system calculates its SHA-256 signature based on the connector configuration required by the task. Then, it directly queries the record corresponding to the signature in the Redis index. If a record is found, and the record indicates the existence of a matching instance in an idle state, the instance can be quickly located and reused. This method is much faster than traversing all instances or comparing all configuration parameters.

[0075] The connector pool management module also integrates the containerd container. The connector pool management module interacts directly with containerd to manage the actual lifecycle of connector instances, which run in a container.

[0076] In one embodiment, the connector pool management module operates as follows: 1. Open the connector pool management service. 2. When a connector registers, its configuration is serialized into a string, its SHA-256 hash H is calculated, and H is used as the key to store the metadata of the connector type in Redis. 3. When creating a connector instance, the corresponding container is started through the containerd API, recording the association between the instance ID and the hash H. When a task request arrives, the required configuration hash H_req is calculated and queried in Redis. 4. If H_req exists and the idleInstances list is not empty, an instance ID is taken from it, its status is changed to active, and it is assigned to a task; if no idle instance exists, a new instance is created through containerd, and Redis is updated. 5. When the task ends, the instance status is changed back to idle, added to the idle list, the timer is reset, and a background thread monitors the idle instance timer. If the timer expires, containerd is called to delete the container, and the instance record is removed from Redis.

[0077] The advantages of this embodiment are: overusing idle instances completely avoids the time-consuming connector initialization, significantly improving task response speed; idle instances occupy memory but release CPU; the recycling mechanism ensures that no longer needed instances are destroyed in a timely manner, avoiding long-term idleness and wasted resources. A balance is achieved between memory and cold start time; SHA-256 hashing provides fast and accurate configuration matching, ensuring that the reused instance configuration fully meets the requirements of new tasks; Redis provides high-performance index queries and state management; and containerd provides stable and reliable container runtime support.

[0078] In one embodiment, such as Figure 3 As shown, a connector operation method for multi-tenant data services is provided. The method is implemented based on the system described above and includes:

[0079] S10: After receiving the tenant's data connection task request, obtain the tenant's identity and permission information, and obtain the tenant priority based on the tenant's identity information;

[0080] S20: Based on tenant priority and the real-time load of each running node, generate scheduling decisions to allocate tasks to running nodes for processing;

[0081] S30: Before the task is sent to the target node, check if there is a connector instance with a load below the load threshold: if it exists, reuse the connector instance with a load below the load threshold directly; if it does not exist, create a new connector instance to execute the task.

[0082] S40: After a task is assigned to a target node and matched with a connector instance, an independent sandbox environment is created for performance isolation so that the execution of the task will not affect the tasks of other tenants.

[0083] In this embodiment, the system first receives a request and performs authentication. The tenant's identity and permission information are obtained through the tenant management module. Then, based on the tenant's identity information, the tenant's priority is obtained from the tenant management module. The policy-driven scheduling module comprehensively considers the tenant priority and the real-time load of all running nodes, using a preset policy to generate a scheduling decision. The core of the decision is determining which specific running node is most suitable for assigning the task. Before the task is sent to the target node, the connector pool management module performs a critical check to see if there is an idle connector instance on the target node or in the global pool that matches the current task requirements and whose current load is below a certain load threshold. If it exists, the idle connector instance can be reused directly to avoid a cold start. If it does not exist, a new connector instance that meets the requirements needs to be created on the target node. After the task is assigned to the target node and a connector instance is successfully matched, reused, or newly created, the runtime environment management module creates an independent sandbox environment for the task. The connector instance is deployed and runs in this dedicated sandbox. The sandbox provides performance isolation and security isolation, ensuring that the execution of this task does not affect other tenants' tasks in terms of resource consumption and secure access.

[0084] The key advantages of the method provided in this embodiment are that it achieves fine-grained hard isolation at the tenant level through tenant context injection and lightweight sandbox isolation, solving the security risks of traditional soft isolation; the policy-driven scheduling module, combined with a dynamic scaling mechanism, improves the flexibility and fairness of resource scheduling in multi-tenant scenarios, avoiding resource waste and system bottlenecks; and the tenant management module enables full-process task auditing and status monitoring, enhancing system observability and security compliance capabilities, and providing reliable operational assurance for multi-tenant environments.

[0085] In one embodiment, the method further includes:

[0086] During the execution of the task in the sandbox environment, the task activity is monitored in real time:

[0087] If the task is in a highly active state, maintain or increase the supply of system resources;

[0088] If a task enters an idle state, the system resources occupied by the task node will be automatically released;

[0089] If the task is completed, system resources will be reclaimed.

[0090] In this embodiment, tasks running in the sandbox environment are also continuously monitored. For a description of the above method, please refer to the corresponding device; it will not be repeated here. This method reduces performance waste by promptly reclaiming resources.

[0091] In one embodiment, such as Figure 4As shown, a connector operating device for multi-tenant data services is provided. This connector operating device for multi-tenant data services can be integrated into the aforementioned cloud computing device 120, and specifically may include:

[0092] The information acquisition unit 510 is used to acquire the tenant's identity and permission information after receiving a task request for a data connection from a tenant, and to acquire the tenant's priority based on the tenant's identity information.

[0093] The decision generation unit 520 is used to generate scheduling decisions based on tenant priority and the real-time load of each running node, so as to allocate tasks to running nodes for processing.

[0094] The load judgment unit 530 is used to check whether there is a connector instance with a load lower than the load threshold before the task is sent to the target node: if it exists, the connector instance with a load lower than the load threshold is reused directly; if it does not exist, a new connector instance is created to execute the task.

[0095] Sandbox creation unit 540 is used to create an independent sandbox environment for performance isolation after a task is assigned to a target node and matched with a connector instance, so that the execution of the task will not affect the tasks of other tenants.

[0096] In the embodiments of this application, the explanation and description of the connector operation device for multi-tenant data services can be referred to the explanation and description of the corresponding method. For the description of the connector operation method for multi-tenant data services, please refer to the above text, which will not be repeated here.

[0097] In this embodiment, the advantages of this device are that it achieves fine-grained hard isolation at the tenant level through tenant context injection and lightweight sandbox isolation, solving the security risks of traditional soft isolation; the policy-driven scheduling module, combined with a dynamic scaling mechanism, improves the flexibility and fairness of resource scheduling in multi-tenant scenarios, avoiding resource waste and system bottlenecks; and full-process auditing and status monitoring enhance system observability and security compliance capabilities, providing reliable operation assurance for multi-tenant environments.

[0098] Figure 5 An internal structural diagram of a computer device in one embodiment is shown. Specifically, this computer device may be... Figure 1 120 cloud-based computing devices. For example... Figure 5As shown, the computer device includes a processor, memory, network interface, input device, and display screen connected via a system bus. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system and may also store a computer program. When executed by the processor, this computer program enables the processor to implement a connector operation method for multi-tenant data services. The internal memory may also store a computer program, which, when executed by the processor, enables the processor to implement a connector operation method for multi-tenant data services.

[0099] In one embodiment, the connector operating device for multi-tenant data services provided in this application can be implemented as a computer program, which can be implemented in the form of, for example... Figure 5 The device described above operates on the device. The device's memory can store various program modules that constitute the connector operation apparatus for multi-tenant data services. The computer program comprised of these program modules causes the processor to execute the steps of the connector operation method for multi-tenant data services according to the various embodiments of this application described herein.

[0100] It should be understood that although the steps in the flowcharts of the various embodiments of this application are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in each embodiment may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0101] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods.

[0102] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A connector operating system for multi-tenant data services, characterized in that, The system includes: The module includes a tenant management module, a policy-driven scheduling module, a connector pool management module, a runtime environment management module, and a connector task management module. The tenant management module is used to receive task requests for data connection tasks from tenants, inject tenant identity and permission information, and audit and record the task process. The strategy-driven scheduling module is used to change the running node of the connector or expand the running node based on system status information and the task characteristics of data connection tasks. The connector pool management module is used to manage the registration, caching and reuse of connectors, and schedule idle instances in the pool to reduce cold start overhead; The runtime environment management module is used to assign an isolated security sandbox to each connector task; The connector task management module is used to deploy and execute connector task logic in a security sandbox based on the scheduling instructions of the policy-driven scheduling module, and complete data processing and transmission. The method for expanding the capacity of running nodes is as follows: Obtain the task weights of the tasks running in the node. The calculation method for the task weights is as follows: in, These represent the priority characteristics and weights of the task tenants, respectively. These represent the task complexity features and their weights, respectively. These represent the timeliness characteristics of the task and their weights, respectively. The system status information also includes a load index L. Based on the task weight and the load index of the running node, the running node is expanded. The specific method is as follows: Get the node's load index L: Where L represents the load index. Indicates CPU utilization. Indicates memory usage. Indicates hard drive usage. Indicates network usage rate. There are 4 scaling factors; For the new capacity value Q, the following conditions must be met: Where Q represents the new capacity value; Round up the newly added capacity value Q to get the number of instances that expand the running node.

2. The system according to claim 1, characterized in that, The tenant management module injects tenant identity and permission information based on KubernetesAdmission Webhook; The tenant management module also includes a Fluent Bit unit for auditing and recording tenant task processes.

3. The system according to claim 1, characterized in that, The strategy-driven scheduling module, based on system status information and the task characteristics of data connection tasks, expands or changes the running nodes of connectors using the following method: Obtain system status information, which includes at least node load; When the node load of the running node exceeds the node threshold load, the running node is expanded once; If the node load of the running node is still higher than the node threshold load for two consecutive cycles after the expansion, the connector with the highest resource consumption in the running node will be replaced with a running node with a lower load.

4. The system according to claim 1, characterized in that, The runtime environment management module is built on the Kubernetes sandbox management platform; The environment management module integrates the Seccomp module, AppArmor module, and SELinux module to achieve access control and resource isolation within the sandbox.

5. The system according to claim 1, characterized in that, The connector pool management module uses Redis as the index storage for connectors and generates unique signatures for connector instances based on the SHA-256 algorithm to enable fast retrieval and reuse judgment of connectors. The connector pool management module integrates a containerd container to realize connector lifecycle management, drive connectors to dynamically switch between active, idle and recycled states, and optimize resource utilization.

6. A connector operation method for multi-tenant data services, characterized in that, The method is implemented based on the system described in any one of claims 1-5, and the method includes: Upon receiving a data connection request from a tenant, obtain the tenant's identity and permission information, and determine the tenant's priority based on the tenant's identity information; Based on tenant priority and the real-time load of each running node, a scheduling decision is generated to allocate tasks to the running nodes for processing. Before the task is sent to the target node, check if there is a connector instance with a load below the load threshold: if it exists, reuse the connector instance with a load below the load threshold directly; if it does not exist, create a new connector instance to execute the task. After a task is assigned to a target node and matched with a connector instance, an independent sandbox environment is created for performance isolation, so that the execution of the task will not affect the tasks of other tenants.

7. The method according to claim 6, characterized in that, The method further includes: During the execution of the task in the sandbox environment, the task activity is monitored in real time: If the task is in a highly active state, maintain or increase the supply of system resources; If a task enters an idle state, the system resources occupied by the task node will be automatically released; If the task is completed, system resources will be reclaimed.

8. A connector operating device for multi-tenant data services, characterized in that, The apparatus is implemented based on the system as described in any one of claims 1-5, and the apparatus comprises: The information acquisition unit is used to acquire the tenant's identity and permission information after receiving a task request for a data connection from a tenant, and to acquire the tenant's priority based on the tenant's identity information. The decision generation unit is used to generate scheduling decisions based on tenant priority and the real-time load of each running node, so as to allocate tasks to running nodes for processing; The load judgment unit is used to check whether there is a connector instance with a load below the load threshold before the task is sent to the target node: if it exists, the connector instance with a load below the load threshold is reused directly; if it does not exist, a new connector instance is created to execute the task. The sandbox creation unit is used to create an independent sandbox environment for performance isolation after a task is assigned to a target node and matched with a connector instance, so that the execution of the task will not affect the tasks of other tenants.

Images