AI intelligent chat access method and device based on multi-tenant isolation, equipment and medium

Abstract

The invention discloses an AI intelligent chat access method and device based on multi-tenant isolation, equipment and a medium, and relates to the field of computer software, and the method comprises the steps: extracting a tenant identifier in a target request sent by an AI chat client; querying a tenant-model instance mapping table based on the tenant identifier and a preset tenant configuration rule; the tenant configuration rule is used for defining an isolation level corresponding to the current tenant; if the target model instance corresponding to the tenant identifier is queried, forwarding the target request to the target model instance; if not, determining a corresponding target model instance for the current tenant according to a tenant configuration rule, and forwarding the target request to the target model instance; carrying out AI reasoning by utilizing the target model instance; the target model instance is used for providing an exclusive model operation environment isolated from tenants not at the current isolation level for the tenants at the current isolation level; and receiving a reasoning result returned by the target model instance, and feeding back the reasoning result to the AI chat client.

Description

A method, device, equipment, and medium for accessing AI-powered intelligent chat based on multi-tenant isolation. Technical Field

[0001] This application relates to the field of computer software, and in particular to an AI intelligent chat access method, device, equipment and medium based on multi-tenant isolation. Background Technology

[0002] With the rapid popularization and maturation of Large Language Model (LLM) technology, enterprise-level AI conversational applications are accelerating their transformation to the Software as a Service (SaaS) model. In this model, service providers offer services to multiple different customer organizations through a single software instance. This architecture is known as a multi-tenant architecture, and its core requirement is to ensure data, configuration, and performance isolation between different tenants while sharing infrastructure. However, in the specific scenario of AI (Artificial Intelligence) intelligent chat services, achieving effective multi-tenant isolation faces unprecedented complexity.

[0003] Currently, the closest existing solution in the industry is the "shared model instance + request tagging" approach. In this approach, the system typically uses a load balancer to connect a group of AI model inference servers. Requests from all different tenants are sent to the same or the same group of model instances shared by these servers for processing. The main method for achieving multi-tenancy differentiation is to append a "tenant ID" tag to the request header or request body. When the backend service completes request processing, logs, or stores data, it uses this "tenant ID" to categorize the data and write it into different database partitions or log indexes.

[0004] However, existing solutions have significant technical flaws: on the one hand, all tenants compete for shared computing resources, and high-concurrency requests or complex computing tasks from a single tenant can directly consume resources, causing delays or fluctuations in the service response of other tenants, making it difficult to guarantee service stability; on the other hand, isolation is limited to the data storage stage. During request processing, all tenants' requests are still executed and temporarily stored in the same model instance and its memory space, posing a potential risk of data leakage across tenants due to program errors or memory leaks, resulting in insufficient security; in addition, due to resource sharing, it is difficult to perform accurate resource measurement and billing, and coarse-grained rate limiting based on the number of requests cannot accurately reflect the actual computing resources consumed. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide an AI intelligent chat access method, device, equipment, and medium based on multi-tenant isolation, which can solve the core isolation and resource contention problems in multi-tenant AI chat services. The specific solution is as follows:

[0006] Firstly, this application provides an AI-powered intelligent chat access method based on multi-tenant isolation, including:

[0007] Receive the target request sent by the AI ​​chat client and verify the target request. If the verification is successful, extract the tenant identifier from the target request.

[0008] The tenant-model instance mapping table is queried based on the tenant identifier and the preset tenant configuration rules. The tenant-model instance mapping table is used to store the access endpoints and status information of the tenant identifier and the corresponding model instance. The tenant configuration rules are used to define the isolation level, available model information and resource configuration requirements of the current tenant. The isolation level is used to specify the model instance type of the current tenant. The model instance type includes dedicated instance, tenant group shared instance and public instance.

[0009] If a first target model instance corresponding to the tenant identifier is found in the tenant-model instance mapping table, the target request is forwarded to the first target model instance to perform AI inference using the first target model instance;

[0010] If no first target model instance corresponding to the tenant identifier is found, a second target model instance is determined for the current tenant according to the tenant configuration rules, and the target request is forwarded to the second target model instance to perform AI inference using the second target model instance; wherein, the target model instance is used to provide a dedicated model runtime environment for tenants at the current isolation level, isolated from tenants at other isolation levels, and capable of performing inference processing;

[0011] Receive the inference results returned by the target model instance and feed the inference results back to the AI ​​chat client.

[0012] Optionally, the isolation levels include dedicated instance level, tenant group shared instance level, and public instance level;

[0013] If it is a dedicated instance level, then a model instance that is independently occupied and not shared with other tenants will be allocated to the current tenant.

[0014] If it is a shared instance level for a tenant group, then a shared model instance pool is allocated for multiple tenants belonging to the same preset tenant group so that the requests of the multiple tenants can be routed within the shared model instance pool.

[0015] If it is a public instance level, the current tenant's request will be routed to a public model instance pool that is open to tenants who are not in the public instance level but share the instance level with non-tenant groups.

[0016] Optionally, determining the corresponding second target model instance for the current tenant according to the tenant configuration rules includes:

[0017] Based on the isolation level in the tenant configuration rules, determine whether a new model instance needs to be created for the current tenant;

[0018] If a new model instance needs to be created for the current tenant, the corresponding computing resources are scheduled according to the tenant configuration rules and an independent service container is started. The corresponding model file is loaded in the service container to complete the creation of the new model instance, and the new model instance and the corresponding access endpoint and status information are stored in the tenant-model instance mapping table.

[0019] If it is not necessary to create a new model instance for the current tenant, then according to the tenant configuration rules, select the corresponding model instance for the current tenant from the existing tenant group shared instances or public instances.

[0020] Optionally, the step of scheduling corresponding computing resources and starting an independent service container according to the tenant configuration rules, and loading the corresponding model file in the service container to complete the creation of a new model instance, includes:

[0021] Based on the resource configuration requirements in the tenant configuration rules, the container orchestration platform schedules nodes that meet the requirements in the computing cluster and starts service containers independently allocated for the current tenant on the nodes.

[0022] After the service container is running, it retrieves the corresponding model weight file from the model repository based on the available model information in the tenant configuration rules, and loads the model weight file into the corresponding graphics processor memory to complete the creation of the model instance;

[0023] The model repository is a versioned object storage system used to store different versions of the base model and the model files corresponding to each tenant.

[0024] Optionally, the AI-powered intelligent chat access method based on multi-tenant isolation further includes:

[0025] Obtain the running status of each model instance; the running status includes health status and load status;

[0026] Update the status information of the corresponding model instance entry in the tenant-model instance mapping table according to the running status;

[0027] When any model instance is detected to be unavailable or needs to be reclaimed, the corresponding model instance entry is deleted from the tenant-model instance mapping table.

[0028] Optionally, the AI-powered intelligent chat access method based on multi-tenant isolation further includes:

[0029] Continuously monitor the overall resource utilization rate of model instance pools belonging to the same isolation level;

[0030] If the overall resource utilization rate continues to exceed the preset upper limit threshold within a preset period, a model instance replica is added to the model instance pool, and the model instance replica and the corresponding instance information are added to the tenant-model instance mapping table.

[0031] When the overall resource utilization rate remains below a preset lower threshold for an extended period, a model instance copy is selected and released from the model instance pool, and the tenant-model instance mapping table is updated.

[0032] Optionally, after forwarding the target request to the target model instance, the process may also include:

[0033] Determine the tenant identifier and forwarding timestamp associated with the target request;

[0034] Upon receiving the inference result returned by the target model instance, determine the completion timestamp and resource consumption information corresponding to this inference;

[0035] The tenant identifier, timestamp, and resource consumption information are associated and stored, and a resource consumption statistics report is generated and fed back to the AI ​​chat client.

[0036] Secondly, this application provides an AI-powered intelligent chat access device based on multi-tenant isolation, comprising:

[0037] The data extraction module is used to receive the target request sent by the AI ​​chat client and verify the target request. If the verification is successful, the tenant identifier in the target request is extracted.

[0038] The data query module is used to query the tenant-model instance mapping table based on the tenant identifier and preset tenant configuration rules. The tenant-model instance mapping table is used to store the access endpoints and status information of the tenant identifier and the corresponding model instance. The tenant configuration rules are used to define the isolation level, available model information and resource configuration requirements of the current tenant. The isolation level is used to specify the model instance type of the current tenant. The model instance type includes dedicated instance, tenant group shared instance and public instance.

[0039] The first inference module is used to forward the target request to the first target model instance if a first target model instance corresponding to the tenant identifier is found in the tenant-model instance mapping table, so as to use the first target model instance for AI inference.

[0040] The second inference module is used to determine a corresponding second target model instance for the current tenant according to the tenant configuration rules if no first target model instance corresponding to the tenant identifier is found, and forward the target request to the second target model instance to perform AI inference using the second target model instance; wherein, the target model instance is used to provide a dedicated model running environment for tenants of the current isolation level, isolated from tenants of non-current isolation levels, and capable of performing inference processing;

[0041] The result feedback module is used to receive the inference results returned by the target model instance and feed the inference results back to the AI ​​chat client.

[0042] Thirdly, this application provides an electronic device, comprising:

[0043] Memory, used to store computer programs;

[0044] A processor is used to execute the computer program to implement the aforementioned AI intelligent chat access method based on multi-tenant isolation.

[0045] Fourthly, this application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned AI intelligent chat access method based on multi-tenant isolation.

[0046] In this application, a target request sent by an AI chat client is received and verified. If the verification is successful, the tenant identifier in the target request is extracted. Based on the tenant identifier and preset tenant configuration rules, a tenant-model instance mapping table is queried. The tenant-model instance mapping table stores the access endpoints and status information of the tenant identifier and the corresponding model instance. The tenant configuration rules define the isolation level, available model information, and resource configuration requirements for the current tenant. The isolation level specifies the model instance type corresponding to the current tenant. The model instance type includes dedicated instances, tenant group shared instances, and public instances. If a matching tenant identifier is found in the tenant-model instance mapping table... If a first target model instance is found, the target request is forwarded to the first target model instance to perform AI inference. If no first target model instance corresponding to the tenant identifier is found, a second target model instance is determined for the current tenant according to the tenant configuration rules, and the target request is forwarded to the second target model instance to perform AI inference. The target model instance is used to provide a dedicated model runtime environment for tenants at the current isolation level, isolated from tenants at other isolation levels, and capable of performing inference processing. The inference result returned by the target model instance is received and fed back to the AI ​​chat client. As can be seen from the above, on the one hand, this application achieves instance-level physical or logical isolation by configuring isolation levels for dedicated instances, shared instances for tenant groups, or public instances for different tenants. This limits resource contention to the instance scope corresponding to each tenant, preventing high concurrency or complex requests from a single tenant from crowding out other tenants' resources and ensuring the stability of service response. On the other hand, the target model instance provides a dedicated runtime environment for tenants with corresponding isolation levels. All tenant requests are executed and temporarily stored independently in their respective instance memory spaces, achieving end-to-end isolation from computation and transmission to storage. This fundamentally eliminates cross-tenant data leakage caused by program errors or memory leaks. Furthermore, based on tenant identifiers and preset resource configuration requirements, dedicated or limited-scope model instances are allocated to tenants. The system can accurately record the resource consumption of each tenant on the corresponding instance, replacing traditional coarse-grained request rate limiting and achieving accurate measurement and billing based on actual resource usage. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0048] Figure 1 is a flowchart of an AI intelligent chat access method based on multi-tenant isolation disclosed in this application;

[0049] Figure 2 is a schematic diagram of the overall system architecture disclosed in this application;

[0050] Figure 3 is a timing diagram of an AI intelligent chat access method based on multi-tenant isolation disclosed in this application;

[0051] Figure 4 is a schematic diagram of the structure of an AI intelligent chat access device based on multi-tenant isolation disclosed in this application;

[0052] Figure 5 is a schematic diagram of the structure of an electronic device disclosed in this application. Detailed Implementation

[0053] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0054] Existing solutions have significant technical flaws: on the one hand, all tenants compete for shared computing resources, making service stability difficult to guarantee; on the other hand, during request processing, all tenant requests are still executed and temporarily stored in the same model instance and its memory space, posing a potential risk of cross-tenant data leakage due to program errors or memory leaks; furthermore, due to resource sharing, it is difficult to perform accurate resource measurement and billing, and coarse-grained rate limiting based on the number of requests cannot accurately reflect the actual computing resources consumed. To address these issues, this application provides an AI intelligent chat access method based on multi-tenant isolation. By intelligently routing tenant requests to a dynamically created and completely isolated dedicated model computing environment, it systematically solves the core isolation and resource contention challenges in multi-tenant AI chat services in terms of data security, service performance, model customization, and operational billing.

[0055] Referring to Figure 1, this application discloses an AI intelligent chat access method based on multi-tenant isolation, including:

[0056] Step S11: Receive the target request sent by the AI ​​chat client and verify the target request. If the verification is successful, extract the tenant identifier from the target request.

[0057] The overall system architecture of this embodiment is shown in Figure 2. It adopts a layered design and mainly includes an access gateway layer, a multi-tenant control layer, and a model runtime layer.

[0058] The access gateway layer serves as the unified entry point, comprising an authentication and authorization module, a request parsing module, and an API gateway, responsible for external protocol conversion, user authentication, and initial traffic control. The multi-tenant control layer is the system's brain, containing three core components: a tenant routing decision engine, a resource management and monitoring system, and a policy center. The tenant routing decision engine directs requests to appropriate computing resources according to policies; the resource management and monitoring system collects system resource usage data and performs scaling up and down; the policy center stores tenant configuration rules, including the current tenant's isolation level, available model information, resource configuration requirements, Service Level Agreement (SLA), and routing rules. The model runtime layer consists of multiple isolated model instance pools. Each instance pool can be an independent container or process, loading a specific AI model to serve a specific tenant, tenant group, or as a public basic service.

[0059] In the access gateway layer, firstly, the authentication module authenticates received HTTP (Hypertext Transfer Protocol) / HTTPS (Hypertext Transfer Protocol Secure) requests. It uses JWT (JSON Web Token, a JSON-based security authentication mechanism) or API Key verification to verify the validity of the token signature. Upon successful verification, the token is decoded to parse the claim and extract the tenant identifier from the request. Next, the request parsing module decodes and standardizes the authenticated request body, extracting user messages, conversation history, and system prompt parameters, and encapsulating them into an internally unified request protocol format. Finally, the API (Application Programming Interface) gateway, while fulfilling its reverse proxy responsibilities, integrates a rate-limiting module based on algorithms such as token buckets. Based on the tenant identifier obtained in the preceding steps, it implements coarse-grained request frequency limits, thus forming the system's first line of defense against traffic congestion.

[0060] Step S12: Query the tenant-model instance mapping table based on the tenant identifier and the preset tenant configuration rules; the tenant-model instance mapping table is used to store the access endpoints and status information of the tenant identifier and the corresponding model instance; the tenant configuration rules are used to define the isolation level, available model information and resource configuration requirements of the current tenant; the isolation level is used to clarify the model instance type of the current tenant; the model instance type includes dedicated instance, tenant group shared instance and public instance.

[0061] In this embodiment, the isolation levels include dedicated instance level, tenant group shared instance level, and public instance level. Specifically, at the dedicated instance level, a model instance is allocated to the current tenant and is not shared with other tenants. At the tenant group shared instance level, a shared model instance pool is allocated to multiple tenants belonging to the same preset tenant group, allowing requests from multiple tenants to be routed within the shared model instance pool. At the public instance level, the current tenant's request is routed to a public model instance pool that is open to tenants who are not at the tenant group shared instance level and are not at the public instance level.

[0062] In the multi-tenant control layer, the tenant routing decision engine maintains a dynamic tenant-model instance mapping table and performs routing based on this table and the tenant configuration rules in the policy center. Upon receiving a request with a tenant identifier, the engine first queries the tenant configuration rules in the policy center to obtain the current tenant's configuration rules, such as "use dedicated instance," "use instance in shared group G," or "fall back to public instance if no instance is found." Next, it queries the tenant-model instance mapping table to obtain the corresponding target instance endpoint. Finally, combining the real-time load information provided by the resource management and monitoring system, it selects the available instance with the lowest load from the tenant-model instance mapping table. If there is no corresponding model instance entry in the tenant-model instance mapping table, it can decide whether to create a new instance or directly return to the public instance endpoint based on the tenant configuration rules.

[0063] The strategy center uses a relational database, such as MySQL, or a configuration center, such as Apollo, to store tenant configuration rules. Each record corresponds to a tenant or tenant group, and the fields include: tenant_id (tenant identifier), model_name (specifies the model file used), solution_level (dedicated / shared / public group), resource_quota (maximum number of tokens per month), sla_latency (maximum allowed latency), etc.

[0064] The resource management and monitoring system periodically collects metrics such as GPU (Graphics Processing Unit) utilization, video memory usage, request processing latency, and queue depth by deploying an agent on each model instance side or by utilizing the Kubernetes Metrics Server, and aggregates the data by tenant or instance dimension.

[0065] In addition, the resource management and monitoring system continuously monitors the overall resource utilization of model instance pools belonging to the same isolation level. If the overall resource utilization consistently exceeds a preset upper threshold within a preset period, a model instance replica is added to the model instance pool, and the model instance replica and its corresponding instance information are added to the tenant-model instance mapping table. When the overall resource utilization consistently falls below a preset lower threshold within a preset period, a model instance replica is selected and released from the model instance pool, and the tenant-model instance mapping table is updated. For example, the controller built into the resource management and monitoring system continuously listens to aggregated metrics. When the average GPU utilization of a certain model instance pool consistently exceeds a preset upper threshold, it automatically calls the API of the cloud platform or container orchestration platform to add a replica (Pod) to that model instance pool to expand computing power. If the average GPU utilization consistently falls below a preset lower threshold, it reduces the number of replicas (Pods) in that model instance pool through the same API to optimize resource configuration.

[0066] It should be noted that the keys in the tenant-model instance mapping table are tenant identifiers or tenant group identifiers, and the values ​​are the list of access endpoints and status information of the model instances. In this embodiment, the running status of each model instance can be obtained in real time; the running status includes, but is not limited to, health status and load status. Then, the status information of the corresponding model instance entry in the tenant-model instance mapping table can be updated according to the running status. When any model instance is detected to be unavailable or needs to be reclaimed, the corresponding model instance entry can be deleted from the tenant-model instance mapping table.

[0067] Step S13: If a first target model instance corresponding to the tenant identifier is found in the tenant-model instance mapping table, the target request is forwarded to the first target model instance to perform AI inference using the first target model instance.

[0068] Step S14: If no first target model instance corresponding to the tenant identifier is found, a second target model instance is determined for the current tenant according to the tenant configuration rules, and the target request is forwarded to the second target model instance to perform AI inference using the second target model instance; wherein, the target model instance is used to provide a dedicated model running environment for tenants of the current isolation level, isolated from tenants of non-current isolation levels, and capable of performing inference processing.

[0069] In this embodiment, determining the corresponding second target model instance for the current tenant based on the tenant configuration rules may include: first, determining whether a new model instance needs to be created for the current tenant based on the isolation level in the tenant configuration rules. If a new model instance needs to be created for the current tenant, then according to the tenant configuration rules, the corresponding computing resources are scheduled and an independent service container is started. The corresponding model file is loaded into the service container to complete the creation of the new model instance, and the new model instance, its corresponding access endpoint, and status information are stored in the tenant-model instance mapping table. If a new model instance does not need to be created for the current tenant, then according to the tenant configuration rules, a corresponding model instance is selected for the current tenant from existing shared instances or public instances in the tenant group.

[0070] The process involves scheduling corresponding computing resources and launching independent service containers based on tenant configuration rules. These service containers then load the corresponding model files to create new model instances. This can include: scheduling nodes that meet the resource configuration requirements in the tenant configuration rules within the computing cluster via a container orchestration platform, and launching service containers independently allocated to the current tenant on those nodes. After the service containers are running, the process retrieves the corresponding model weight files from the model repository based on the available model information in the tenant configuration rules, and loads these model weight files into the corresponding graphics processor memory to complete the model instance creation. The model repository is a versioned object storage system used to store different versions of the base model and the model files corresponding to each tenant.

[0071] In the model runtime layer, each instance in the model instance pool is an independent Docker container, running a model service framework such as vLLM, TGI, or Truss. The container image predefines the environment, dependency libraries, and startup scripts adapted to model operation. When the resource manager and monitor receive an instance creation command, they submit a Pod deployment declaration via the Kubernetes API. After the Pod starts, its startup script, based on the model name specified by the policy center, downloads the corresponding model weight file from the versioned object storage system (i.e., the model repository) that stores different versions of the base model and fine-tuned model files for each tenant, and loads it into GPU memory. After successful loading, the instance actively registers its service endpoint with the control layer, completing the construction and readiness of the isolated inference environment.

[0072] It should be noted that, in this embodiment, after forwarding the target request to the target model instance, the process further includes: first, determining the tenant identifier and forwarding timestamp associated with the target request; upon receiving the inference result returned by the target model instance, determining the completion timestamp and resource consumption information corresponding to this inference; then, associating and storing the tenant identifier, timestamp, and resource consumption information, generating a resource consumption statistics report, and feeding the resource consumption statistics report back to the AI ​​chat client.

[0073] Step S15: Receive the inference result returned by the target model instance and feed the inference result back to the AI ​​chat client.

[0074] The technical solution in this embodiment will be described below using Figure 3 as an example.

[0075] The user client first sends a chat request with an identity token to the system. Upon receiving the chat request, the access gateway authenticates the identity token. If authentication is successful, it extracts the tenant ID and internally forwards the request with that tenant ID to the tenant routing decision engine. The tenant routing decision engine queries the policy center and the tenant-model instance mapping table. If there is no corresponding active model instance for the tenant ID, it issues an instruction to the resource management and monitoring system to create a dedicated instance for that tenant. Upon receiving the instruction, the resource management and monitoring system calls the container orchestration platform's API to schedule nodes and start a new Pod (container). After the Pod starts, the model service program within it loads the specified AI model file from the model repository onto the GPU according to the tenant configuration rules in the policy center. After the model instance is loaded, it actively registers its network access endpoint with the resource management and monitoring system. The resource management and monitoring system then updates the "tenant ID-instance endpoint" mapping relationship to the tenant-model instance mapping table. After obtaining the routing target, the tenant routing decision engine forwards the temporarily stored user request to the newly created model instance. The model instance performs AI inference and generates a response, which is then returned to the tenant routing decision engine. The tenant routing decision engine further forwards the response to the access gateway, which ultimately returns the AI ​​response to the user client. Subsequent requests from this tenant will then be directly routed to this created instance until the instance is reclaimed by the system.

[0076] As shown above, the tenant-identified intelligent routing decision-making mechanism in this embodiment can dynamically guide requests to physically or logically isolated dedicated model computing environments, fundamentally eliminating the risk of data confusion and leakage between different tenants during processing, and ensuring enterprise-level data security and privacy. Simultaneously, the dynamic creation and lifecycle management of model instances at the tenant level restricts computing resource competition to a single tenant or tenant group, ensuring the independence and stability of each tenant's service performance. Furthermore, the system supports independently loading customized models and knowledge bases for different tenants, achieving complete decoupling of model updates and deployment, greatly improving service customization flexibility and customer satisfaction. Finally, the refined resource measurement and monitoring throughout the entire chain provides a reliable data foundation for accurate billing and cost allocation based on actual resource consumption, enabling refined management of commercial operations.

[0077] Referring to Figure 4, this application also discloses an AI intelligent chat access device based on multi-tenant isolation, including:

[0078] Data extraction module 11 is used to receive target requests sent by AI chat client and verify the target requests. If the verification is successful, the tenant identifier in the target request is extracted.

[0079] The data query module 12 is used to query the tenant-model instance mapping table based on the tenant identifier and preset tenant configuration rules. The tenant-model instance mapping table is used to store the access endpoints and status information of the tenant identifier and the corresponding model instance. The tenant configuration rules are used to define the isolation level, available model information and resource configuration requirements of the current tenant. The isolation level is used to specify the model instance type of the current tenant. The model instance type includes dedicated instance, tenant group shared instance and public instance.

[0080] The first inference module 13 is used to forward the target request to the first target model instance if a first target model instance corresponding to the tenant identifier is found in the tenant-model instance mapping table, so as to use the first target model instance for AI inference.

[0081] The second inference module 14 is used to determine a corresponding second target model instance for the current tenant according to the tenant configuration rules if no first target model instance corresponding to the tenant identifier is found, and forward the target request to the second target model instance to perform AI inference using the second target model instance; wherein, the target model instance is used to provide a dedicated model running environment for tenants of the current isolation level, isolated from tenants of non-current isolation levels, and capable of performing inference processing;

[0082] The result feedback module 15 is used to receive the inference results returned by the target model instance and feed the inference results back to the AI ​​chat client.

[0083] In some specific implementations, the isolation levels include dedicated instance level, tenant group shared instance level, and public instance level;

[0084] If it is a dedicated instance level, then a model instance that is independently occupied and not shared with other tenants will be allocated to the current tenant.

[0085] If it is a shared instance level for a tenant group, then a shared model instance pool is allocated for multiple tenants belonging to the same preset tenant group so that the requests of the multiple tenants can be routed within the shared model instance pool.

[0086] If it is a public instance level, the current tenant's request will be routed to a public model instance pool that is open to tenants who are not in the public instance level but share the instance level with non-tenant groups.

[0087] In some specific embodiments, the second inference module 14 includes:

[0088] The judgment unit is used to determine whether a new model instance needs to be created for the current tenant based on the isolation level in the tenant configuration rules.

[0089] The instance creation submodule is used to schedule corresponding computing resources and start an independent service container according to the tenant configuration rules if it is necessary to create a new model instance for the current tenant. The corresponding model file is loaded in the service container to complete the creation of the new model instance, and the new model instance and the corresponding access endpoint and status information are stored in the tenant-model instance mapping table.

[0090] The instance selection unit is used to select a corresponding model instance for the current tenant from existing shared or public instances of the tenant group, according to the tenant configuration rules, if it is not necessary to create a new model instance for the current tenant.

[0091] In some specific implementations, the instance creation submodule includes:

[0092] The container startup unit is used to schedule nodes that meet the resource configuration requirements in the tenant configuration rules through the container orchestration platform in the computing cluster, and start service containers independently allocated for the current tenant on the nodes.

[0093] The instance creation unit is used to obtain the corresponding model weight file from the model repository according to the available model information in the tenant configuration rules after the service container is running, and load the model weight file into the corresponding graphics processor memory to complete the creation of the model instance;

[0094] The model repository is a versioned object storage system used to store different versions of the base model and the model files corresponding to each tenant.

[0095] In some specific embodiments, the AI-powered smart chat access device based on multi-tenant isolation further includes:

[0096] The unit is used to obtain the running status of each model instance; the running status includes health status and load status;

[0097] The data update unit is used to update the status information of the corresponding model instance entry in the tenant-model instance mapping table according to the running status.

[0098] The data deletion unit is used to delete the corresponding model instance entry from the tenant-model instance mapping table when it is detected that any model instance is unavailable or needs to be recycled.

[0099] In some specific embodiments, the AI-powered smart chat access device based on multi-tenant isolation further includes:

[0100] The data monitoring unit is used to continuously monitor the overall resource utilization of model instance pools belonging to the same isolation level;

[0101] The instance addition unit is used to add a model instance replica to the model instance pool if the overall resource utilization rate continues to exceed a preset upper limit threshold within a preset period, and add the model instance replica and the corresponding instance information to the tenant-model instance mapping table.

[0102] The instance release unit is used to select and release model instance copies from the model instance pool and update the tenant-model instance mapping table when the overall resource utilization rate is continuously lower than a preset lower threshold within a preset period.

[0103] In some specific embodiments, the AI-powered smart chat access device based on multi-tenant isolation further includes:

[0104] The first data determination unit is used to determine the tenant identifier and forwarding timestamp associated with the target request;

[0105] The second data determination unit is used to determine the completion timestamp and resource consumption information corresponding to this inference when it receives the inference result returned by the target model instance;

[0106] The report generation unit is used to associate and store the tenant identifier, timestamp, and resource consumption information, generate a resource consumption statistical report, and feed the resource consumption statistical report back to the AI ​​chat client.

[0107] Furthermore, this application also discloses an electronic device. FIG5 is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the figure should not be considered as any limitation on the scope of use of this application.

[0108] Figure 5 is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. The electronic device 20 specifically includes: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the AI ​​intelligent chat access method based on multi-tenant isolation disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment can specifically be an electronic computer.

[0109] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0110] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0111] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the multi-tenant isolation-based AI intelligent chat access method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0112] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned AI intelligent chat access method based on multi-tenant isolation. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0113] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0114] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0115] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0116] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0117] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An AI-powered intelligent chat access method based on multi-tenant isolation, characterized in that, include: Receive the target request sent by the AI ​​chat client and verify the target request. If the verification is successful, extract the tenant identifier from the target request. The tenant-model instance mapping table is queried based on the tenant identifier and preset tenant configuration rules. The tenant-model instance mapping table stores the access endpoints and status information of the tenant identifier and the corresponding model instance. The tenant configuration rules define the isolation level, available model information, and resource configuration requirements of the current tenant. The isolation level clarifies the model instance type corresponding to the current tenant. The model instance type includes dedicated instances, tenant group shared instances, and public instances. If a first target model instance corresponding to the tenant identifier is found in the tenant-model instance mapping table, the target request is forwarded to the first target model instance to perform AI inference. If no first target model instance corresponding to the tenant identifier is found, a second target model instance is determined for the current tenant according to the tenant configuration rules, and the target request is forwarded to the second target model instance to perform AI inference. The target model instance provides a dedicated, isolated model runtime environment for tenants at the current isolation level, which is isolated from tenants at other isolation levels, and is capable of performing inference processing. The inference results returned by the target model instance are received and fed back to the AI ​​chat client.

2. The AI ​​intelligent chat access method based on multi-tenant isolation according to claim 1, characterized in that, The isolation levels include dedicated instance level, tenant group shared instance level, and public instance level. Specifically, if it is a dedicated instance level, a model instance is allocated to the current tenant and is not shared with other tenants. If it is a tenant group shared instance level, a shared model instance pool is allocated to multiple tenants belonging to the same preset tenant group, so that requests from these multiple tenants can be routed within the shared model instance pool. If it is a public instance level, the current tenant's request is routed to a public model instance pool that is open to tenants who are not in the tenant group shared instance level and are not in the public instance level.

3. The AI ​​intelligent chat access method based on multi-tenant isolation according to claim 1, characterized in that, The step of determining the corresponding second target model instance for the current tenant according to the tenant configuration rules includes: determining whether a new model instance needs to be created for the current tenant based on the isolation level in the tenant configuration rules; if a new model instance needs to be created for the current tenant, then scheduling the corresponding computing resources and starting an independent service container according to the tenant configuration rules, loading the corresponding model file in the service container to complete the creation of the new model instance, and storing the new model instance and the corresponding access endpoint and status information in the tenant-model instance mapping table; if a new model instance does not need to be created for the current tenant, then selecting the corresponding model instance for the current tenant from existing tenant group shared instances or public instances according to the tenant configuration rules.

4. The AI ​​intelligent chat access method based on multi-tenant isolation according to claim 3, characterized in that, The step of scheduling corresponding computing resources and starting independent service containers according to the tenant configuration rules, and loading corresponding model files in the service containers to complete the creation of new model instances, includes: scheduling nodes that meet the resource configuration requirements in the tenant configuration rules through a container orchestration platform, and starting service containers independently allocated for the current tenant on the nodes; after the service containers are running, obtaining the corresponding model weight files from the model repository according to the available model information in the tenant configuration rules, and loading the model weight files into the corresponding graphics processor memory to complete the creation of model instances; wherein, the model repository is a versioned object storage system used to store different versions of basic models and model files corresponding to each tenant.

5. The AI ​​intelligent chat access method based on multi-tenant isolation according to claim 1, characterized in that, Also includes: Obtain the running status of each model instance; the running status includes health status and load status; Based on the operating status, update the status information of the corresponding model instance entry in the tenant-model instance mapping table; when any model instance is detected to be unavailable or needs to be reclaimed, delete the corresponding model instance entry from the tenant-model instance mapping table.

6. The AI ​​intelligent chat access method based on multi-tenant isolation according to claim 1, characterized in that, Also includes: Continuously monitor the overall resource utilization rate of model instance pools belonging to the same isolation level; If the overall resource utilization rate continuously exceeds the preset upper limit threshold within a preset period, a model instance copy is added to the model instance pool, and the model instance copy and its corresponding instance information are added to the tenant-model instance mapping table; if the overall resource utilization rate continuously falls below the preset lower limit threshold within a preset period, a model instance copy is selected and released from the model instance pool, and the tenant-model instance mapping table is updated.

7. The AI ​​intelligent chat access method based on multi-tenant isolation according to any one of claims 1 to 6, characterized in that, After forwarding the target request to the target model instance, the process further includes: determining the tenant identifier and forwarding timestamp associated with the target request; upon receiving the inference result returned by the target model instance, determining the completion timestamp and resource consumption information corresponding to this inference; associating and storing the tenant identifier, timestamp, and resource consumption information, generating a resource consumption statistics report, and feeding the resource consumption statistics report back to the AI ​​chat client.

8. An AI-powered intelligent chat access device based on multi-tenant isolation, characterized in that, include: The data extraction module is used to receive the target request sent by the AI ​​chat client and verify the target request. If the verification is successful, the tenant identifier in the target request is extracted. The data query module is used to query the tenant-model instance mapping table based on the tenant identifier and preset tenant configuration rules. The tenant-model instance mapping table is used to store the access endpoints and status information of the tenant identifier and the corresponding model instance. The tenant configuration rules are used to define the isolation level, available model information, and resource configuration requirements of the current tenant. The isolation level is used to specify the model instance type corresponding to the current tenant. The model instance type includes dedicated instances, tenant group shared instances, and public instances. The first inference module is used to forward the target request to the first target model instance if a first target model instance corresponding to the tenant identifier is found in the tenant-model instance mapping table, so as to use the first target model instance for AI inference. The second inference module is used to determine a corresponding second target model instance for the current tenant according to the tenant configuration rules if no first target model instance corresponding to the tenant identifier is found, and forward the target request to the second target model instance to perform AI inference using the second target model instance; wherein, the target model instance is used to provide a dedicated model running environment for tenants of the current isolation level, isolated from tenants of other isolation levels, and capable of performing inference processing; the result feedback module is used to receive the inference result returned by the target model instance and feed the inference result back to the AI ​​chat client.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the AI ​​smart chat access method based on multi-tenant isolation as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer programs, which, when executed by a processor, implement the AI ​​intelligent chat access method based on multi-tenant isolation as described in any one of claims 1 to 7.

Images