AI service processing method and device, medium and equipment

Abstract

The invention relates to an AI service processing method and device, a medium and equipment, and the AI service processing method is applied to an edge processing node, and comprises the steps: receiving an access request of a request end for the AI service of a large model, and extracting a prompt text in the access request; converting the prompt text into a feature vector of a specified dimension, and generating a cache key based on the feature vector; querying a local cache, and if the cache key and the cache content are queried, responding to the cache content to the request end; if the cache key is not queried, determining a target model from a plurality of large models based on the prompt text; and obtaining response content from the target model, responding the response content to the request end, and storing the response content as a value of the cache key in a local cache. Corresponding caches can be generated according to different semantics of the prompt text, the problem that semantic similarity cannot be recognized through traditional character string matching is solved, the cache hit rate is increased, the large model resource utilization rate is optimized, the AI service success rate is increased, and cost is reduced.

Description

Technical Field

[0001] This application relates to the field of edge computing, and more particularly to a method, apparatus, medium, and device for processing AI services. Background Technology

[0002] Existing large language model interface aggregation platforms generally adopt a centralized architecture, requiring payment to the model vendor for each large model call, without a caching mechanism. Real-world testing data shows that approximately 35% of user inquiries in customer service scenarios are repetitive, with numerous repetitive or similar requests repeatedly calling large models, resulting in significant financial waste. Furthermore, cross-regional calls to large models suffer from high latency, failing to meet real-time interaction requirements. Meanwhile, CDN edge nodes have an average CPU load of only 20%, with 80% of their computing power unutilized. Therefore, leveraging the remaining computing power of existing CDN edge nodes to provide AI services and cache AI ​​requests accessing large models can significantly reduce the number of large model calls, saving costs. It can also accelerate AI services and improve response speed. However, the following technical challenges remain to be addressed in building an AI service cache using edge computing power: On the one hand, traditional caching technologies, based on exact string matching, cannot handle the semantic similarity of AI requests, resulting in low cache hit rates. On the other hand, when CDN and AI services are provided simultaneously at edge nodes, memory congestion or I / O contention can easily occur between the two services, making it difficult to guarantee the service quality of the core CDN business. Summary of the Invention

[0003] To overcome the problems existing in related technologies, this application provides a method, apparatus, medium and device for processing AI services.

[0004] According to a first aspect of the embodiments of this application, a method for processing AI services is provided, applied to an edge processing node, including: Receive the requesting client for access to AI services for a large model, and extract the prompt text from the access request; The prompt text is converted into a feature vector of a specified dimension, and a cache key is generated based on the feature vector. The local cache is queried. If the cache key and cached content are found, the cached content is sent back to the requesting end. The response latency of sending the cached content back to the requesting end is less than the response latency of calling the large model to obtain the response content. If the cache key is not found, the target model is determined from multiple large models based on the prompt text. The response content is obtained from the target model, and the response content is sent to the requesting end. It is then stored in the local cache as the value of the cache key.

[0005] Based on the foregoing scheme, in some embodiments of this application, generating a cache key based on the feature vector includes: Sort the feature vectors of the specified dimension in descending order of absolute value, extract the top N features, calculate the hash value, and generate a cache key based on the hash value.

[0006] Based on the foregoing scheme, in some embodiments of this application, generating a cache key based on the hash value includes: Extract the first M bits of the hash value and embed them into a specified format to generate the cache key.

[0007] Based on the foregoing scheme, in some embodiments of this application, determining the target model from multiple large models based on the prompt text includes: If the access request includes information about a specified model, then the specified model is determined to be the target model. If the access request does not include information about the specified model, the target model is determined based on the model selection logic.

[0008] Based on the foregoing scheme, in some embodiments of this application, determining the target model according to the model selection logic includes: Get the real-time price, remaining request limit, and latency for each large model; A weighted score is determined based on the real-time price, remaining request quota, and latency, so that the AI ​​service can dynamically adjust its calling preference for the multiple large models according to the weighted score; Select the target model from the larger models with lower weighted scores.

[0009] Based on the foregoing scheme, in some embodiments of this application, determining the weighted score for the real-time price, remaining request quota, and delay time includes: The real-time price is normalized relative to the benchmark model price to obtain a price normalization value; the remaining request quota is inversely calculated to obtain an availability inverse ratio value; the latency time is normalized relative to a preset latency threshold to obtain a latency normalization value; the price normalization value, availability inverse ratio value, and latency normalization value are weighted and summed according to a first weight, a second weight, and a third weight to obtain the weighted score.

[0010] Based on the foregoing scheme, in some embodiments of this application, selecting the target model from the large models with lower weighted scores includes: Select the model with the lowest weighted score as the target model; or... The target model is the large model whose weighted score is lower than the preset value and whose remaining request amount is greater than the preset value.

[0011] Based on the foregoing solution, in some embodiments of this application, before sending the response content to the requesting end, the method further includes: cleaning the response content; The cleaning of the response content includes: Use a JSON parser to convert the response content into an object; Iterate through the objects and delete fields including usage, system_fingerprint, and logprobs; Reserialize to a JSON string.

[0012] Based on the foregoing solution, in some embodiments of this application, the processing method for the AI ​​service further includes: Set the priority of the AI ​​service process to the lowest priority; Configure the maximum CPU utilization, maximum memory usage, and maximum read / write bandwidth of the AI ​​service.

[0013] Based on the foregoing solution, in some embodiments of this application, the processing method for the AI ​​service further includes: Real-time monitoring of cache hit rate, total CPU utilization, and memory usage of AI services; When the cache hit rate is less than a first preset threshold, or the total CPU utilization is greater than a second preset threshold, or the memory usage of the AI ​​service is greater than a third preset threshold, the response content will no longer be stored.

[0014] Based on the foregoing solution, in some embodiments of this application, the processing method for the AI ​​service further includes: For each AI service access request, record the access cost, customer ID, target model, and whether the cache was hit. Generate customer bills and display the cumulative savings, model usage distribution, cache hit rate, and return on investment to customers through the management interface of the AI ​​service.

[0015] Based on the aforementioned scheme, in some embodiments of this application, the edge processing node reuses the file push and configuration distribution channel of the CDN management system to receive the image package of the AI ​​service.

[0016] According to a second aspect of the embodiments of this application, an AI service processing apparatus is provided, applied to an edge processing node, comprising: The request receiving module is used to receive access requests from the requesting end for AI services of large models and extract the prompt text in the access requests. The cache key generation module is used to convert the prompt text into a feature vector of a specified dimension, and generate a cache key based on the feature vector; The query module is used to query the local cache. If the cache key and cache content are found, the cache content is sent back to the requesting end. The response latency of sending the cache content back to the requesting end is less than the response latency of calling the large model to obtain the response content. If the cache key is not found, the target model is determined from multiple large models based on the prompt text. The caching module is used to obtain response content from the target model, send the response content to the requesting end, and store it in the local cache as the value of the cache key.

[0017] Based on the foregoing solution, in some embodiments of this application, the AI ​​service processing device further includes: The configuration module is used to set the priority of the AI ​​service process to the lowest priority; and to configure the maximum CPU utilization, maximum memory usage, and maximum read / write bandwidth of the AI ​​service.

[0018] Based on the foregoing solution, in some embodiments of this application, the AI ​​service processing device further includes: The circuit breaker module is used to collect cache hit rate, total CPU utilization, and memory usage of AI service in real time; when the cache hit rate is less than a first preset threshold, or the total CPU utilization is greater than a second preset threshold, or the memory usage of AI service is greater than a third preset threshold, the response content will no longer be stored.

[0019] Based on the foregoing solution, in some embodiments of this application, the AI ​​service processing device further includes: The billing generation module is used to record the access cost, customer ID, target model, and cache hit status for each AI service access request; generate customer bills, and display the cumulative savings, model usage distribution, cache hit rate, and return on investment to customers through the management interface of the AI ​​service.

[0020] According to a third aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, wherein when the computer program is executed, the steps of a processing method for an AI service are implemented.

[0021] According to a fourth aspect of the present application, a computer device is provided, including a processor, a memory, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of a processing method for AI services.

[0022] Using the method described in this application, when providing AI services, edge processing nodes extract prompt text from the access requests for AI services, convert the prompt text into multi-dimensional feature vectors, generate cache keys based on the feature vectors, determine the target model from multiple large models based on the prompt text, obtain the corresponding content from the target model, and store the cleaned response content as the value of the cache key in the local cache. This method offers the following advantages: It generates corresponding caches based on the different semantics of the prompt text, solving the problem that traditional string matching cannot identify semantic similarity and improving cache hit rate. Selecting the target model from multiple large models for invocation can fully utilize the real-time price fluctuations and availability of large models, improving the success rate of AI services and reducing costs. By dynamically adjusting the invocation preference for different models, the strategy for determining the target model automatically shifts towards low-cost or high-availability models while ensuring service quality. Furthermore, the management interface intuitively displays relevant indicators, quantifying the commercial value of the technology optimization.

[0023] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0025] Figure 1 This is a flowchart illustrating a method for processing an AI service according to an exemplary embodiment.

[0026] Figure 2 This is a block diagram illustrating an AI service processing apparatus according to an exemplary embodiment.

[0027] Figure 3 This is a block diagram illustrating an AI service processing apparatus according to an exemplary embodiment.

[0028] Figure 4 This is a block diagram illustrating an AI service processing apparatus according to an exemplary embodiment.

[0029] Figure 5 This is a block diagram illustrating an AI service processing apparatus according to an exemplary embodiment.

[0030] Figure 6 This is a block diagram illustrating a computer device according to an exemplary embodiment. Detailed Implementation

[0031] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0032] Existing large language model API aggregation platforms generally adopt a centralized architecture, requiring payment to the model vendor for each large model call, with no caching mechanism. Real-world testing data shows that approximately 35% of user inquiries in customer service scenarios are repetitive, leading to a significant waste of funds due to numerous repeated or similar requests repeatedly calling large models. Furthermore, cross-regional calls to large models experience high latency, failing to meet the needs of real-time interactive scenarios. Meanwhile, CDN edge nodes have an average CPU load of only 20%, leaving 80% of their computing power unutilized.

[0033] To address the problems existing in the prior art, this application provides a method for processing AI services. Figure 1 This is a flowchart illustrating a processing method for an AI service according to an exemplary embodiment. (Reference) Figure 1 The AI ​​service processing method is applied to the edge processing node and includes at least steps S11 to S14.

[0034] Step S11: Receive the request from the requesting end for access to the AI ​​service of the large model, and extract the prompt text in the access request.

[0035] Edge processing nodes include, but are not limited to, network nodes with caching capabilities such as CDN nodes, edge computing gateways, and private cloud access layers. Prompt text is not limited to natural language text, but also includes serialized data converted from multimodal input data (such as image features, speech-to-text, and code logic blocks).

[0036] CDN nodes can be central nodes, upper-layer nodes, or edge nodes. In practical applications, the deployment scope of the FastAPI service can be determined based on actual usage requirements. The FastAPI service can be deployed on some nodes (e.g., central nodes and upper-layer nodes) to achieve limited coverage. Alternatively, the FastAPI service can be deployed on all nodes to achieve wide-area coverage, fully leveraging the advantages of CDN (closer to users, faster response, and full utilization of idle computing power on CDN edge nodes). A larger deployment scope will consume more CDN resources and may potentially impact existing CDN services.

[0037] Access requests for AI services targeting large models are redirected to edge processing nodes. The FastAPI service is deployed on these edge processing nodes, extracting prompt text from the access requests.

[0038] Step S12: Convert the prompt text into a feature vector of a specified dimension, and generate a cache key based on the feature vector.

[0039] The feature vectors of a specified dimension are determined based on the model architecture. For example, a node calls a high-performance open-source inference engine to load a lightweight semantic vector model, performs inference, and obtains a 384-dimensional floating-point feature vector. Then, a cache key is generated based on the feature vector.

[0040] Step S13: Query the local cache. If the cache key and cache content are found, respond with the cache content to the requesting end. The response latency of responding with the cache content to the requesting end is less than the response latency of calling the large model to obtain the response content. If the cache key is not found, determine the target model from multiple large models based on the prompt text.

[0041] If the cache key and content are cached in the local cache, the cached content can be sent to the requesting party, reducing the number of accesses to the large model and lowering costs. Furthermore, since the cached data is used directly to respond to the requesting party, the response latency is lower than that of calling the large model to retrieve the response content, thus improving response speed. If the cache key is not found in the local cache, it means that the current prompt text has not been cached, and the large model needs to be accessed to retrieve the response content. In this embodiment, the edge processing node can access multiple large models and determine the target model from among them based on the prompt text. For example, it can select the large model with the lowest cost as the target model, or select the large model with the fastest response time as the target model.

[0042] Step S14: Obtain the response content from the target model, send the response content to the requesting end, and store it in the local cache as the value of the cache key.

[0043] Sending the access request's prompt text to the target model, retrieving the response content from the target model, and then sending the response content back to the requesting client can effectively improve response speed. The response content is then used as the cache key value, and both the cache key and its value are stored in the local cache. When the same or semantically similar access request is received subsequently, the prompt text is converted into a 384-dimensional feature vector, and a cache key is generated. The cached content can then be quickly retrieved by querying the local cache.

[0044] In this embodiment, the prompt text of the access request is extracted and converted into a feature vector of a specified dimension. A cache key is generated based on the feature vector, and the target model is determined from multiple large models according to the prompt text. After obtaining the response content from the target model and responding to the requesting end, the response content is used as the value of the cache key and stored in the local cache. This enables semantic caching based on access requests, improving the cache hit rate of AI services. Access requests with similar semantics do not need to repeatedly access the large model; they can directly obtain the response content from the cache, improving the cache hit rate, optimizing the utilization of large model resources, and improving response speed. In practical applications, when an access request hits the semantic cache, the edge processing node does not need to access the backend large model across the network, and its response latency is typically in the millisecond range (e.g., 10ms-100ms), while the latency of real-time invocation of the large model is typically in the second range (e.g., 1s-5s). This significant latency difference significantly improves the user experience.

[0045] In an exemplary embodiment, step S12, generating a cache key based on the feature vector, includes: Sort the feature vectors of the specified dimension in descending order of absolute value, extract the top N features, calculate the hash value, and generate a cache key based on the hash value.

[0046] The prompt text is converted into a feature vector of a specified dimension. For example, a lightweight semantic vector model is loaded using a high-performance open-source inference engine, resulting in a 384-dimensional feature vector during inference. Directly hashing the complete vector fails to effectively remove invalid noise and can amplify subtle differences, obscuring the semantic information of the valid dimensions. For instance, for long texts like a book, features are extracted from each page, constructing a 384-dimensional feature vector for the entire book. However, due to numerous blank pages and copyright pages, the content of these pages is added to the feature vector as noise, making it difficult to extract key features. Similarly, for short texts (less than 10 pages), constructing a 384-dimensional feature vector only results in substantial content for the dimensions corresponding to the first 10 pages, while the remaining dimensions are blank. Hashting the complete vector would lead to misjudgments due to the noise from numerous blank pages. Therefore, in this embodiment, the feature vectors of the specified dimensions are sorted in descending order of absolute value, and the top N features are extracted to calculate their hash values. This is equivalent to extracting only the features of the key pages for hashing, accurately identifying core differences. This improves the encoding efficiency of long texts and optimizes short texts, reducing the conflict rate. For example, in an exemplary embodiment, the Top-10 saliency feature extraction algorithm is used to extract only the index values ​​of the top 10 dimensions to form a feature fingerprint. Then, SHA256 hash calculation is performed on the feature fingerprint to generate a cache key, reducing the cache collision rate from 15.3% to 0.3%.

[0047] In one exemplary embodiment, generating a cache key based on a hash value includes: Extract the first M bits of the hash value and embed them into a specified format to generate a cache key.

[0048] The complete hash value can be used as the cache key, or the first M bits of the hash value can be extracted as the cache key. In this embodiment, M can be 8, that is, the first 8 bits of the hash value are extracted, simplifying the long hash into a short hash and reducing the memory usage of the cache.

[0049] For example, in a CDN system providing AI services, the cache key uses the format "ai_cache:{embedding_hash}". "ai_cache" identifies the cached content, indicating that it is a semantic cache, effectively distinguishing it from the cached data of the CDN service. The content within curly braces is the first M bits of the hash value. Since all semantic caches have the same format and differ from the cached data format of the CDN service, if needed later, the KEYS command can be used to easily delete all semantic cache data in batches.

[0050] In an exemplary embodiment, step S13, determining the target model from multiple large models based on the prompt text, includes: If the access request includes information about a specified model, then the specified model is identified as the target model. If the access request does not include information about the specified model, the target model is determined based on the model selection logic.

[0051] The requesting client can specify the large model to be accessed based on its needs, preferences, and expertise. This allows the edge processing node, upon receiving the access request, to initiate a session with the specified large model and retrieve the response content. Alternatively, the requesting client can choose not to specify a large model, allowing the edge processing node to select one independently. Therefore, when an edge processing node receives an access request, if the request includes information about a specified model, it determines that the specified model as the target model. If the request does not include information about a specified model, it selects the target model from among several available large models according to model selection logic.

[0052] In one exemplary embodiment, determining the target model according to model selection logic includes: Get the real-time price, remaining request limit, and latency for each large model; A weighted score is determined based on real-time price, remaining request limit, and latency, so that the AI ​​service can dynamically adjust its calling preference for multiple large models based on the weighted score; Select the target model from the larger models with lower weighted scores.

[0053] The real-time price of each large model may fluctuate in real time; for example, the price of large model A might drop by 30% overnight. Furthermore, some large models may limit the request frequency per minute, i.e., the request quota per minute. Therefore, it's necessary to determine in real time whether each large model has remaining request quota. If a large model has no remaining request quota, routing requests to it will result in request failure. Additionally, differences in call paths, network locations, and the inherent performance of different large models can all lead to varying latency for the same request.

[0054] Therefore, when determining the target model based on the model selection logic, this application fully considers various factors such as real-time price fluctuations of large models, remaining request quota, and latency time, determines the weighted score of each large model, dynamically adjusts the calling tendency of multiple large models based on the weighted score, and selects the target model from the large models with lower weighted scores, thereby reducing costs and improving availability.

[0055] In one exemplary embodiment, determining a weighted score based on real-time price, remaining request limit, and latency includes: The real-time price is normalized relative to the benchmark model price to obtain a price normalization value; the remaining request quota is inversely calculated to obtain an availability inverse ratio value; the latency time is normalized relative to a preset latency threshold to obtain a latency normalization value; the price normalization value, availability inverse ratio value, and latency normalization value are weighted and summed according to a first weight, a second weight, and a third weight to obtain the weighted score.

[0056] You can choose one of several large models as the benchmark model, with a benchmark value of 1.0. The ratio of the unit price of a large model to the unit price of the benchmark model is used as the price normalization value of that large model. For example, if you choose large model A as the benchmark model, its unit price is $0.015 / 1K tokens, and the benchmark value is 1.0; for large model B, its unit price is $0.003 / 1K tokens, and the price normalization value of large model B is 0.003 / 0.015 = 0.2.

[0057] Get the remaining request quota (rpm) for each large model. Availability inverse ratio = 1.0 - 100 / rpm.

[0058] For example, if the remaining request limit for a large model is 500, the inverse availability ratio is 1.0 - 100 / 500 = 0.8.

[0059] If the remaining quota of the large model is 120, the inverse availability ratio = 1.0 - 100 / 120 = 0.167.

[0060] As can be seen, the higher the remaining request quota of a large model, the greater the inverse availability ratio, and the higher the availability of the model. If the remaining request quota is less than 100, the inverse availability ratio is less than 0. In this case, the large model can be directly excluded to prevent request failures due to the exhaustion of the remaining request quota.

[0061] The preset latency threshold is the historical maximum acceptable latency, assuming a preset latency threshold of 2000ms. The latency of the large model B is 800ms, and the latency normalization value is: 800 / 2000 = 0.4. The lower the actual latency, the lower the latency normalization value.

[0062] After obtaining the price normalization value, availability inverse ratio value, and delay normalization value for each large model, a weighted sum can be performed. For example, in this embodiment, the weighted score is: Score = 0.5 × price normalization value + 0.3 × availability inverse ratio value + 0.2 × delay normalization value.

[0063] In this embodiment, a weighted score is determined based on real-time price, remaining request quota, and latency. The largest model is then selected based on this weighted score. This dynamically captures price advantages, improves the utilization rate of large models, and reduces request failure rates. A dynamic balance is achieved between "extremely low price but high latency" and "high price but high availability models." This balance is reflected in the dynamic adjustment of the calling preference for multiple large models. For example, when the weight of a high-performance model is increased, the frequency of calls to that model will increase significantly, achieving an average reduction of over 20% in model call costs while ensuring business success rates.

[0064] In one exemplary embodiment, selecting a target model from large models with lower weighted scores includes: Choose the largest model with the lowest weighted score as the target model; or... Select the large model whose weighted score is lower than the preset value and whose remaining request quota is greater than the preset value as the target model.

[0065] When selecting a target model from multiple large models, the model with the lowest weighted average score can be chosen as the target model, thus minimizing the implementation cost.

[0066] However, according to the weighted scoring formula: Score = 0.5 × Price Normalization + 0.3 × Inverse Availability Ratio + 0.2 × Latency Normalization, when two large models have the same price and latency, the model with a larger remaining request limit will score higher than the model with a smaller remaining request limit. If both models have a remaining request limit greater than 100, selecting the model with the lowest score will result in choosing the model with the smaller remaining request limit. Therefore, one can also select the large model with a weighted average score lower than a preset value and a remaining request limit greater than a preset value from among multiple large models as the target model, balancing cost and availability.

[0067] In an exemplary embodiment, in step S14, before sending the response content to the requesting end, the AI ​​service processing method further includes: cleaning the response content, which includes: Use a JSON parser to convert the response content into an object; Iterate through the objects and delete fields including usage, system_fingerprint, and logprobs; Reserialize to a JSON string.

[0068] By converting the response content into an object and removing specific fields from the object, we can ensure the complete removal of all redundant fields, reduce the size of the response body, send the cleaned response content to the requesting end, and store it as the value of the cache key. This can improve the response rate and reduce the use of cache space.

[0069] In one exemplary embodiment, the processing method for AI services further includes: Set the priority of the AI ​​service process to the lowest priority.

[0070] Configure the maximum CPU utilization, maximum memory usage, and maximum read / write bandwidth for the AI ​​service.

[0071] For example, in scenarios where a CDN system provides AI services, to avoid impacting normal CDN operations, the priority of the AI ​​service process is set to the lowest, and the maximum CPU utilization, maximum memory usage, and maximum read / write bandwidth of the AI ​​service are configured. This ensures the normal operation of core CDN services.

[0072] For example, the edge server configuration includes: a multi-core high-performance processor (e.g., 4 cores, 2.5GHz or higher clock speed, 8GB or more RAM), a 10 Gigabit network card, and CentOS 8 system. To enable the edge server to run AI services while simultaneously providing core CDN services, the AI ​​service is configured with the following settings: maximum CPU utilization: cpu.max = 20000 (20% per core); maximum memory usage: memory.max = 2GB; maximum read / write bandwidth: io.max = 1MB / s. The reverse proxy service main process's nice value is set to -10 (real-time scheduling), and the AI ​​service process's nice value is set to 10 (lowest priority). This ensures that providing AI services does not impact the operation of core CDN services. It also prevents situations where traditional technologies only limit CPU quotas without considering memory consumption, leading to sudden large memory consumption by the AI ​​service and consuming core service cache; and I / O contention, where AI service logs are written to disk, causing the number of disk I / O requests that can be processed per unit time to be filled by AI service logs, resulting in the loss of core CDN logs. By configuring the priority of AI service processes and the maximum CPU utilization, maximum memory usage, and maximum read / write bandwidth of AI services, multi-dimensional resource isolation is achieved, reducing the fluctuation of core CDN business metrics.

[0073] In one exemplary embodiment, the processing method for AI services further includes: Real-time monitoring of cache hit rate, total CPU utilization, and memory usage of AI services; When the cache hit rate is less than the first preset threshold, or the total CPU utilization is greater than the second preset threshold, or the memory usage of the AI ​​service is greater than the third preset threshold, the response content will no longer be stored.

[0074] For example, when a CDN system provides AI services, the core CDN business process and the AI ​​service process run concurrently, sharing the server resources of the nodes. If the core CDN business experiences a sudden surge in traffic, even if the maximum CPU utilization, maximum memory usage, and maximum read / write bandwidth of the AI ​​service are limited, it will still impact the core CDN business.

[0075] Therefore, this application collects CDN cache hit rate, total CPU utilization, and AI service memory usage in real time. The collection frequency can be set to once every 10 seconds, once every 20 seconds, etc.

[0076] CDN cache hit rate includes the cache hit rate of CDN core services and the cache hit rate of AI services. When the CDN cache hit rate is less than the first preset threshold (e.g., CDN cache hit rate < 80%), or the total CPU utilization is greater than the second preset threshold (CPU utilization > 80%), or the memory usage of AI services is greater than the third preset threshold (memory usage > 1.9GB), any of these indicators exceeding the limit may affect CDN core services. In this case, the AI ​​service is circuit-broken, and response content is no longer stored. For example, a lock key with a 300-second expiration is written to the cache. During this period, the response content of AI services will no longer be written to the cache. At the same time, the maximum CPU utilization parameter cpu.max of AI services is reduced to 5000 (5% per core). Although AI service data is no longer written to the cache, semantic cache can still be searched to quickly respond to requests from the requesting end. The computing resources of the node servers are mainly used to handle sudden CDN core service requests, which can achieve rapid fault recovery, reducing the fault recovery time from 90 seconds to 15 seconds. The aforementioned circuit breaker and resource quota configurations, by dynamically adjusting the computing resource allocation for AI services, achieve multi-dimensional isolation for multiple services running concurrently at edge nodes. This ensures that the QoS metrics of the core CDN distribution service are unaffected by fluctuations in AI inference tasks, without increasing hardware costs. When the circuit breaker mechanism is triggered, the system not only stops cache writing for AI services but also initiates service degradation strategies. For example, it may retrieve the most frequently requested responses from the cache for direct response or redirect requests to locally deployed lightweight models to ensure business continuity for CDN nodes under extreme pressure.

[0077] In one exemplary embodiment, the processing method for AI services further includes: For each AI service access request, record the access cost, customer ID, target model, and whether the cache was hit. Generate customer bills and display cumulative savings, model usage distribution, cache hit rate, and return on investment to customers through the AI ​​service management interface. Bills and statistical metrics can be visualized through the AI ​​service's backend management interface. Customers can intuitively see the reduction in large model API calls and the corresponding cumulative savings resulting from semantic caching. This translates technological optimization into quantifiable business value, increasing customer trust and renewal rates, improving response speed, and reducing model call costs.

[0078] In one exemplary embodiment, the edge processing node multiplexes the management system's file push and configuration distribution channel to receive the image package of the AI ​​service.

[0079] For example, in a scenario where a CDN system provides AI services, the AI ​​service image package is distributed to the local path / var / cdn / images / on the edge processing node through the file push function of the CDN management system. The startup script is executed by configuring the distribution channel: loading the local image, creating cgroups, starting the container, and binding resource configurations.

[0080] Additionally, during the gray-scale phase, select nodes with IP addresses ending in 0-9 (10%) for gray-scale testing. Once stable, use automated operation and maintenance tools (such as Ansible) to perform a full deployment in batches. This can keep deployment bandwidth usage below 1% and rollback time below 30 seconds.

[0081] The processing method of the AI ​​service provided in this application will be further described below with specific embodiments. Specific Implementation Example 1 The AI ​​service image package is distributed to the local path / var / cdn / images / on the edge node using the file push function of the CDN management system. The startup script is executed through the configured distribution channel: loading the local image, creating cgroups, starting the container and binding resource configurations.

[0083] Configure the AI ​​service on the edge node, setting the AI ​​service process to the lowest priority. Configure the maximum CPU utilization, maximum memory usage, and maximum read / write bandwidth for the AI ​​service.

[0084] For example, the edge server configuration is as follows: 4-core Intel Xeon 2.5GHz, 8GB RAM, 10 Gigabit Ethernet, CentOS 8 system. The maximum CPU utilization is configured as cpu.max = 20000 (20% per core), the maximum memory usage is configured as memory.max = 2GB, and the maximum read / write bandwidth is configured as io.max = 1MB / s. The Nginx master process's nice value is set to -10 (real-time scheduling), and the AI ​​service process's nice value is set to 10 (lowest priority).

[0085] Lightweight API services deployed in containers (such as FastAPI services) receive access requests from clients for AI services on large models and extract the hint text from the access requests. The hint text is converted into a 384-dimensional vector, and the 384-dimensional vector is sorted in descending order of absolute value. The index values ​​of the first 10 dimensions are used to form a feature fingerprint. Then, SHA256 calculation is performed on the feature fingerprint, and the first 8 bits of the hash value are used to form an 8-bit short hash. The cache key uses the format "ai_cache:{embedding_hash}", where the 8-bit short hash is in curly braces.

[0086] The cache key is queried in the local cache. If the cache key and the corresponding cached content are found, the cached content is responded to the requesting end. At this time, the access request for the AI ​​service for the large model hits the cache, and there is no need to make a request to the large model again, saving the cost of the AI ​​service and improving the response speed.

[0087] If the cache key and corresponding cached content are not found in the local cache, it means that the access request for the AI ​​service of the large model has not yet been cached. In this case, it is further determined whether the access request includes information about the specified model. If the access request specifies a model, for example, the specified model is large model A, then the access request is routed to large model A, and large model A generates the response information.

[0088] If no model is specified in the access request, obtain the real-time price, remaining request limit, and latency of each backend large model. In this case, large model A can be used as the baseline model. The real-time price is normalized relative to the baseline model price to obtain a normalized price value; the remaining request limit is inversely calculated to obtain an inverse availability value; and the latency is normalized relative to a preset latency threshold to obtain a normalized latency value. The normalized price value, inverse availability value, and normalized latency value are weighted and summed according to a first weight, a second weight, and a third weight to obtain a weighted score. The large model with a weighted score lower than a preset value and a remaining request limit greater than a preset value is selected as the target model. Assuming that large model B has the lowest weighted score and a remaining request limit greater than 100, then large model B is selected as the target model, and the access request is routed to large model B. Response information is generated by large model B.

[0089] After receiving the response information, the edge node uses a JSON parser to convert the response content into a JSON object. It then recursively traverses the JSON object, removing redundant fields such as `usage`, `system_fingerprint`, and `logprobs` to clean the response information. The cleaned response information is then reserialized into JSON and stored locally as the cache key. Finally, the cleaned response information is sent back to the requesting client.

[0090] When providing AI services, the maximum CPU utilization rate should always be kept below 20,000 (20% per core), the maximum memory usage below 2GB, and the maximum read / write bandwidth below 1MB / s. This achieves three-dimensional isolation of CPU, memory, and I / O, ensuring the stability of core CDN business metrics.

[0091] Every 10 seconds, the CDN cache hit rate, total CPU utilization, and AI service memory usage of the edge nodes are collected. When the CDN cache hit rate is less than 80%, the total CPU utilization is greater than 80%, or the AI ​​service memory usage exceeds 1.9GB, a circuit breaker operation is executed. A lock key with a 300-second expiration is written to the cache. During this period, the response information of the AI ​​large model is no longer written to the cache, ensuring that the data of the core CDN business can be cached and processed in a timely manner. The maximum CPU utilization of the AI ​​service is modified to 5000 (5% per core). While no longer writing AI service data to the cache, semantic cache can still be searched to quickly respond to requests from the requesting end. In the event of a sudden surge in traffic, such as during a major e-commerce promotion, causing a spike in the data volume of the core CDN business, the AI ​​service can be quickly circuit-broken to ensure the operation of the core CDN business. The AI ​​service will automatically resume after the lock key expires.

[0092] Through the above embodiments, the AI ​​service processing method disclosed in this application can effectively improve the hit rate of semantic cache, avoid excessive calls to large models by repeated or similar requests, and avoid generating large model costs. It fully utilizes the remaining computing power of edge nodes, improving resource utilization. Furthermore, it ensures the smooth operation of core CDN services.

[0093] Figure 2 This is a block diagram illustrating a processing apparatus for an AI service according to an exemplary embodiment. (Reference) Figure 2 The AI ​​service processing device, applied to edge processing nodes, includes a request receiving module 201, a cache key generation module 202, a query module 203, and a cache module 204.

[0094] The request receiving module 201 is configured to receive access requests from the requesting end for AI services for large models and extract the prompt text from the access requests.

[0095] The cache key generation module 202 is configured to convert the prompt text into a feature vector of a specified dimension and generate a cache key based on the feature vector.

[0096] The query module 203 is configured to query the local cache. If the cache key and cache content are found, the cache content is responded to the requesting end. The response latency of responding to the cache content is less than the response latency of calling the large model to obtain the response content. If the cache key is not found, the target model is determined from multiple large models based on the prompt text.

[0097] The caching module 204 is configured to retrieve response content from the target model, send the response content back to the requesting end, and store it in the local cache as the value of the cache key.

[0098] The applicant discovered that directly hashing a complete high-dimensional vector often fails to effectively filter out invalid noise, easily amplifying minor perturbations in the vector and thus obscuring the core semantic information of the effective dimensions. Taking long text processing as an example, if a feature vector is constructed from the content of a book, low-information content such as blank pages and copyright pages will be incorporated into the feature vector as noise, making it difficult to extract key features. Similarly, for short texts with highly uneven information density, directly hashing the complete vector can lead to misjudgments of cache keys due to noise interference from numerous redundant dimensions. Therefore, in an exemplary embodiment, the cache key generation module 202 is further configured to sort the feature vectors of a specified dimension in descending order of absolute value, extract the top N features to calculate hash values, and generate cache keys based on the hash values; that is, only the key dimensions with the highest semantic contribution are extracted for hash calculation.

[0099] In one exemplary embodiment, the cache key generation module 202 is further configured to extract the first M bits of the hash value and embed them in a specified format to generate the cache key. This process not only reduces the valuable memory footprint of edge nodes through short hashes, but also logically isolates semantic cache data from regular CDN service data through a specific prefix identifier. In one exemplary embodiment, the query module 203 is configured to determine the specified model as the target model if the access request includes information about the specified model; If the access request does not include information about the specified model, the target model is determined based on the model selection logic.

[0100] In one exemplary embodiment, the query module 203 is configured to obtain the real-time price, remaining request quota, and latency for each large model; A weighted score is determined based on real-time price, remaining request limit, and latency, so that the AI ​​service can dynamically adjust its calling preference for multiple large models based on the weighted score; Select the target model from the larger models with lower weighted scores.

[0101] In one exemplary embodiment, the caching module 204 is configured to convert the response content into an object using a JSON parser; Iterate through the objects and delete fields including usage, system_fingerprint, and logprobs; Reserialize to a JSON string.

[0102] Figure 3 This is a block diagram illustrating a processing apparatus for an AI service according to an exemplary embodiment. (Reference) Figure 3 The AI ​​service processing device also includes a configuration module 301.

[0103] The configuration module 301 is configured to set the priority of the AI ​​service process to the lowest priority; and to configure the maximum CPU utilization, maximum memory usage, and maximum read / write bandwidth of the AI ​​service.

[0104] Figure 4 This is a block diagram illustrating a processing apparatus for an AI service according to an exemplary embodiment. (Reference) Figure 4 The AI ​​service processing device also includes a circuit breaker module 401.

[0105] The circuit breaker module 401 is configured to collect cache hit rate, total CPU utilization, and AI service memory usage in real time; when the cache hit rate is less than a first preset threshold, or the total CPU utilization is greater than a second preset threshold, or the AI ​​service memory usage is greater than a third preset threshold, the response content will no longer be stored.

[0106] Figure 5 This is a block diagram illustrating a processing apparatus for an AI service according to an exemplary embodiment. (Reference) Figure 5 The AI ​​service processing unit also includes a bill generation module 501.

[0107] The billing module 501 is configured to record the access cost, customer ID, target model, and cache hit rate for each AI service access request; generate a customer bill; and display the cumulative savings, model usage distribution, cache hit rate, and return on investment to the customer through the AI ​​service management interface.

[0108] Figure 6 This is a block diagram illustrating a computer device 600 for data processing according to an exemplary embodiment. For example, the computer device 600 may be provided as a server. (Refer to...) Figure 6 The computer device 600 includes a processor 601, the number of which can be set to one or more as needed. The computer device 600 also includes a memory 602 for storing instructions executable by the processor 601, such as application programs. The number of memories can be set to one or more as needed. The stored application programs can be one or more. The processor 601 is configured to execute instructions to perform the processing methods of the aforementioned AI services.

[0109] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus (devices), or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code. Computer storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data), including but not limited to RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and can include any information delivery medium.

[0110] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0111] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0112] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that an article or device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such an article or device. Without further limitation, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the article or device that includes said element.

[0113] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0114] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations fall within the scope of the claims of this application and their equivalents, the intent of this application also includes these modifications and variations.

Claims

1. A method for processing AI services, characterized in that, Applied to edge processing nodes, including: Receive the requesting client for access to AI services for a large model, and extract the prompt text from the access request; The prompt text is converted into a feature vector of a specified dimension, and a cache key is generated based on the feature vector. The local cache is queried. If the cache key and cached content are found, the cached content is sent back to the requesting end. The response latency of sending the cached content back to the requesting end is less than the response latency of calling the large model to obtain the response content. If the cache key is not found, the target model is determined from multiple large models based on the prompt text. The response content is obtained from the target model, and the response content is sent to the requesting end. It is then stored in the local cache as the value of the cache key.

2. The AI ​​service processing method as described in claim 1, characterized in that, The generation of cache keys based on the feature vector includes: Sort the feature vectors of the specified dimension in descending order of absolute value, extract the top N features, calculate the hash value, and generate a cache key based on the hash value.

3. The AI ​​service processing method as described in claim 2, characterized in that, Generating a cache key based on the hash value includes: Extract the first M bits of the hash value and embed them into a specified format to generate the cache key.

4. The AI ​​service processing method as described in claim 1, characterized in that, The step of determining the target model from multiple large models based on the prompt text includes: If the access request includes information about a specified model, then the specified model is determined to be the target model. If the access request does not include information about the specified model, the target model is determined based on the model selection logic.

5. The AI ​​service processing method as described in claim 4, characterized in that, The process of determining the target model based on model selection logic includes: Get the real-time price, remaining request limit, and latency for each large model; A weighted score is determined based on the real-time price, remaining request quota, and latency, so that the AI ​​service can dynamically adjust its calling preference for the multiple large models according to the weighted score; Select the target model from the larger models with lower weighted scores.

6. The AI ​​service processing method as described in claim 5, characterized in that, The weighted score determination based on the real-time price, remaining request limit, and delay time includes: The real-time price is normalized relative to the benchmark model price to obtain a price normalization value; the remaining request quota is inversely calculated to obtain an availability inverse ratio value; the latency time is normalized relative to a preset latency threshold to obtain a latency normalization value; the price normalization value, availability inverse ratio value, and latency normalization value are weighted and summed according to a first weight, a second weight, and a third weight to obtain the weighted score.

7. The AI ​​service processing method as described in claim 5, characterized in that, Selecting a target model from the larger models with lower weighted scores includes: Select the model with the lowest weighted score as the target model; or... The target model is the large model whose weighted score is lower than the preset value and whose remaining request amount is greater than the preset value.

8. The AI ​​service processing method as described in claim 1, characterized in that, Before sending the response content to the requesting client, the method further includes: cleaning the response content; The cleaning of the response content includes: Use a JSON parser to convert the response content into an object; Iterate through the objects and delete fields including usage, system_fingerprint, and logprobs; Reserialize to a JSON string.

9. The AI ​​service processing method as described in claim 1, characterized in that, The processing method for the AI ​​service also includes: Set the priority of the AI ​​service process to the lowest priority; Configure the maximum CPU utilization, maximum memory usage, and maximum read / write bandwidth of the AI ​​service.

10. The method for processing AI services as described in claim 1, characterized in that, The processing method for the AI ​​service also includes: Real-time monitoring of cache hit rate, total CPU utilization, and memory usage of AI services; When the cache hit rate is less than a first preset threshold, or the total CPU utilization is greater than a second preset threshold, or the memory usage of the AI ​​service is greater than a third preset threshold, the response content will no longer be stored.

11. The method for processing AI services as described in claim 1, characterized in that, The processing method for the AI ​​service also includes: For each AI service access request, record the access cost, customer ID, target model, and whether the cache was hit. Generate customer bills and display the cumulative savings, model usage distribution, cache hit rate, and return on investment to customers through the management interface of the AI ​​service.

12. The AI ​​service processing method as described in claim 1, characterized in that, The edge processing node reuse management system's file push and configuration distribution channel receives the AI ​​service image package.

13. A processing device for AI services, characterized in that, Applied to edge processing nodes, including: The request receiving module is used to receive access requests from the requesting end for AI services of large models and extract the prompt text in the access requests. The cache key generation module is used to convert the prompt text into a feature vector of a specified dimension, and generate a cache key based on the feature vector; The query module is used to query the local cache. If the cache key and cache content are found, the cache content is sent back to the requesting end. The response latency of sending the cache content back to the requesting end is less than the response latency of calling the large model to obtain the response content. If the cache key is not found, the target model is determined from multiple large models based on the prompt text. The caching module is used to obtain response content from the target model, send the response content to the requesting end, and store it in the local cache as the value of the cache key.

14. The AI ​​service processing apparatus as described in claim 13, characterized in that, Also includes: The configuration module is used to set the priority of the AI ​​service process to the lowest priority; and to configure the maximum CPU utilization, maximum memory usage, and maximum read / write bandwidth of the AI ​​service.

15. The AI ​​service processing apparatus as described in claim 13, characterized in that, Also includes: The circuit breaker module is used to collect cache hit rate, total CPU utilization, and memory usage of AI service in real time; when the cache hit rate is less than a first preset threshold, or the total CPU utilization is greater than a second preset threshold, or the memory usage of AI service is greater than a third preset threshold, the response content will no longer be stored.

16. The AI ​​service processing apparatus as described in claim 13, characterized in that, Also includes: The billing module is used to record the access cost, customer ID, target model, and whether the cache was hit for each AI service access request. Generate customer bills and display the cumulative savings, model usage distribution, cache hit rate, and return on investment to customers through the management interface of the AI ​​service.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the steps of the method as described in any one of claims 1-12.

18. A computer device comprising a processor, a memory, and a computer program stored in the memory, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1-12.

Images