Load balancing method, device, electronic device and storage medium for inference service

Abstract

The present disclosure provides a load balancing method, device, electronic device and storage medium for an inference service, which relates to the field of computer technology, and includes: in the process of an inference service node set processing a session request, obtaining load status information of any inference service node in the inference service node set; when the load status information indicates that the node pressure of any inference service node is greater than a pressure threshold, creating a proxy service corresponding to any inference service node; creating a replica set of the proxy service according to the node pressure; according to a request allocation strategy, distributing the session request corresponding to any inference service node to each replica in the replica set for processing, and performing a load balancing operation between the proxy service and each replica; when the pressure information corresponding to the replica set meets the pressure requirement, determining a first replica in the replica set, releasing the replicas in the replica set except the first replica, and replacing the proxy service with the first replica.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a load balancing method, device, electronic device, and storage medium for an inference service. Background Art

[0002] With the advancement of science and technology and the continuous evolution of artificial intelligence (AI), inference services have become one of the core forces driving the intelligent transformation of society. For example, resource usage information can be used to determine the forwarding path for user requests. However, in large-scale deployments, when a node is unable to provide a service, how to distribute user requests to the appropriate inference node based on the current operating status of the system, thereby reducing resource bottlenecks and service congestion, has become a key research topic. Summary of the Invention

[0003] The present disclosure provides a load balancing method, device, electronic device, and storage medium for an inference service, the main purpose of which is to solve the problem of poor inference efficiency when the node pressure of the inference service node is high.

[0004] According to a first aspect of the present disclosure, a load balancing method for an inference service is provided, comprising:

[0005] In the process of the inference service node set processing the session request, obtaining load status information of any inference service node in the inference service node set;

[0006] When the load status information indicates that the node pressure of any one of the inference service nodes is greater than a pressure threshold, creating a proxy service corresponding to the any one of the inference service nodes;

[0007] Creating a replica set of the proxy service according to the node pressure;

[0008] Distribute the session request corresponding to any inference service node to each replica in the replica set for processing according to the request distribution strategy, and perform load balancing operations between the proxy service and each replica;

[0009] When the pressure information corresponding to the replica set meets the pressure requirement, a first replica is determined in the replica set, replicas other than the first replica in the replica set are released, and the first replica is used to replace the proxy service.

[0010] According to a second aspect of the present disclosure, a load balancing device for an inference service is provided, comprising:

[0011] An information acquisition set, used to acquire load status information of any inference service node in the inference service node set during the process of the inference service node set processing a session request;

[0012] a service creating unit, configured to create a proxy service corresponding to any inference service node when the load status information indicates that the node pressure of any inference service node is greater than a pressure threshold;

[0013] A set creation unit, configured to create a replica set of the proxy service according to the node pressure;

[0014] a request distribution unit, configured to distribute the session request corresponding to any inference service node to each replica in the replica set for processing according to a request distribution strategy, and perform a load balancing operation between the proxy service and each replica;

[0015] The proxy release unit is used to determine the first replica in the replica set, release the replicas in the replica set except the first replica, and replace the proxy service with the first replica when the pressure information corresponding to the replica set meets the pressure requirement.

[0016] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.

[0020] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0021] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method as described in the first aspect above.

[0022] Through the present disclosure, in the process of processing session requests by a set of inference service nodes, the load status information of any inference service node in the set of inference service nodes is obtained; when the load status information indicates that the node pressure of any inference service node is greater than the pressure threshold, a proxy service corresponding to any inference service node is created; based on the node pressure, a replica set of the proxy service is created; based on the request allocation strategy, the session request corresponding to any inference service node is distributed to each replica in the replica set for processing, and a load balancing operation is performed between the proxy service and each replica; when the pressure information corresponding to the replica set meets the pressure requirement, the first replica is determined in the replica set, the replicas in the replica set except the first replica are released, and the proxy service is replaced by the first replica. Therefore, when a node is under high pressure, load balancing can be performed through proxy services to reduce the situation where the node cannot process session requests and cause service interruptions, reduce resource bottlenecks and service congestion, and perform load balancing operations among multiple replicas, which can improve the overall throughput and request processing efficiency. When the pressure is reduced, downgrade processing can be performed and only one replica can be retained, which can release unused resources and achieve elastic recovery and resource utilization efficiency, thereby improving the reasoning efficiency when the node is under high pressure while improving resource utilization efficiency.

[0023] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0025] Figure 1 A flow chart of a load balancing method for an inference service provided by an embodiment of the present disclosure;

[0026] Figure 2 A flow chart of another load balancing method for inference services provided by an embodiment of the present disclosure;

[0027] Figure 3 This is a schematic diagram illustrating an example of a load balancing method provided in an embodiment of the present disclosure;

[0028] Figure 4 This is a schematic diagram illustrating an example of a key-value cache synchronization method provided by an embodiment of the present disclosure;

[0029] Figure 5 This is a schematic diagram illustrating an example of a method for removing a proxy service provided in an embodiment of the present disclosure;

[0030] Figure 6 A schematic diagram of the structure of a load balancing device for inference services provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0032] According to some embodiments, pre-trained models, for example, possess powerful language understanding and generation capabilities. These models not only play an important role in a variety of fields, including education, healthcare, law, finance, and scientific research, but also profoundly change people's production and lifestyles. In education, pre-trained models can serve as personalized intelligent tutors, helping students plan their studies based on their individual levels. In healthcare, they can assist doctors in analyzing medical records and conducting preliminary diagnoses, improving diagnostic and treatment efficiency. In various professional tasks, the application of pre-trained models significantly lowers the threshold for acquiring complex knowledge, enabling more people to participate equally in the knowledge society and significantly promoting information equality and universal access to knowledge. However, as the scale of pre-trained model services continues to expand and application scenarios become increasingly complex, the underlying technical infrastructure faces significant challenges. Pre-trained model inference is not only computationally intensive but also places extremely high demands on real-time performance and availability. Therefore, load balancing, as the core scheduling mechanism connecting user requests and inference nodes, is becoming increasingly important. It not only affects service performance but is also a key factor in determining the stable operation and flexible scalability of the entire system.

[0033] In some embodiments, load balancing is not only a key scheduler for the efficient and stable operation of pre-trained model services, but also an indispensable cornerstone in their evolution towards large-scale, real-time, and intelligent operations. An efficient load balancing mechanism must comprehensively consider multiple dimensions, such as the model replica's graphics processing unit (GPU) utilization, central processing unit (CPU) utilization, memory usage, response latency, key-value cache (KV) cache hits, and network conditions, to ensure that each request is processed along the optimal path. In pre-trained model services deployed on multiple nodes and multiple replicas, the load balancer also performs fault tolerance, hot migration, and traffic control. When a node fails or becomes overloaded, it can promptly redirect traffic to healthy nodes, ensuring uninterrupted service and a sustained user experience. Therefore, load balancing has become a key focus.

[0034] The following describes a load balancing method, apparatus, electronic device, and storage medium for an inference service according to an embodiment of the present disclosure with reference to the accompanying drawings.

[0035] Figure 1 A flow chart of a load balancing method for an inference service provided in an embodiment of the present disclosure.

[0036] like Figure 1 As shown, the method comprises the following steps:

[0037] Step 101: in the process of the inference service node set processing a session request, obtaining load status information of any inference service node in the inference service node set;

[0038] According to some embodiments, the execution subject of the embodiments of the present disclosure may be, for example, an electronic device. The electronic device may be, for example, a server. The electronic device is not specifically a fixed device. For example, when the device identifier corresponding to the electronic device changes, the electronic device may also change accordingly.

[0039] In some embodiments, the inference service node set may be, for example, a collection of at least one inference service node. The inference service node may be used, for example, to process session requests. The inference service node set does not specifically refer to a fixed set. For example, when the number of services corresponding to the inference service node set changes, the inference service node set may also change accordingly. For example, when a certain inference service node in the inference service node set changes, the inference service node set may also change accordingly. The inference service node set may also be referred to as a backend service node set, for example.

[0040] According to some embodiments, a session request may, for example, refer to at least one session request, and the session requests processed by the set of inference service nodes may, for example, be a set of session requests. The session request does not specifically refer to a fixed request. For example, when the number of sessions corresponding to a session request changes, the session request may also change accordingly.

[0041] In some embodiments, load status information is used to indicate the load information borne by any service node. This load status information is not specifically fixed information. For example, when the service node changes, the load status information may also change accordingly. For example, when the node identifier of any service node changes, the load status information may also change accordingly.

[0042] In some embodiments, the load status information of any inference service node in the inference service node set may be obtained during the process of the inference service node set processing a session request.

[0043] Step 102: When the load status information indicates that the node pressure of any reasoning service node is greater than a pressure threshold, create a proxy service corresponding to any reasoning service node;

[0044] In some embodiments, node pressure can be, for example, the pressure of any service node. This node pressure is not specifically a fixed pressure. For example, if the number of session requests processed by any service node changes, the node pressure may also change accordingly. For example, if the resources required by any service node to process a session request change, the node pressure may also change accordingly.

[0045] In some embodiments, a stress threshold may be used, for example, to determine whether to create a proxy service for any inference service node. This stress threshold is not a fixed threshold. For example, upon receiving a modification instruction for the stress threshold, the stress threshold may change accordingly. For example, upon changes in business demand information, the stress threshold may also change accordingly.

[0046] According to some embodiments, a proxy service may be, for example, a proxy service corresponding to any inference service node. The proxy service, for example, does not directly participate in business computations but is only responsible for forwarding requests. Different inference service nodes may correspond to different proxy services.

[0047] According to some embodiments, when the load status information indicates that the node pressure of any inference service node is greater than a pressure threshold, a proxy service corresponding to the inference service node is created. That is, a proxy service corresponding to the inference service node whose node pressure is greater than the pressure threshold can be created.

[0048] Step 103: Create a replica set of the proxy service based on node pressure;

[0049] In some embodiments, a replica set can be, for example, a collection of at least one replica. This replica set can include, for example, multiple replicas corresponding to a proxy service. This replica set is not a fixed set. For example, as node pressure changes, the number of replicas in the replica set can also change accordingly.

[0050] In some embodiments, a replica set of the proxy service may be created based on node pressure.

[0051] Step 104: Distribute the session request corresponding to any inference service node to each replica in the replica set for processing according to the request distribution strategy, and perform load balancing between the proxy service and each replica.

[0052] According to some embodiments, a request allocation strategy may, for example, be a strategy for allocating session requests corresponding to any inference service node to a replica set. This request allocation strategy does not specifically refer to a fixed strategy. For example, when the traffic of any inference service node changes, the request allocation strategy may also change accordingly. For example, when the replica set changes, the request allocation strategy may also change accordingly. The session request corresponding to any inference service node may, for example, be at least one.

[0053] In some embodiments, performing load balancing operations between the proxy service and each replica can be used to achieve balanced traffic distribution, for example, to reduce the situation where traffic imbalance between the replicas leads to low request processing efficiency.

[0054] In some embodiments, a session request corresponding to any inference service node is distributed to each replica in a replica set according to a request distribution strategy, and a load balancing operation is performed between the proxy service and each replica.

[0055] Step 105 : When the pressure information corresponding to the replica set meets the pressure requirement, determine the first replica in the replica set, release the replicas in the replica set except the first replica, and use the first replica to replace the proxy service.

[0056] In some embodiments, the pressure requirement may be, for example, a requirement for determining whether to downgrade a replica set. The pressure requirement is not a fixed requirement. For example, the pressure requirement may be that the pressure information is less than a certain pressure threshold. For example, when the pressure threshold changes, the pressure requirement may also change accordingly.

[0057] In some embodiments, the first replica can be, for example, the remaining replica in the replica set, and can be a replacement proxy service, serving as a replica of any inference service node. The "first" in the first replica is used to distinguish it from the remaining replicas and does not specifically refer to a fixed replica. For example, other replicas in the replica set can be released.

[0058] According to some embodiments, the first replica is used to replace the proxy service. For example, the proxy service can be offline, and the first replica becomes the master replica node to implement a single point service.

[0059] In some embodiments, when the pressure information corresponding to the replica set meets the pressure requirement, the replica set is downgraded, the first replica is retained, and the proxy service is replaced by the first replica.

[0060] Through the present disclosure, in the process of processing session requests by the inference service node set, the load status information of any inference service node in the inference service node set is obtained; when the load status information indicates that the node pressure of any inference service node is greater than the pressure threshold, a proxy service corresponding to any inference service node is created; based on the node pressure, a replica set of the proxy service is created; based on the request allocation strategy, the session request corresponding to any inference service node is distributed to each replica in the replica set for processing, and a load balancing operation is performed between the proxy service and each replica; when the pressure information corresponding to the replica set meets the pressure requirements, the first replica is determined in the replica set, the replicas in the replica set except the first replica are released, and the first replica is used to replace the proxy service. Therefore, when a node is under high pressure, load balancing can be performed through proxy services to reduce the situation where the node cannot process session requests and cause service interruptions, reduce resource bottlenecks and service congestion, and perform load balancing operations among multiple replicas, which can improve the overall throughput and request processing efficiency. When the pressure is reduced, downgrade processing can be performed and only one replica can be retained, which can release unused resources and achieve elastic recovery and resource utilization efficiency, thereby improving the reasoning efficiency when the node is under high pressure while improving resource utilization efficiency.

[0061] It should be noted that the embodiments of the present disclosure may include multiple steps. For the convenience of description, these steps are numbered, but these numbers do not limit the execution time slots or execution order between the steps; these steps can be implemented in any order, and the embodiments of the present disclosure do not limit this.

[0062] Furthermore, in a possible implementation of this embodiment, Figure 2 This is a flow chart of another load balancing method for inference services provided by an embodiment of the present disclosure. Figure 2 As shown, the method comprises the following steps:

[0063] Step 201: in the process of the inference service node set processing a session request, obtain the load status information of any inference service node in the inference service node set;

[0064] The relevant description may be as described above, and will not be repeated here.

[0065] In some embodiments, the disclosed embodiments can be used in large-scale, dynamically load-balanced pre-trained model inference service scenarios. A pre-trained model can, for example, be a model trained on a large dataset through unsupervised or self-supervised learning. This pre-trained model is not specific to a fixed model. When the specific type of the pre-trained model changes, the pre-trained model can also change accordingly.

[0066] Step 202: When the load status information indicates that the node pressure of any reasoning service node is greater than a pressure threshold, create a proxy service corresponding to any reasoning service node;

[0067] The relevant description may be as described above, and will not be repeated here.

[0068] According to some embodiments, the high seas method further comprises:

[0069] When the inference service node set is started and all inference service nodes in the inference service node set pass the consistent hash load balancing check, the same session request in the session request is routed to the same inference service node. This can improve the KV cache utilization, increase the inference speed, and improve the inference efficiency.

[0070] According to some embodiments, obtaining load status information of any inference service node in the inference service node set includes:

[0071] Using a monitoring service corresponding to each inference service node in the inference service node set to collect data from each inference service node, and obtaining indicator information corresponding to each inference service node, wherein the indicator information includes at least one of a central processing unit usage rate, memory information, a graphics processing unit usage rate, and a number of active sessions;

[0072] The indicator information is aggregated and analyzed to obtain the load status information corresponding to each inference service node. Therefore, load status information can be determined based on the indicator information, improving the accuracy of load status information acquisition and the accuracy of proxy service creation, reducing the situation where excessive node pressure leads to slow inference speed, and improving inference efficiency. The usage rate can also be referred to as the utilization rate, which is not limited in the present embodiment.

[0073] In some embodiments, for example, a monitoring service may be deployed on each inference service node, where the monitoring service is not limited. Furthermore, the data collected by the monitoring service is not limited. For example, at least one of CPU usage, memory information, GPU usage, and the number of active sessions may be collected. This is not limited in the presently disclosed embodiments. The collected indicator information may, for example, be matched with an alarm rule.

[0074] In some embodiments, for example, data collected by the monitoring service may be obtained every preset time period, such as 5 seconds. Specifically, the data may be obtained by a central monitoring module in the electronic device.

[0075] The indicator information can be aggregated and analyzed to obtain the load status information corresponding to each inference service node. For example, the indicator information can also be visualized and displayed.

[0076] Among them, according to the load status information, it is determined whether the node pressure of any inference service node is greater than the pressure threshold. A specific example may be that each piece of information in the load status information is greater than the corresponding threshold. Among them, the judgment rule or the alarm rule can be determined based on the rule setting instruction, and the embodiment of the present disclosure is not limited to this. Among them, the alarm information can be one of the indicator information or multiple. For example, when the CPU utilization rate is greater than 80%, the GPU utilization rate is greater than 70%, and the number of active sessions is greater than 100, it is determined that the node pressure of any inference service node is greater than the pressure threshold.

[0077] According to some embodiments, the method further comprises:

[0078] Get the business requirement information corresponding to the session request;

[0079] Based on the business demand information, pressure thresholds and request allocation strategies are obtained. This improves the matching between pressure thresholds and business demand information, improves the matching between request allocation strategies and business demand information, improves the accuracy of pressure threshold acquisition, and improves the accuracy of proxy service creation.

[0080] According to some embodiments, the method further comprises:

[0081] Get the business requirement information corresponding to the session request;

[0082] According to the business demand information, the pressure threshold is obtained. Therefore, the matching between the pressure threshold and the business demand information can be improved, the accuracy of obtaining the pressure threshold is improved, and the accuracy of creating the proxy service is improved.

[0083] In some embodiments, for example, when the resource usage of any inference service node exceeds a threshold or the number of active sessions exceeds a threshold, a proxy service corresponding to any inference service node can be created, which can trigger the capacity expansion process.

[0084] For example, when it is determined that an alarm message is received, an overloaded node may be determined, and a proxy service corresponding to the overloaded node may be created. The overloaded node is a node whose node pressure is greater than a pressure threshold.

[0085] For example, a proxy service can be deployed in front of any inference service node using automated scripts or container orchestration services. The proxy service can run as a standalone container or process and has service registration and discovery capabilities. Once started, the proxy service registers itself with the service discovery server and marks the original node as a replica.

[0086] Step 203: Create a replica set of the proxy service based on node pressure;

[0087] The relevant description may be as described above, and will not be repeated here.

[0088] According to some embodiments, after creating a replica set of a proxy service, the replica set may be checked at preset intervals to determine whether the running status of each replica meets status requirements, thereby improving service availability.

[0089] In some embodiments, after creating a proxy service, the load balancer or Domain Name System (DNS) configuration can be updated to switch all session requests directed to any inference service node to the proxy service. For example, a "hot switch" or "lossless migration" mechanism can be employed to reduce session interruptions and improve session request processing continuity. The load balancer type is not limited.

[0090] According to some embodiments, obtaining a replica set of a proxy service based on node pressure includes:

[0091] According to the node pressure and load information, obtain the number of replicas corresponding to the proxy service;

[0092] Create a replica set for the proxy service based on the number of replicas, and add any inference service node as a replica to the replica set.

[0093] Load information can include, for example, at least one of request rate, average response time, and replica utilization. Therefore, the number of replicas can be determined based on load information and node pressure, reducing resource waste caused by excessive replica creation. Specifically, a replica set can be created through a container orchestration server. Each replica in the replica set is registered with a proxy service and confirmed available through a status check. For example, the GPU and CPU resources corresponding to each replica can be determined.

[0094] According to some embodiments, the method further comprises:

[0095] Get the business requirement information corresponding to the session request;

[0096] Based on the business demand information, a request allocation strategy is obtained. The request allocation strategy can be executed before step 203. Therefore, the strategy can be switched dynamically to adapt to different business load scenarios, improve the accuracy of request allocation strategy acquisition, improve the matching of the request allocation strategy with the traffic of any inference service node, and improve request processing efficiency.

[0097] According to some embodiments, the external service port of any inference service node is taken over by the proxy service, and any inference service node serves as a copy of the proxy service.

[0098] There are multiple request allocation strategies, such as round-robin, minimum number of connections, minimum load, etc., which can achieve uniform traffic distribution.

[0099] Step 204: Distribute the session request corresponding to any inference service node to each replica in the replica set for processing according to the request distribution strategy, and perform load balancing between the proxy service and each replica.

[0100] The relevant description may be as described above, and will not be repeated here.

[0101] According to some embodiments, distributing a session request of any inference service node to each replica in a replica set according to a request distribution strategy includes:

[0102] Switch the session request of any inference service node to the proxy service;

[0103] The control proxy service distributes the session request to each replica in the replica set according to at least one of the request distribution policy and the session identifier.

[0104] For example, Figure 3 This is an example diagram of a load balancing method provided by an embodiment of the present disclosure. Figure 3As shown, the Proxy service can uniformly receive and forward all session requests, and distribute them to the replica set according to the session identifier ID or load balancing strategy, which can improve session stickiness and reduce the situation where the same session is interleaved by different replicas. Among them, the inference service node can include, for example, pre-trained model service A, pre-trained model service B and pre-trained model service C, wherein the node pressure of pre-trained model service B is greater than the pressure threshold, and a proxy service corresponding to pre-trained model service B can be created. Traffic distribution can be performed through polling or minimum connection number load balancing. Among them, replica B1, replica B2 and replica B3 are the replica sets of pre-trained model service B. Key-value cache synchronization can be performed between replicas B1, replica B2 and replica B3. Among them, Client is the client and User is the user.

[0105] According to some embodiments, the method further comprises:

[0106] Obtain at least one uncompleted session request during the process of switching traffic from any inference service node to the proxy service;

[0107] If traffic from any inference service node is confirmed to have been switched to the proxy service, at least one session request is distributed to each replica. This allows unfinished requests in the cache or queue to continue processing after the switch is complete, making user requests invisible and improving the user experience.

[0108] According to some embodiments, each replica may be tested once every preset time period, and if the test fails, the replica may be removed and retested to improve the high availability of each replica.

[0109] Step 205: After each replica completes processing the target session request and there is a change in the key-value cache, the key-value caches between the replicas are synchronized through the serialization and broadcast mechanism;

[0110] According to some embodiments, a key-value cache can be used to cache keys and values ​​during autoregressive inference, reducing repeated computations. Changes to the key-value cache can be caused by, for example, adding a new token or expanding the execution context. The disclosed embodiments do not limit how the key-value cache is modified.

[0111] In some embodiments, serialization can be, for example, serializing a changed key-value cache. Broadcasting can be, for example, broadcasting a changed key-value cache across replicas. The broadcasting method is not limited. The broadcasting method can include, for example, a full broadcast method and an incremental broadcast method.

[0112] In some embodiments, after each replica completes processing the target session request and there is a change in the key-value cache, the key-value cache between the replicas is synchronized through a serialization and broadcast mechanism.

[0113] According to some embodiments, after each replica completes processing a target session request and there is a change in the key-value cache, the key-value caches between the replicas are synchronized through a serialization and broadcast mechanism, including:

[0114] After each replica completes processing the target session request and there are changes in the key-value cache, the serialized key-value cache is uploaded to the centralized storage system;

[0115] Send synchronization notification information to the replica subset via network broadcast, where the synchronization notification information includes information about changes in the key-value cache;

[0116] When it is determined that each replica has received the synchronization notification information, the key-value cache closest to the current time is pulled from the centralized storage system according to the session identifier and key-value cache version number in the synchronization notification information and deserialized to obtain a deserialized key-value cache;

[0117] If the deserialized key-value cache passes the verification, the deserialized key-value cache is stored and the stored key-value cache is deleted. Therefore, the KV cache can be synchronized between replicas based on a high-speed network, improving the synchronization efficiency of the key-value cache and the efficiency of request processing.

[0118] According to some embodiments, the KV cache can be, for example, a tensor or dictionary (dict) structured data, which can be serialized into binary data. This data can be serialized into a memory object or written directly to a disk file, which can improve the convenience of subsequent KV cache storage and transmission. A dictionary structure is a data structure that stores data in the form of key-value pairs.

[0119] In some embodiments, the centralized storage system may be, for example, at least one of a remote dictionary server (RemoteDictionary Server, Redis), a distributed key-value pair (Key-Value, KV), and an object storage.

[0120] According to some embodiments, uploading the serialized key-value cache to a centralized storage system includes:

[0121] Use the serialization interface of the inference framework to serialize the changed key-value cache and obtain the key value corresponding to the serialized key-value cache;

[0122] According to the key value, the serialized key value cache is uploaded to the centralized storage system. This can improve the accuracy of the serialized key value cache storage, improve the efficiency of subsequent retrieval, and improve the convenience of subsequent use.

[0123] According to some embodiments, sending synchronization notification information to all replicas via network broadcasting includes:

[0124] Synchronization notification information is sent to all replicas through a message queue or a distributed bus, wherein the synchronization notification information includes at least one of a session identifier, a key-value cache version number, a change timestamp information, and a central storage location information, and the central storage location information is used to indicate the storage location of the serialized key-value cache in the centralized storage system.

[0125] In some embodiments, for example, each replica can be controlled to subscribe to a KV cache change topic, employing a publish-subscribe model. This reduces synchronization time when KV cache changes occur. Each replica process continuously listens to a message queue / event bus for KV cache change events. Upon receiving a KV cache change notification related to the local session, it immediately initiates synchronization.

[0126] According to some embodiments, when pulling the latest KV cache binary data from the central storage, for example, idempotent pulling and breakpoint resuming can be supported, which can reduce the probability of repeated or partial pulling of the KV cache.

[0127] In some embodiments, the key-value cache closest to the current time is pulled from the centralized storage system and deserialized to obtain a deserialized key-value cache. For example, the key-value cache closest to the current time can be pulled from the centralized storage system and deserialized into a local Tensor or dict structure using a framework interface to obtain a deserialized key-value cache.

[0128] According to some embodiments, the deserialized key-value cache verification can be, for example, verifying the metadata of the kv cache, such as session_id, token number, version number, etc., which can improve the accuracy of data acquisition so that the acquired data is the most recent to the current time and has not been concurrently overwritten.

[0129] In some embodiments, if a higher version of the kv cache is already available locally, the current synchronization data is discarded. Alternatively, if the data is abnormal or the verification fails, a log is recorded and the next synchronization is awaited.

[0130] According to some embodiments, if the deserialized key-value cache passes verification, the deserialized key-value cache can be stored and the stored key-value cache can be deleted, ensuring a consistent context the next time the session is processed. For example, the deserialized key-value cache can also be atomically replaced to reduce data inconsistencies caused by concurrent reads and writes, thereby improving data storage accuracy.

[0131] In some embodiments, when storing a deserialized key-value cache, for example, the changed portion can be synchronized, which can reduce bandwidth and latency, improve the accuracy of data storage, and improve data storage efficiency. The synchronized changed portion can, for example, add a new token. In the event that the synchronized changed portion fails, the attempt can be retried or downgraded to a full synchronization. The failure to synchronize the changed portion can, for example, include a pull timeout or a deserialization failure. For example, the synchronization status and failure log can also be recorded to facilitate subsequent monitoring and alerting.

[0132] According to some embodiments, sending synchronization notification information to a subset of replicas via network broadcasting includes:

[0133] Obtaining a network broadcast mode corresponding to the business demand information, wherein the network broadcast mode includes one of a full broadcast mode and an incremental broadcast mode;

[0134] Send synchronization notification information to the replica subset corresponding to the network broadcast method. Improve the accuracy of synchronization notification information sending and reduce resource waste.

[0135] A full broadcast method, for example, notifies all replicas, while an incremental broadcast method, for example, notifies relevant replicas within a replica set, such as departmental replicas. Relevant replicas, for example, can be replicas corresponding to a single session. Network broadcasting supports message retry and deduplication, improving availability and consistency.

[0136] According to some embodiments, when the network broadcast mode is an incremental broadcast mode, the method further includes:

[0137] At preset intervals, a full check is performed on the key-value cache of each replica set to obtain the key-value cache check result. The key-value cache check result indicates the consistency between the key-value caches of each replica set. Therefore, regular full check can prevent data drift caused by long-term incremental synchronization and improve the consistency of key-value cache storage.

[0138] In some embodiments, for example, multi-copy redundancy processing and master-slave switching processing can be performed to reduce the impact on the overall KV cache consistency in the event of a single point failure.

[0139] In some embodiments, Figure 4This is an example diagram of a key-value cache synchronization method provided by an embodiment of the present disclosure. Figure 4 As shown, for example, it can include replica A processing the completed session request, determining whether the key-value cache has changed, and if there is no change, waiting for the next request. In the case of a change, the key-value cache can be serialized, stored in the central storage system, and the change notification can be broadcast. When other replicas hear the notification, the other replicas can pull the latest key-value cache, that is, the key-value cache closest to the current time, deserialize the key-value cache, and perform a consistency check. When the consistency check passes, the local key-value cache can be replaced. In this process, for example, incremental synchronization and / or failure retry operations can be performed. In the case of a consistency check failure, the key-value cache can be discarded.

[0140] Step 206: If the pressure information corresponding to the replica set meets the pressure requirement and lasts for a preset period of time, select the first replica in the replica set as the primary replica;

[0141] The relevant description may be as described above, and will not be repeated here.

[0142] According to some embodiments, selecting a first replica in a set of replicas as a primary replica includes:

[0143] At least one of priority weight and round-robin is used to select the first replica in the replica set as the primary replica.

[0144] According to some embodiments, when distributing traffic from any inference service node to each replica in a replica set, metrics such as CPU memory, GPU usage, number of active sessions, and request rate for each replica can be continuously collected, for example, through a monitoring system integrated with or connected to a proxy server. This can be monitored, for example, using a monitoring solution in some embodiments or a pre-configured lightweight monitoring module.

[0145] In some embodiments, pressure requirements can be determined based on business demand information. For example, these pressure requirements may include a CPU memory usage of less than 30% and a session count of less than 10. The preset duration, for example, may be 5 minutes, to reduce the likelihood of frequent scaling caused by occasional fluctuations and improve the accuracy of scaling control.

[0146] According to some embodiments, Figure 5 This is an example diagram of a proxy service removal method provided by an embodiment of the present disclosure. Figure 5As shown, if the pressure information corresponding to the replica set meets the pressure requirements and persists for a preset period of time, the scale-down process can be triggered. The triggering event and related monitoring data can be recorded to facilitate subsequent tracing and parameter optimization. In particular, if it is determined that the pressure corresponding to the replica set is less than the pressure threshold, the proxy service can be removed.

[0147] In some embodiments, a replica that meets operational requirements and has a low load can be selected as the primary service node based on policies such as inspection information, current load information, and historical stability of each replica. For example, the selection can be performed based on priority weight or round-robin.

[0148] Step 207: Synchronize the key-value cache of all session requests to the first copy to obtain the second copy;

[0149] The relevant description may be as described above, and will not be repeated here.

[0150] In some embodiments, specifically, the kv caches of all sessions can be merged / synchronized to the master copy through central storage or message queues, wherein the kv caches of the sessions of the master copy can be detected by incremental synchronization detection or conflict detection to improve the consistency of data acquisition of the second copy, i.e., the master copy.

[0151] In some embodiments, data detection may be performed on the second copy to determine whether the kvcache of the second copy includes key-value caches for all session requests, thereby reducing the probability of data loss and improving the integrity of the services provided.

[0152] Step 208: If the second replica passes the verification, the second replica is used as the primary service node, and at least one third replica in the replica set is released according to the replica release order, where the at least one third replica is a replica in the replica set other than the first replica.

[0153] The relevant description may be as described above, and will not be repeated here.

[0154] According to some embodiments, releasing at least one third replica in the set of replicas includes:

[0155] Release at least one third replica in the replica set in descending order of load information of the at least one third replica;

[0156] At least one tertiary replica is released in descending order of priority of the non-primary replicas. This frees up the resources occupied by the at least one tertiary replica, improving resource utilization.

[0157] According to some embodiments, before releasing the at least one third replica, for example, it is possible to confirm whether the request corresponding to the at least one third replica is completed, and uncompleted requests can be transferred to other nodes for processing. After releasing the at least one third replica, the available resource pool of the resource management system can be monitored.

[0158] Step 209: Adjust the routing of the proxy service and remove the proxy service.

[0159] The relevant description may be as described above, and will not be repeated here.

[0160] In some embodiments, when it is determined that at least one third replica has been released and the running status of the second replica meets the status requirement, the removal of the proxy service can be triggered.

[0161] According to some embodiments, adjusting the routing of the proxy service includes at least one of the following:

[0162] The control proxy service retains the routing of the first replica and removes the forwarding configuration of at least one third replica;

[0163] Configure the ingress load balancing to point to the first replica.

[0164] According to some embodiments, for example, the ingress load balancing configuration can be switched from a proxy to a direct connection to the primary replica, enabling seamless traffic migration. This process can also support hot switching, minimizing session interruptions.

[0165] In some embodiments, after the proxy service is released, the resources occupied by the proxy service can be released and the registration in the discovery system can be updated, which can improve the convenience of external requests to access the master copy.

[0166] In some embodiments, the electronic device may, for example, continuously monitor the pressure of each inference service node, and re-trigger the expansion process when the pressure is greater than a pressure threshold, thereby performing elastic scaling closed-loop control and improving resource utilization efficiency.

[0167] In some embodiments, creating a proxy service corresponding to any inference service node can decouple business traffic from nodes with greater pressure, thereby improving the flexibility of capacity expansion and traffic migration. Secondly, KV cache synchronization can be performed to improve the integrity of the context corresponding to the master copy and improve data consistency. Then, when the pressure of the replica set is less than the pressure threshold, degradation processing can be performed to reduce resource usage and improve resource utilization efficiency. It can achieve an organic combination of high availability, high performance, dynamic scaling and context consistency, and improve inference efficiency.

[0168] According to an embodiment of the present disclosure, the present disclosure also provides a load balancing device for an inference service.

[0169] For example, Figure 6 This is a structural diagram of a load balancing device for an inference service provided by an embodiment of the present disclosure. The load balancing device 600 for an inference service includes: an information acquisition set 601, a service creation unit 602, a set creation unit 603, a request distribution unit 604, and an agent release unit 605; wherein,

[0170] Information acquisition set 601 is used to obtain load status information of any inference service node in the inference service node set during the process of the inference service node set processing a session request;

[0171] A service creation unit 602 is configured to create a proxy service corresponding to any reasoning service node when the load status information indicates that the node pressure of any reasoning service node is greater than a pressure threshold;

[0172] A set creation unit 603 is used to create a replica set of the proxy service according to the node pressure;

[0173] The request distribution unit 604 is used to distribute the session request corresponding to any inference service node to each replica in the replica set for processing according to the request distribution strategy, and perform load balancing operations between the proxy service and each replica;

[0174] The proxy releasing unit 605 is configured to determine the first replica in the replica set, release the replicas except the first replica in the replica set, and replace the proxy service with the first replica when the pressure information corresponding to the replica set meets the pressure requirement.

[0175] Furthermore, the information acquisition set 601 is used to obtain the load status information of any inference service node in the inference service node set, specifically for:

[0176] Using a monitoring service corresponding to each inference service node in the inference service node set to collect data from each inference service node, and obtaining indicator information corresponding to each inference service node, wherein the indicator information includes at least one of a central processing unit usage rate, memory information, a graphics processing unit usage rate, and a number of active sessions;

[0177] Aggregate and analyze the indicator information to obtain the load status information corresponding to each inference service node.

[0178] Furthermore, the information acquisition set 601 is also used to:

[0179] Get the business requirement information corresponding to the session request;

[0180] Obtain pressure thresholds and request allocation strategies based on business demand information.

[0181] Furthermore, the set creation unit 603 is used to create a replica set of the proxy service according to the node pressure, specifically to:

[0182] According to the node pressure and load information, obtain the number of replicas corresponding to the proxy service;

[0183] Create a replica set for the proxy service based on the number of replicas, and add any inference service node as a replica to the replica set.

[0184] Furthermore, the request distribution unit 604 is configured to distribute the session request of any inference service node to each replica in the replica set according to the request distribution strategy, specifically to:

[0185] Switch the session request of any inference service node to the proxy service;

[0186] The control proxy service distributes the session request to each replica in the replica set according to at least one of the request distribution policy and the session identifier.

[0187] Furthermore, the request distribution unit 604, the method is further specifically configured to:

[0188] Obtain at least one uncompleted session request of any inference service node during the process of switching the session request to the proxy service;

[0189] When it is confirmed that the session request of any inference service node is switched to the proxy service, at least one session request is distributed to each replica.

[0190] Furthermore, the proxy release unit 605 is configured to, when the pressure information corresponding to the replica set meets the pressure requirement, determine the first replica in the replica set, release the replicas other than the first replica in the replica set, and use the first replica to replace the proxy service, specifically for:

[0191] If the pressure information corresponding to the replica set meets the pressure requirements and lasts for a preset period of time, the first replica in the replica set is selected as the primary replica;

[0192] Synchronize the key-value cache of all session requests to the first copy to obtain the second copy;

[0193] If the second replica passes the verification, the second replica is used as the primary service node, and at least one third replica in the replica set is released in the replica release order, where the at least one third replica is a replica in the replica set other than the first replica.

[0194] Adjust the routing of the proxy service and remove the proxy service.

[0195] Furthermore, the proxy release unit 605 is configured to select the first replica in the replica set as the primary replica, specifically to:

[0196] At least one of priority weight and round-robin is used to select the first replica in the replica set as the primary replica.

[0197] Furthermore, the proxy releasing unit 605 is configured to adjust the routing of the proxy service by performing at least one of the following operations:

[0198] The control proxy service retains the routing of the first replica and removes the forwarding configuration of at least one third replica;

[0199] Configure the ingress load balancing to point to the first replica.

[0200] Furthermore, the proxy releasing unit 605 is configured to release at least one third replica in the replica set, specifically for at least one of the following:

[0201] Release at least one third replica in the replica set in descending order of load information of the at least one third replica;

[0202] Release at least one tertiary replica in descending order of the offline priority of non-primary replicas.

[0203] Furthermore, the proxy release unit 605 is further configured to:

[0204] After each replica completes processing the target session request and there are changes in the key-value cache, the key-value cache between the replicas is synchronized through the serialization and broadcast mechanism.

[0205] Furthermore, the proxy release unit 605 is configured to synchronize the key-value caches between the replicas through a serialization and broadcast mechanism after each replica completes processing the target session request and there is a change in the key-value cache. Specifically, it is configured to:

[0206] After each replica completes processing the target session request and there are changes in the key-value cache, the serialized key-value cache is uploaded to the centralized storage system;

[0207] Send synchronization notification information to the replica subset via network broadcast, where the synchronization notification information includes information about changes in the key-value cache;

[0208] When it is determined that each replica has received the synchronization notification information, the key-value cache closest to the current time is pulled from the centralized storage system according to the session identifier and key-value cache version number in the synchronization notification information and deserialized to obtain a deserialized key-value cache;

[0209] When the deserialized key-value cache passes the verification, the deserialized key-value cache is stored and the stored key-value cache is deleted.

[0210] Furthermore, the proxy release unit 605 is used to upload the serialized key-value cache to the centralized storage system, specifically to:

[0211] Use the serialization interface of the inference framework to serialize the changed key-value cache and obtain the key value corresponding to the serialized key-value cache;

[0212] According to the key value, the serialized key value cache is uploaded to the centralized storage system.

[0213] Furthermore, the proxy release unit 605 is configured to send synchronization notification information to all replicas via network broadcast, specifically to:

[0214] Synchronization notification information is sent to all replicas through a message queue or a distributed bus, wherein the synchronization notification information includes at least one of a session identifier, a key-value cache version number, a change timestamp information, and a central storage location information, and the central storage location information is used to indicate the storage location of the serialized key-value cache in the centralized storage system.

[0215] Furthermore, the proxy release unit 605 is configured to send synchronization notification information to the replica subset via network broadcast, specifically to:

[0216] Obtaining a network broadcast mode corresponding to the business demand information, wherein the network broadcast mode includes one of a full broadcast mode and an incremental broadcast mode;

[0217] Send synchronization notification information to the subset of replicas corresponding to the network broadcast method.

[0218] Furthermore, when the network broadcast mode is an incremental broadcast mode, the proxy release unit 605 is used in the method to further specifically:

[0219] At preset intervals, a full check is performed on the key-value cache of each replica set to obtain a key-value cache check result, wherein the key-value cache check result is used to indicate the consistency between the key-value caches of each replica set.

[0220] It should be noted that, for the description of the features in the embodiment corresponding to the load balancing device of the inference service, reference can be made to the relevant description of the embodiment corresponding to the load balancing method of the inference service, which will not be repeated here.

[0221] An embodiment of the present disclosure further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned load balancing method embodiments for inference services.

[0222] An embodiment of the present disclosure further provides a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps of any of the above-mentioned load balancing method embodiments for inference services when running.

[0223] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0224] An embodiment of the present disclosure further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned load balancing method embodiments for inference services are implemented.

[0225] An embodiment of the present disclosure also provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps in any of the above-mentioned load balancing method embodiments of the inference service.

[0226] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0227] The above is a detailed introduction to a load balancing method for an inference service provided by the present disclosure. This article uses specific examples to illustrate the principles and implementation methods of the present disclosure. The description of the above embodiments is only used to help understand the method and core ideas of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present disclosure, several improvements and modifications can be made to the present disclosure, and these improvements and modifications also fall within the scope of protection of the claims of the present disclosure.

Claims

1. A load balancing method for an inference service, characterized in that: include: In the process of the inference service node set processing the session request, obtaining load status information of any inference service node in the inference service node set; When the load status information indicates that the node pressure of any one of the inference service nodes is greater than a pressure threshold, creating a proxy service corresponding to the any one of the inference service nodes; Creating a replica set of the proxy service according to the node pressure; Distribute the session request corresponding to any inference service node to each replica in the replica set for processing according to the request distribution strategy, and perform load balancing operations between the proxy service and each replica; If the pressure information corresponding to the replica set meets the pressure requirement, determine a first replica in the replica set, release replicas in the replica set except the first replica, and replace the proxy service with the first replica; The method further comprises: After each copy completes processing the target session request and there is a change in the key-value cache, uploading the serialized key-value cache to the centralized storage system; Sending synchronization notification information to the replica subset via network broadcast, wherein the synchronization notification information includes relevant information about changes in the key-value cache; When it is determined that each replica has received the synchronization notification information, according to the session identifier and key-value cache version number in the synchronization notification information, pull the key-value cache closest to the current time from the centralized storage system and deserialize it to obtain a deserialized key-value cache; When the deserialized key-value cache passes verification, the deserialized key-value cache is stored and the stored key-value cache is deleted.

2. The method according to claim 1, characterized in that The acquiring of load status information of any inference service node in the inference service node set includes: Using a monitoring service corresponding to each inference service node in the inference service node set to collect data from each inference service node, and obtaining indicator information corresponding to each inference service node, wherein the indicator information includes at least one of a central processing unit usage rate, memory information, a graphics processing unit usage rate, and a number of active sessions; Aggregate and analyze the indicator information to obtain load status information corresponding to each inference service node.

3. The method according to claim 1, characterized in that The method further comprises: Obtaining business requirement information corresponding to the session request; The pressure threshold and the request allocation strategy are acquired according to the business demand information.

4. The method according to claim 1, wherein The creating a replica set of the proxy service according to the node pressure includes: Obtaining the number of replicas corresponding to the proxy service based on the node pressure and load information; A replica set of the proxy service is created according to the number of replicas, and any one of the inference service nodes is added to the replica set as a replica.

5. The method according to claim 1, wherein The distributing the session request of any one of the inference service nodes to each replica in the replica set according to the request distribution strategy includes: Switching the session request of any one of the inference service nodes to the proxy service; According to at least one of the request distribution policy and the session identifier, the proxy service is controlled to distribute the session request to each replica in the replica set.

6. The method according to claim 1, characterized in that The method further comprises: Acquire at least one uncompleted session request during a process of switching a session request of any inference service node to the proxy service; When it is confirmed that the session request of any one of the inference service nodes is switched to the proxy service, the at least one session request is distributed to the replicas.

7. The method according to claim 1, characterized in that When the pressure information corresponding to the replica set meets the pressure requirement, determining a first replica in the replica set, releasing replicas other than the first replica in the replica set, and replacing the proxy service with the first replica includes: If the pressure information corresponding to the replica set meets the pressure requirement and lasts for a preset period of time, select the first replica in the replica set as the primary replica; Synchronize the key-value cache of all session requests to the first copy to obtain the second copy; If the second replica passes the verification, use the second replica as the primary service node and release at least one third replica in the replica set according to the replica release order, where the at least one third replica is the remaining replicas in the replica set except the first replica; The routing of the proxy service is adjusted, and the proxy service is removed.

8. The method according to claim 7, characterized in that The selecting the first replica in the replica set as the primary replica includes: At least one selection method of priority weight and round-robin is adopted to select the first replica in the replica set as the primary replica.

9. The method according to claim 7, characterized in that The adjusting of the routing of the proxy service includes at least one of the following: Controlling the proxy service to retain the route of the first replica and remove the forwarding configuration of the at least one third replica; Configure the ingress load balancing to point to the first replica.

10. The method according to claim 7, characterized in that The releasing of at least one third replica in the replica set includes at least one of the following: Releasing the at least one third replica in the replica set in descending order of load information of the at least one third replica; Release the at least one third copy in descending order of the offline priority of the non-primary copies.

11. The method according to claim 1, characterized in that The step of uploading the serialized key-value cache to the centralized storage system includes: Use the serialization interface of the inference framework to serialize the changed key-value cache and obtain the key value corresponding to the serialized key-value cache; According to the key value, the serialized key value cache is uploaded to the centralized storage system.

12. The method according to claim 1, characterized in that Sending synchronization notification information to all replicas via network broadcasting includes: Synchronization notification information is sent to all replicas through a message queue or a distributed bus, wherein the synchronization notification information includes at least one of a session identifier, a key-value cache version number, a change timestamp information, and a central storage location information, and the central storage location information is used to indicate the storage location of the serialized key-value cache in the centralized storage system.

13. The method according to claim 1, wherein The sending of synchronization notification information to the replica subset by network broadcasting includes: Obtaining a network broadcast mode corresponding to the service demand information, wherein the network broadcast mode includes one of a full broadcast mode and an incremental broadcast mode; Send synchronization notification information to the replica subset corresponding to the network broadcast mode.

14. The method according to claim 13, characterized in that in, When the network broadcast mode is an incremental broadcast mode, the method further includes: At preset time intervals, a full check is performed on the key-value cache of each replica set to obtain a key-value cache check result, wherein the key-value cache check result is used to indicate the consistency between the key-value caches of each replica set.

15. A load balancing device for inference service, characterized in that: include: An information acquisition set, used to acquire load status information of any inference service node in the inference service node set during the process of the inference service node set processing a session request; a service creating unit, configured to create a proxy service corresponding to any inference service node when the load status information indicates that the node pressure of any inference service node is greater than a pressure threshold; A set creation unit, configured to create a replica set of the proxy service according to the node pressure; a request distribution unit, configured to distribute the session request corresponding to any inference service node to each replica in the replica set for processing according to a request distribution strategy, and perform a load balancing operation between the proxy service and each replica; a proxy releasing unit, configured to, if the pressure information corresponding to the replica set meets the pressure requirement, determine a first replica in the replica set, release the replicas in the replica set except the first replica, and replace the proxy service with the first replica; The device is also used to: upload the serialized key-value cache to the centralized storage system after each replica completes processing the target session request and if there is a change in the key-value cache; send synchronization notification information to the replica subset via network broadcast, wherein the synchronization notification information includes relevant information about the change in the key-value cache; when it is determined that each replica has received the synchronization notification information, pull the key-value cache closest to the current time from the centralized storage system and deserialize it according to the session identifier and key-value cache version number in the synchronization notification information to obtain a deserialized key-value cache; when the deserialized key-value cache passes verification, store the deserialized key-value cache and delete the stored key-value cache.

16. An electronic device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 14.

17. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 14.

18. A computer program product, characterized in that A computer program is included which, when executed by a processor, implements the method according to any one of claims 1 to 14.

Images