Session request processing method, electronic equipment, storage medium and program product

Abstract

The invention provides a session request processing method, electronic equipment, a storage medium and a program product, and belongs to the field of artificial intelligence. The method comprises the steps that after a first session request of a first session is received, a first server corresponding to a first session identifier of the first session is determined, and the first server is a candidate server with the minimum session connection total number when the first-round session request of the first session is received; the total session connection number is used for representing the load capacity of the currently unprocessed sessions of the candidate server; sending the first session request to a first server, and adding the first number to a preset number to obtain a second number; and in the process of processing the first session request, if a second session request of the first round of a second session is received, determining a second server with the least session connection total number according to the second number and the session connection total number corresponding to other candidate servers. According to the invention, the load balance of each server in the server cluster can be ensured, and the performance of the session system is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method for processing a session request, an electronic device, a storage medium, and a program product. Background Art

[0002] Conversational systems are an important application in the field of artificial intelligence. Through technologies such as speech recognition and natural language processing, they enable computers to interact with humans through natural language. They are currently widely used in various scenarios such as conversational robots, virtual assistants, and intelligent customer service.

[0003] In related technologies, after receiving a session request for any session, the session system determines the server cluster corresponding to the model identifier based on the model identifier carried in the session request. It then uses Nginx's hash algorithm to hash the session identifier carried in the session request to obtain a hash value for the session identifier. Based on the hash value, the system randomly selects a load-bearing server from multiple servers and sends the session request to that server for processing. Subsequent session requests carrying that session identifier are sent to that server for processing.

[0004] Although the above method ensures that the context of the same session is cached by the same server, which improves the inference speed and effect of the model, in a multi-session scenario, the load of different servers in the server cluster is unbalanced, and the performance of the session system is poor. Summary of the Invention

[0005] The embodiments of the present application provide a method for processing session requests, an electronic device, a storage medium, and a program product, which can ensure load balancing among servers in a server cluster and improve the performance of the session system. The technical solution is as follows:

[0006] In a first aspect, a method for processing a session request is provided, the method comprising:

[0007] After receiving a first session request of any round of a first session, determining a first server corresponding to a first session identifier of the first session, where the first server is a candidate server having the least total number of session connections among multiple candidate servers when receiving the session request of the first round of the first session, the multiple candidate servers being online servers in a target server cluster, the target server cluster being configured to run a speech conversation model corresponding to a model identifier carried in the session request, and the total number of session connections corresponding to any candidate server being used to represent a load of sessions currently unprocessed by any candidate server;

[0008] Sending the first session request to the first server, and adding a first quantity to a preset quantity to obtain a second quantity, where the first quantity is the total number of session connections corresponding to the first server when the first session request is received, the preset quantity being used to represent the load of the first session in the first round, and the second quantity is the total number of session connections corresponding to the first server when the first session request is started to be processed;

[0009] During the processing of the first session request, if a second session request for the first round of the second session carrying the model identifier is received, a second server with the least total number of session connections is determined based on the second number and the total number of session connections corresponding to other candidate servers, and the second server is used to process the second session request.

[0010] In a second aspect, a device for processing a session request is provided, the device comprising:

[0011] A first determination module is configured to, upon receiving a first session request of any round of a first session, determine a first server corresponding to a first session identifier of the first session, wherein the first server is a candidate server having the fewest total number of session connections among multiple candidate servers when receiving the session request of the first round of the first session, the multiple candidate servers being online servers in a target server cluster, the target server cluster being configured to run a speech conversation model corresponding to a model identifier carried in the session request, and the total number of session connections corresponding to any candidate server being used to represent a load of sessions currently unprocessed by any candidate server;

[0012] A first sending module, configured to send the first session request to the first server;

[0013] an adding module, configured to add a first number and a preset number to obtain a second number, wherein the first number is the total number of session connections corresponding to the first server when the first session request is received, the preset number being used to represent the load of the first session in the first round, and the second number is the total number of session connections corresponding to the first server when the first session request is started to be processed;

[0014] A second determination module is configured to, when processing the first session request, determine, based on the second number and the total number of session connections corresponding to other candidate servers, a second server with the least total number of session connections if a second session request for the first round of the second session carrying the model identifier is received, and to process the second session request.

[0015] In a third aspect, an electronic device is provided, comprising a processor and a memory; the memory stores at least one program code; the at least one program code is used to be called and executed by the processor to implement the method for processing session requests described in the first aspect.

[0016] In a fourth aspect, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, and when the at least one computer program is executed by a processor, the method for processing a session request according to the first aspect can be implemented.

[0017] In a fifth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, can implement the method for processing a session request according to the first aspect.

[0018] The beneficial effects of the technical solution provided by the embodiments of the present application are:

[0019] The method for processing session requests provided in the embodiment of the present application can ensure load balancing of each server in the server cluster and improve the performance of the session system under the premise of ensuring that session requests with the same identifier are processed by the same server. Taking the first session as an example, after receiving the session request of the first round of the first session, the corresponding target service cluster is determined based on the model identifier carried in the session request, and then multiple candidate servers in an online state are determined from the multiple servers included in the target server cluster, and then the total number of session connections corresponding to the multiple candidate servers is obtained. The total number of session connections corresponding to each candidate server is used to characterize the load of the currently unprocessed sessions of each candidate server, and then the candidate server with the least total number of session connections is selected from the multiple candidate servers as the first server. The first server is the candidate server with the smallest load among the multiple candidate servers when the session request of the first round of the first session is received. The first session request of any round of the first session will be sent to the first server for processing. When a first server processes a first session request, its load increases. To accurately characterize the load on the first server, a preset number representing the load of the first session in the first round is added to the first number to obtain a second number. The first number is the total number of session connections corresponding to the first server when the first session request is received, i.e., the load of the sessions processed by the first server before the first session request is fully processed. The second number is the total number of session connections corresponding to the first server when processing the first session request begins, i.e., the load of the sessions processed by the first server when processing the first request. During the processing of the first session request, if a second session request for the first round of a second session carrying a model identifier is received, the second server with the least total number of session connections, i.e., the server with the least current load, is determined based on the second number and the total number of session connections corresponding to other candidate servers. The method provided in the embodiments of the present application not only ensures that session requests from different rounds of the same session are sent to the same server for processing, thus preventing the expiration of session content cached by the server, but also ensures load balancing among candidate servers in the target server cluster, thereby improving the performance of the session system. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] Figure 1 This is an architectural diagram of a dialogue system provided by related technologies;

[0022] Figure 2This is an architectural diagram of a dialogue system provided by an embodiment of the present application;

[0023] Figure 3 This is a block diagram of the entire system for voice conversation provided by an embodiment of the present application;

[0024] Figure 4 This is a flowchart of a method for processing a session request provided by an embodiment of the present application;

[0025] Figure 5 This is a flowchart of another method for processing a session request provided by an embodiment of the present application;

[0026] Figure 6 This is a flowchart of another method for processing a session request provided by an embodiment of the present application;

[0027] Figure 7 This is a schematic diagram of another session request processing process provided by an embodiment of the present application;

[0028] Figure 8 This is a schematic diagram of the structure of a session request processing device provided in an embodiment of the present application;

[0029] Figure 9 A structural block diagram of an electronic device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0030] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0031] It should be understood that the terms "each," "plurality," and "any" used in the embodiments of this application include two or more, "each" refers to each of the corresponding plurality, and "any" refers to any one of the corresponding plurality. For example, if a plurality of words includes 10 words, "each" refers to each of the 10 words, and "any" refers to any one of the 10 words.

[0032] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0033] Before executing the embodiments of the present application, the terms involved in the embodiments of the present application are first explained.

[0034] After receiving input, the Streaming Large Format Model (LLM) doesn't generate a final result all at once. Instead, it gradually generates intermediate results, concatenating the final result. The Streaming Large Format Model can return intermediate results in real time, reducing user reading wait time and the risk of request timeouts.

[0035] Load Balancing distributes requests to multiple servers through algorithms, optimizing resource utilization and avoiding single-point overload.

[0036] Distributed deployment splits system modules into independent services and deploys them on different computing nodes to improve scalability and fault tolerance.

[0037] The consistent hashing algorithm can assign requests to specific servers, reduce cache invalidation, and improve data locality.

[0038] The SLU (Spoken Language Understanding) model is used to understand and parse spoken language input. SLU converts user voice input into semantic information, including intent and slot information, that machines can understand and process. The SLU model plays a crucial role in task-based dialogue systems, helping the system understand user requests and respond accordingly.

[0039] The DP (Dialogue Policy) module is responsible for handling the conversation context, state management, and decision-making logic. Its main function is to determine the system's next action (such as reply content, call external services, jump to a specific process, etc.) based on the user input and the current conversation state.

[0040] The SOP (Standard Operating Procedure) model aims to standardize repetitive tasks through structured, standardized steps, improving efficiency and reducing errors. At its core, SOP breaks down complex tasks into reusable processes, ensuring consistent and controllable execution.

[0041] In the Nginx load balancing configuration, session stickiness based on the client IP address can be implemented so that requests from the same client will be directed to the same server.

[0042] Figure 1 Shows a conversation system in related art, see Figure 1The conversation system includes multiple functional modules, including SOP, DSM, SLU, DP, and SFT (Supervised Fine-Tuning). These functional modules can implement their functions in the form of models. These functional modules are deployed in a coupled manner on a single device and provide voice conversation services by invoking resources such as the GPU and memory. There are multiple GPU models in the conversation system, including GPU1, GPU2, and GPU3. These functional modules are invoked through HTTP domain name requests. Due to the large differences in the requirements of different functional modules for resources such as GPUs and memory, related technologies couple the functional modules for deployment, resulting in insufficient hardware performance.

[0043] In addition, in the related art, after receiving a conversation request, the conversation system determines the server cluster corresponding to the model identifier of the voice conversation model carried in the conversation request, and then uses a random weight value method to determine the server with the least load from the server cluster, and then sends the conversation request to the server with the least load for processing. Although the random weight value method is simple and easy to use, the load of each server in the server cluster is different at different times, resulting in conversation requests for different rounds of the same conversation being sent to different servers, so that the context of the same conversation is cached on different servers. However, the same conversation usually consists of multiple rounds of conversations, each round of conversation is a separate request. Because conversation requests for different rounds of the same conversation are routed to different servers, the caching advantage of the streaming large model is not fully utilized, the cache failure rate is high, resulting in inaccurate output results of the voice conversation model, and reducing the user's voice call experience.

[0044] To fully leverage the caching advantages of large streaming models and avoid cache failure, in related technologies, upon receiving a session request, the session system determines the server cluster corresponding to the model identifier based on the model identifier of the voice session model carried in the session request. The system then uses Nginx's hash algorithm to hash the session identifier carried in the session request, obtaining a hash value for the session identifier. Random routing is then performed based on the hash value. Upon receiving a subsequent session request carrying the session identifier, the system uses the same Nginx hash algorithm to calculate the hash value for the session identifier and sends subsequent session requests carrying the session identifier to the server. While Nginx's hash algorithm ensures that session requests from different rounds of the same session are routed to the same server, in a multi-session concurrent scenario, the initiation time and number of sessions for each session are unpredictable. Some sessions may have more rounds, while others may have fewer. This results in unbalanced loads on different servers in the server cluster, with load differences potentially reaching 3-5 times. Furthermore, as the load increases, the response time of a heavily loaded server to different rounds of the same session may vary, and the response delay difference may reach 16 times, resulting in a poor user experience.

[0045] In order to solve the problem of low resource utilization existing in the related art, an embodiment of the present application provides a session system, which includes multiple functional modules deployed in a distributed manner. Different functional modules are split into independent microservices, and different heterogeneous computing resources are deployed according to the different resource requirements of different microservices. For GPU-intensive functional modules, they can be deployed in a GPU cluster; for memory-intensive functional modules, they can be deployed in a memory cluster. For example, RAG is a memory-intensive functional module, and RAG can be deployed in a memory cluster; SFT is a GPU-intensive functional module, and SFT can be deployed in a GPU cluster. In order to realize the distributed deployment of each functional module, the session system also includes a VIP (Virtual Internet Protocol) server and a load balancing device. The VIP server is used to respond to the session request received by the session system, obtain the IP (Internet Protocol) addresses of at least two load balancing devices, and send the session request to the load balancing device. The load balancing device stores the server identifiers of each server in the server cluster bound to different functional modules, such as the host identifier (including the IP address and port number). After receiving a session request from a VIP server, the load balancing device queries the server cluster corresponding to the functional module based on the identifier of the functional module carried in the session request, and then determines the server in the server cluster to handle the session request. The session system provided in this embodiment fully utilizes hardware performance and improves resource utilization by implementing a distributed deployment of different functional modules.

[0046] based on Figure 2 The illustrated session system and the session request processing method provided by the embodiments of the present application use a distributed cache counter to record the total number of session connections corresponding to each online candidate server in a server cluster. The total number of session connections corresponding to each candidate server is used to represent the load of the currently unprocessed sessions on each candidate server. Taking the first session as an example, after receiving the first session request of any round of the first session, the target server cluster corresponding to the model identifier is determined based on the model identifier carried in the first session request. Then, based on the first session identifier carried in the first session request, the first server corresponding to the first session identifier is determined from multiple online candidate servers in the target server cluster. The first server is the candidate server with the least total number of session connections among the multiple candidate servers when the session request of the first round of the first session is received, that is, the candidate server with the least load among the multiple candidate servers when the session request of the first round of the first session is received. The first session request is sent to the first server, and the first number is added to a preset number to obtain a second number. The first number is the total number of session connections corresponding to the first server when the first session request is received. The preset number is used to represent the load of the first session in the first round. The second number is the total number of session connections corresponding to the first server when processing the first session request begins. During the processing of the first session request, a second session request for the first round of the second session is received. The model identifier carried in the second session request is the same as the model identifier carried in the first session request, indicating that the second session request also needs to be processed by a server in the target server cluster. After receiving the second session request, a second server with the smallest total number of session connections is determined based on the second number and the total number of session connections corresponding to other candidate servers. The second session request is then sent to the second server for processing.

[0047] The method for processing session requests provided in the embodiments of the present application uses a distributed cache counter to achieve load balancing across servers in a server cluster, improving system performance, reducing response delay differences between different rounds of the same session, and minimizing end-to-end latency. The embodiments of the present application support explicit model selection and cache acceleration, enabling session-specific model selection, configurable model information, cacheable model calculations, monitorable model load, and downgradable model routing. The method is suitable for high-concurrency scenarios such as intelligent customer service, AI outbound calls, and voice interaction.

[0048] Figure 3 The diagram shows an architecture diagram of a voice conversation system involved in an embodiment of the present application, which includes multiple clients, multiple outbound call systems, multiple conversation systems, and multiple server clusters.

[0049] The multiple clients include Client 1, Client 2, and Client 3. The clients provide voice conversation services in different scenarios. For example, Client 1 can be used in sales scenarios, Client 2 can be used in intelligent question-and-answer scenarios, and Client 3 can be used in financial scenarios. Each client can initiate a conversation with the outbound call system through an agent.

[0050] Multiple outbound call systems include outbound call system 1, outbound call system 2, and so on. The outbound call system is used to receive sessions initiated by any client and, in response to the client-initiated session, generate a session request. The session request may carry a channel identifier, a session identifier, and a model identifier. The channel identifier is used to select whether to use a streaming session system or a non-streaming session system for the voice conversation. The session identifier is used to distinguish between different sessions. The model identifier is used by the session system to determine the model to process the session request. The channel identifier and model identifier carried in the session request can be default or can be modified by technical personnel. The outbound call system is also used to send the session request to the corresponding session system. For example, if the channel identifier included in the session request corresponds to a non-streaming session system, the session request is sent to the non-streaming session system; if the channel identifier corresponds to a streaming session system, the session request is sent to the streaming session system.

[0051] The multiple session systems include streaming session systems and non-streaming session systems. The session system is used to receive session requests sent by the outbound call system, determine the corresponding server cluster based on the model identifier carried in the session request, select a server from the determined server cluster to process the session request, and then send the session request to the server for processing. The specific architecture of the session system can be found in Figure 2 .

[0052] A server cluster includes multiple servers, each of which is used to process received session requests. The servers in the server cluster can be deployed in a variety of ways, including ECS (Elastic Compute Service) or Pod.

[0053] The present invention provides a method for processing a session request. Figure 2 The session system shown is used as an example to implement the embodiment of the present application. The session system can be a streaming session system or a non-streaming session system. Figure 4 , the method process provided in the embodiment of the present application includes:

[0054] 401. After receiving a first session request of any round of a first session, determine a first server corresponding to a first session identifier of the first session.

[0055] Among them, the first session is any session received by the session system. The first session request includes a channel identifier (channel), a model identifier (model_code) of the voice session model, a first session identifier (session_id), etc. The channel identifier is used to specify whether to use a streaming session system or a non-streaming session system. The server cluster corresponding to the model identifier is the target server cluster, that is, the server cluster that runs the voice session model corresponding to the model identifier. According to the resource requirements of the voice session model corresponding to the model identifier, the type of the target server cluster is different. When the voice session model corresponding to the model identifier is a memory-intensive model, the target server cluster is a memory cluster; when the voice session model corresponding to the model identifier is a graphics processor (GPU)-intensive model, the target server cluster is a GPU cluster. For example, if the voice session model corresponding to the model identifier is an SFT model, the target server cluster is a GPU cluster; if the voice session model corresponding to the model identifier is a RAG model, the target server cluster is a memory cluster. The target server cluster includes multiple servers, and these servers have different states, some are online and some are offline. In the embodiment of the present application, the servers in the online state are referred to as candidate servers. The first session identifier uniquely identifies the first session. For the first session, session requests from different rounds include the same first session identifier. By hashing the first session identifier, session requests from different rounds within the same session are ensured to be routed (i.e., sent) to the same server, preventing the cached first session context from becoming invalid.

[0056] In an embodiment of the present application, the cache counter can maintain the current load status of each server in the server cluster based on the distributed cache middleware technology, such as the online status, the total number of session connections, the load of the GPU and CPU, the response time, etc. Based on the load status of each server in the target service cluster maintained by the cache counter, after receiving the first session request of any round of the first session, the first server corresponding to the first session identifier of the first session can be determined. Among them, the first server is the candidate server with the least total number of session connections among multiple candidate servers when receiving the session request of the first round of the first session. The total number of session connections corresponding to each candidate server is used to characterize the load of the currently unprocessed sessions of each candidate server.

[0057] In the embodiment of the present application, the first session request of any round of the first session may be a session request of the first round of the first session, or a session request of a non-first round of the first session. For different first session requests, when determining the first server corresponding to the first session identifier of the first session, the following two situations may be included but are not limited to:

[0058] In the first case, the first session request is the session request of the first round of the first session

[0059] In this case, after receiving the session request of the first round of the first session, the total number of session connections corresponding to each candidate server in multiple candidate servers can be obtained according to the model identifier carried in the session request of the first round of the first session, and then the candidate server with the least total number of session connections is selected as the first server from the total number of session connections corresponding to the multiple candidate servers.

[0060] Furthermore, after determining the first server corresponding to the first session identifier of the first session, the first session identifier and the first server identifier of the first server are stored in a routing table. The routing table is used to store the correspondence between the session identifiers of the sessions and the server identifiers of the servers processing the sessions, so that upon subsequent receipt of a non-first-round session request for the first session, the first server identifier corresponding to the first session identifier can be queried from the routing table.

[0061] The second case: the second session request is a non-first round session request of the first session

[0062] In this case, after receiving the first session request, the server identifier corresponding to the first session identifier can be queried from the routing table based on the first session identifier. If the server identifier corresponding to the first session identifier is the first server identifier, the server corresponding to the first server identifier is determined as the first server.

[0063] Furthermore, if the corresponding server identifier is not found in the routing table based on the first session identifier, that is, the routing fails, the total number of session connections corresponding to each candidate server in the multiple candidate servers can be obtained from the target server cluster according to the model identifier carried in the first session request, and then the candidate server with the least total number of session connections is selected as the first server from the total number of session connections corresponding to the multiple candidate servers.

[0064] Optionally, in an outbound call scenario, after the conversation system receives a conversation request for the first round of the first conversation sent by the outbound call system, there will be a long instruction processing time before placing a voice call to the user. To significantly reduce the first sentence delay, after receiving the conversation request for the first round of the first conversation, the voice conversation model invoked by the first conversation request can be preheated, and the first server corresponding to the voice conversation model can be determined in advance, thereby shortening the first sentence delay. The model preheating process includes: responding to the conversation request for the first round of the first conversation (since it is a preheating request for the voice conversation model), based on the model identifier carried in the conversation request, determining the target server cluster corresponding to the model identifier, then querying multiple online candidate servers included in the target server cluster, obtaining the total number of conversation connections corresponding to each of the multiple candidate servers, and then selecting the candidate server with the least total number of conversation connections from the total number of conversation connections corresponding to each of the multiple candidate servers as the first server. In fact, for any conversation, the model preheating process can also be regarded as the process of determining the server corresponding to the conversation request after receiving the conversation request for the first round of the conversation. The embodiment of the present application shortens the first sentence response delay by preheating the model before formally initiating a conversation.

[0065] 402. Send the first session request to the first server.

[0066] After the first server is determined, the first session request is sent to the first server for processing.

[0067] 403. Add the first quantity to the preset quantity to obtain a second quantity.

[0068] After the first server begins processing the first session request, the load on the first server increases, necessitating real-time updates to the load on the first server to facilitate load monitoring. Since the total number of session connections corresponding to the first server can represent the load of the first server's currently unprocessed sessions, updating the load on the first server effectively updates the total number of session connections corresponding to the first server. Specifically, a first number can be obtained, a preset number can be estimated, and then the first number and the preset number can be added to obtain a second number. The first number is the total number of session connections corresponding to the first server at the time of receiving the first session request. It represents the load of the first server's currently unprocessed sessions when the first session request is received but not processed. This load does not include the load of the first session request. The preset number represents the load of the first session in the first round. The second number is the total number of session connections corresponding to the first server at the time of starting to process the first session request. It represents the load of the first server's currently unprocessed sessions when the first session request is started. This load includes the load of the first session request.

[0069] Optionally, considering that the voice conversation scenario in the embodiment of the present application is a multi-session concurrent scenario, the conversation system may receive at least two conversation requests simultaneously or within a very short period of time. These two conversation requests may be routed to the same server or to different servers. For the server to which the conversation request is routed, while starting to process the conversation request, it may have completed processing various rounds of conversation requests for another conversation. At this time, the server's load may increase and decrease, and the total number of conversation connections corresponding to the server needs to be updated twice. To prevent the total number of conversation connections on the server from being updated simultaneously for two conversations, the embodiment of the present application may add an optimistic lock to the cache counter recording the total number of conversation connections on the server when any session updates the total number of conversation connections on the server, so that other sessions cannot operate on the cache counter. After the session completes the update of the total number of conversation connections on the server, the optimistic lock is released so that the cache counter can be updated for other sessions. By adopting optimistic locking, the embodiment of the present application ensures the security of multi-session concurrency.

[0070] 404. During the processing of the first session request, if a second session request for the first round of the second session carrying the model identifier is received, a second server with the least total number of session connections is determined based on the second number and the total number of session connections corresponding to other candidate servers. The second server is used to process the second session request.

[0071] The method provided in the embodiment of the present application is applied to a multi-session concurrent scenario. During the processing of a first session request, if a second session request of the first round of a second session is received, it is necessary to determine the second server that processes the second session. The second session request includes a model identifier, a second session identifier of the second session, etc. The model identifier is the same as the model identifier carried in the first session request, and the corresponding server clusters are all target server clusters. When determining the second server, it is necessary to obtain the total number of session connections corresponding to each of the multiple candidate servers in the target server cluster when the second session request is received, and then select the second server with the least total number of session connections from the multiple candidate servers. The multiple candidate servers include the first server and other candidate servers. When the second session request is received, the total number of session connections corresponding to the first server is the second number. Therefore, the second server with the least total number of session connections is selected from the multiple candidate servers, that is, the server with the least total number of session connections is found from the second number and the total number of session connections corresponding to the other candidate servers, and then the server corresponding to the least total number of session connections is determined as the second server.

[0072] Furthermore, after the second server is determined, the second session request may be sent to the second server for processing, thereby ensuring that the second session proceeds smoothly.

[0073] Furthermore, after determining the second server with the fewest total session connections based on the second number and the total number of session connections corresponding to other candidate servers, the second session identifier and the second server identifier of the second session request are stored in the routing table. Upon subsequent receipt of a session request carrying the second session identifier, the routing table is searched to find the second server identifier corresponding to the second session identifier, and the subsequently received session request is then sent to the second server for processing. This ensures that session requests in different rounds of the same session are processed by the same server, avoids cache invalidation, and improves the accuracy of the speech conversation model inference results.

[0074] In another embodiment of the present application, after the first session request is processed, the load on the first server is reduced. At this point, the load on the first server needs to be updated promptly to monitor the load on the first server. Since the total number of session connections corresponding to the first server can represent the load of the first server's currently unprocessed sessions, updating the load on the first server effectively updates the total number of session connections corresponding to the first server. Specifically, a third number can be obtained, and then a preset number can be subtracted from the third number to obtain a fourth number. The third number is the sum of the total number of session connections corresponding to the first server at the time the first session request is processed and the preset number, and can represent the sum of the load of the first server's currently unprocessed sessions at the time the first session request is processed and the load corresponding to the first session request. The fourth number is the total number of session connections corresponding to the first server at the time the first session request is processed, and can represent the load of the first server's currently unprocessed sessions after the first session request is processed. This load does not include the load of the first session request.

[0075] The embodiment of the present application monitors the load of each candidate server by updating the load of each candidate server in real time. In one possible implementation, when the load of the first server is monitored to exceed the preset load, the first server with excessive load can be taken offline. The preset load can be set by a technician based on the maximum load of the first server. When the load of the first server is monitored to exceed the preset load, the weight value of the first server can also be downgraded. The weight value is used to reflect the current load of the first server. When the current load of the first server is large, the weight value of the first server can be reduced to increase the total number of session connections corresponding to the first server, thereby reducing the probability of the first server being selected subsequently; when the current load of the first server is small, the weight value of the first server can be increased to reduce the total number of session connections corresponding to the first server, thereby increasing the probability of the first server being selected subsequently.

[0076] In another embodiment of the present application, after the first session request is sent to the first server, the model identifier and the server identifier of the target server may be sent to a data analysis and monitoring system, allowing the data analysis and monitoring system to perform fine-grained analysis of the target server, such as analyzing the target server's response speed. The server identifier of the target server may be a host identifier, including the target server's IP address and port number. The data analysis and monitoring system may be a system that integrates monitoring and analysis functions, or may be a system consisting of a monitoring subsystem and a data analysis subsystem.

[0077] In another embodiment of the present application, the response time of each server within a preset time period is also detected. The preset time period can be 1 minute, 2 minutes, etc. Taking the first server as an example, if it is detected that the response time of the first server within the preset time period exceeds a threshold, the first server can be marked as unavailable and the unprocessed sessions on the first server can be migrated. The preset threshold can be, for example, the 95th percentile latency within the preset time period exceeding 500 milliseconds. The 95th percentile latency is an important indicator for evaluating the distribution of system response time and is used to measure the reliability of service performance. The 95th percentile latency specifically means that among the response time data of all requests, 95% of the request completion time is less than or equal to this value, and only 5% of the request response time is higher than this value. For example, if the 95th percentile latency of an interface is 200ms, then 95% of user requests can be completed within 200ms, and only 5% of the requests take longer than 200ms.

[0078] The method provided by an embodiment of the present application, upon receiving a session request for the first round of any session, determines the server with the least load to handle the session request based on the load of each candidate server at that time. Even if different servers in the target server cluster handle different rounds of sessions, the server selected each time is the server with the least load. This prevents any candidate server in the target server cluster from being overloaded, ensuring load balancing among the candidate services in the target server cluster. Furthermore, by dynamically monitoring the load of the candidate servers by updating the total number of session connections corresponding to the candidate servers, the embodiment of the present application automatically triggers a circuit breaker when any candidate server is overloaded, thereby improving the overall performance of the system.

[0079] All of the above optional technical solutions can be combined in any way to form optional embodiments of the present application, and will not be described in detail here.

[0080] The present invention provides a method for processing a session request. Figure 2 The session system shown in the embodiment of the present application is used as an example. The session system can be a streaming session system or a non-streaming session system. Figure 5 , the method process provided in the embodiment of the present application includes:

[0081] 501. After receiving a first session request of any round of a first session, determine a first server corresponding to a first session identifier of the first session.

[0082] When implementing this step specifically, reference may be made to the above-mentioned step 401 , which will not be described in detail here.

[0083] 502. Send the first session request to the first server.

[0084] When implementing this step specifically, reference may be made to the above-mentioned step 402 , which will not be described in detail here.

[0085] 503. Determine whether the first server can process the first session request.

[0086] In an embodiment of the present application, an underlying cache service anomaly may occur, resulting in a failure to read the total number of session connections of the first server recorded in the cache counter. In this case, since the load of the first server cannot be monitored in real time, it can be determined that the first server is unable to process the first session request, thereby affecting the normal progress of the first session. To ensure the normal progress of the first session, after sending the first session request to the first server, it is further determined whether the first server can process the first session request, that is, whether the total number of session connections of the first server can be successfully read from the cache counter. If the total number of session connections of the first server is successfully read from the cache counter, it can be determined that the first server can process the first session request, and step 504 is executed. If the total number of session connections of the first server cannot be successfully read from the cache counter, it is determined that the first server cannot process the first session request.

[0087] 504. If the first server is able to process the first session request, add the first number to the preset number to obtain a second number.

[0088] When implementing this step specifically, reference may be made to the above-mentioned step 403 , which will not be described in detail here.

[0089] 505. During processing of the first session request, if a second session request for the first round of the second session carrying the model identifier is received, a second server having the least total number of session connections is determined based on the second number and the total number of session connections corresponding to other candidate servers. The second server is used to process the second session request.

[0090] When implementing this step specifically, reference may be made to the above-mentioned step 404 , which will not be described in detail here.

[0091] In an embodiment of the present application, a load balancing mechanism is implemented based on the total number of session connections of the server recorded by a cache counter, which is called a cache counter load balancing mechanism. The cache counter load balancing mechanism can not only route different session requests of the same session to the same server, but also ensure load balancing of each server in the server cluster.

[0092] The present invention provides a method for processing a session request. Figure 2 The session system shown in the embodiment of the present application is used as an example. The session system can be a streaming session system or a non-streaming session system. Figure 6, the method process provided in the embodiment of the present application includes:

[0093] 601. After receiving a first session request of any round of a first session, determine a first server corresponding to a first session identifier of the first session.

[0094] When implementing this step specifically, reference may be made to the above-mentioned step 401 , which will not be described in detail here.

[0095] 602. Send the first session request to the first server.

[0096] When implementing this step specifically, reference may be made to the above-mentioned step 402 , which will not be described in detail here.

[0097] 603. Determine whether the first server can process the first session request.

[0098] When implementing this step specifically, reference may be made to the above-mentioned step 503 , which will not be described in detail here.

[0099] 604. If the first server cannot process the first session request, and if the first session request includes the first session identifier, determine a third server based on the first session identifier and a first preset algorithm, and send the first session request to the third server.

[0100] When the total number of session connections of the first server cannot be successfully read from the cache counter, it can be determined that the first server cannot process the first session request. In the case that the first server cannot process the first session request, it is determined that the load balancing mechanism of the cache counter has failed, and at this time it can be downgraded to a hash load balancing mechanism; if the hash load balancing mechanism fails, it can be downgraded to a load balancing mechanism with a random weight value. When judging whether the hash load balancing mechanism has failed, it can be checked whether the first session request includes the first session identifier. If the first session request includes the first session identifier, the third server can be determined based on the first session identifier and the first preset algorithm. The first preset algorithm can be the hash algorithm of Ngiinx. When determining the third server based on the first session identifier and the first preset algorithm, the first preset algorithm can be used to perform a hash calculation on the first session identifier to obtain the hash value of the first session identifier, and then select the candidate server with the least bound hash value from the multiple candidate servers included in the target server cluster as the third server.

[0101] Furthermore, after determining the third server, the hash value of the first session identifier is bound to the server identifier of the third server, so that the third server processes the first session request and subsequent session requests for the first session. Subsequently, upon receiving a session request carrying the first session identifier, the subsequent session requests for the first session are sent to the third server for processing.

[0102] 605. When the first server cannot process the first session request and the first session request does not include the first session identifier, use the second preset algorithm to determine a fourth server and send the first session request to the fourth server.

[0103] If the first server cannot process the first session request, the cache counter load balancing mechanism has failed, and the server can be downgraded to a hash load balancing mechanism. If the first session request does not include the first session identifier, the hash load balancing mechanism has failed, and the server can be downgraded to a random weight load balancing mechanism. When using a random weight load balancing mechanism, a second preset algorithm can be used to determine a fourth server, which is the server with the lowest load among the multiple candidate servers when the first session request is received. The second preset algorithm is used to calculate the random weight of the server. The random weight of the server can be calculated based on the server's maximum load and actual load. For example, the actual load of the server can be divided by the server's maximum load to obtain the server's random weight. The random weight of a server refers to the probability that a session request will be routed to the server. A higher random weight indicates a lower probability of the session request being routed to the server. Conversely, a lower random weight indicates a higher probability of the session request being routed to the server.

[0104] The method provided in this application embodiment utilizes a lightweight load-balancing system built on a distributed cache, supporting dynamic load sensing and automatic circuit breaking. Combining explicit models with consistent hashing enables flexible model selection, communication continuity, and automatic degradation. A refined routing hierarchy enables three-level scheduling: model selection, cluster allocation, and instance routing, improving resource utilization.

[0105] Figure 7 The entire processing process of the session request provided by the embodiment of the present application is shown in FIG. Figure 7Client 1 initiates any session through Agent 1. If the channel identifier configured by Client 1 indicates that a non-streaming session system is used for the session, the outbound call system corresponding to the non-streaming session system initiates a session request to the non-streaming session system. This session request includes the channel identifier, global identifier, session identifier, etc. After receiving the session request, the non-streaming session system can determine the corresponding server cluster based on the global identifier, such as server cluster A, server cluster B, and server cluster C. The global identifier can be used to determine the corresponding server cluster A. Then, based on the load balancing mechanism, the server in server cluster A is determined to execute the session request. If the channel identifier configured by Client 1 indicates that a streaming session system is used for the session, the outbound call system corresponding to the streaming session system initiates a session request to the streaming session system. This session request includes the channel identifier, model identifier, session identifier, etc. After receiving the session request, the streaming session system can determine the corresponding server cluster based on the model identifier, such as server cluster A. Then, based on the load balancing mechanism, the server in server cluster A is determined to execute the session request. The load balancing mechanism for streaming session systems and non-streaming session systems is the same. The following will not distinguish between streaming session systems and non-streaming session systems, but will use the session system as an example to introduce the load balancing method.

[0106] If the session request is for the first round, upon receiving it, the load balancing server can identify the online pods in server cluster A and read the total number of session connections (also known as the minimum number of connections) for each pod from the cache counters corresponding to these online pods. If any pod is newly created, a cache counter can be created for the pod before reading the total number of session connections from the cache counter. The total number of session connections for the pod can then be read from the cache counter. After determining the pod with the lowest total number of session connections, the routing table can be updated to store the correspondence between the server ID and session ID of the pod. After the routing table is updated, the session system initiates a session request to the pod. If the session request is not for the first round, upon receiving it, the routing table can be queried. If the pod ID corresponding to the session ID is found in the routing table, the session system initiates a session request to the pod.

[0107] When the session system initiates a session request to the determined pod, if the request is successful, the total number of session connections corresponding to the pod is increased by 1 through the cache counter. After the request ends, an optimistic lock is added to the cache counter, and then the total number of session connections of the pod recorded by the cache counter is reduced by 1. If the request fails, it is downgraded to a hash load balancing solution. Specifically, the session identifier in the session request is hashed to obtain the hash value corresponding to the session identifier, and then the hash value is bound to the corresponding ECS server identifier (including IP address and port number), and then the session request is sent to the ECS for processing. Subsequent session requests carrying the session identifier will be routed to the ECS. If the hash load solution fails, it is downgraded to a random weight value solution, and the session request is routed to the ECS with the smallest load.

[0108] Figure 7 It also includes distributed cache middleware, persistent configuration middleware, distributed log collection middleware, real-time monitoring and alarm middleware and other middleware. Among them, the distributed cache middleware is used to record and store the number of requests for each server. The persistent configuration middleware is used to store the channel identifier, model identifier and model parameters corresponding to each intelligent agent. The distributed log collection middleware is used to collect logs generated during the session request processing. The real-time monitoring and alarm middleware is used to monitor the operation of the entire system, including the server load, system delay, etc., and to issue an immediate alarm when the indicators are abnormal.

[0109] Please refer to Figure 8 , which shows a schematic diagram of the structure of a session request processing device provided in an embodiment of the present application. The device can be implemented by software, hardware, or a combination of both, and becomes all or part of an electronic device. The device includes:

[0110] A first determining module 801 is configured to, upon receiving a first session request of any round of a first session, determine a first server corresponding to a first session identifier of the first session, where the first server is a candidate server having the fewest total number of session connections among multiple candidate servers when receiving the session request of the first round of the first session, the multiple candidate servers being online servers in a target server cluster, the target server cluster being a server cluster running the speech conversation model corresponding to the model identifier, and the total number of session connections corresponding to any candidate server being used to represent a load of sessions currently unprocessed by any candidate server;

[0111] A first sending module 802, configured to send the first session request to the first server;

[0112] an adding module 803 configured to add a first number to a preset number to obtain a second number, where the first number is the total number of session connections corresponding to the first server upon receiving the first session request, the preset number being used to represent the load of the first session in the first round, and the second number is the total number of session connections corresponding to the first server upon starting to process the first session request;

[0113] The second determining module 804 is configured to, when processing the first session request, determine a second server having the least total number of session connections based on the second number and the total number of session connections corresponding to other candidate servers if a second session request for the first round of the second session carrying the model identifier is received. The second server is configured to process the second session request.

[0114] In another embodiment of the present application, when the first session request is a session request for the first round of the first session, the first determination module 801 is configured to, after receiving the session request for the first round of the first session, obtain, based on a model identifier carried in the session request for the first round of the first session, a total number of session connections corresponding to each of the multiple candidate servers, and select, from the total number of session connections corresponding to the multiple candidate servers, a candidate server with the least total number of session connections as the first server.

[0115] In another embodiment of the present application, the device further comprises:

[0116] The first storage module is configured to store the first session identifier and the first server identifier of the first server in a routing table, wherein the routing table is configured to store a correspondence between a session identifier and a server identifier of a server processing the session.

[0117] In another embodiment of the present application, when the first session request is not a session request of the first round of the first session, the first determination module 801 is configured to query the routing table for a first server identifier corresponding to the first session identifier based on the first session identifier; and determine the server corresponding to the first server identifier as the first server.

[0118] In another embodiment of the present application, the device further comprises:

[0119] a determination module, configured to determine whether the first server is capable of processing the first session request;

[0120] An adding module is configured to add the first number to a preset number to obtain a second number when the first server is capable of processing the first session request.

[0121] In another embodiment of the present application, the device further comprises:

[0122] a second determining module, configured to, if the first server cannot process the first session request, determine a third server based on the first session identifier and a first preset algorithm, the third server being a candidate server with the smallest load among the multiple candidate servers when the first session request is received, and the third server being configured to process the first session request and subsequent rounds of session requests for the first session;

[0123] The second sending module is configured to send the first session request to the third server.

[0124] In another embodiment of the present application, the device further comprises:

[0125] a second determining module, configured to determine a third server based on the first session identifier and a first preset algorithm if the first session request includes the first session identifier;

[0126] a third determining module, configured to determine a fourth server using a second preset algorithm if the first session identifier is not included in the first session request;

[0127] The third sending module is configured to send the first session request to the fourth server, where the fourth server is the server with the smallest load among the multiple candidate servers when the first session request is received.

[0128] In another embodiment of the present application, the device further comprises:

[0129] The second storage module is used to store the second session identifier of the second session request and the second server identifier of the second server in a routing table, where the routing table is used to store the correspondence between the session identifier of a session and the server identifier of a server processing the session.

[0130] In another embodiment of the present application, the device further comprises:

[0131] a subtraction module configured to, after the first session request is processed, subtract the preset number from the third number to obtain a fourth number, where the third number is the sum of the fourth number and the preset number, and the fourth number is the number of sessions corresponding to the first server when the first session request is processed.

[0132] In another embodiment of the present application, the device further comprises:

[0133] A fourth sending module is configured to send the model identifier and the server identifier of the target server to a data analysis and monitoring system, so that the data analysis and monitoring system analyzes the first conversation.

[0134] In another embodiment of the present application, the device further comprises:

[0135] an offline module, configured to take the first server offline if the load of the first server exceeds a preset load; or

[0136] The weight reduction module is used to reduce the weight value of the first server if the load of the first server exceeds a preset load, and the weight value is used to reflect the current load of the first server.

[0137] In another embodiment of the present application, the device further comprises:

[0138] a marking module, configured to mark the first server as unavailable if a response time of the first server to the first session request exceeds a preset time;

[0139] A migration module is used to migrate unprocessed sessions on the first server.

[0140] In another embodiment of the present application, when the voice conversation model corresponding to the model identifier is a memory-intensive model, the target server cluster is a memory cluster;

[0141] When the voice conversation model corresponding to the model identifier is a graphics processor (GPU) intensive model, the target service cluster is a GPU cluster.

[0142] Figure 9 FIG. 1 is a block diagram of an electronic device 900 according to an exemplary embodiment of the present application. Generally, the electronic device 900 includes a processor 901 and a memory 902 .

[0143] The processor 901 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 901 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state; the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 901 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 901 may also include an artificial intelligence processor, which is used to process computing operations related to machine learning.

[0144] The memory 902 may include one or more computer-readable storage media, which may be non-transitory computer-readable storage media, such as CD-ROMs (Compact Disc Read-Only Memory), ROMs, RAMs (Random Access Memory), magnetic tapes, floppy disks, and optical data storage devices. The computer-readable storage media may store at least one computer program, which, when executed, implements the aforementioned method for processing session requests.

[0145] Of course, the electronic device described above may also include other components, such as input / output interfaces and communication components. The input / output interface provides an interface between the processor and a peripheral interface module, which may be an output device, an input device, etc. The communication component is configured to facilitate wired or wireless communication between the electronic device and other devices.

[0146] Those skilled in the art will understand that Figure 9 The structure shown in the figure does not constitute a limitation on the electronic device 900, and the electronic device 900 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0147] An embodiment of the present application provides an electronic device, including a processor and a memory; the memory stores at least one program code; the at least one program code is used to be called and executed by the processor to implement the above-mentioned method for processing session requests.

[0148] An embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores at least one computer program. When the at least one computer program is executed by a processor, it can implement the above-mentioned method for processing a session request.

[0149] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0150] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for processing a session request, characterized in that: The method comprises: After receiving a first session request of any round of a first session, determining a first server corresponding to a first session identifier of the first session, where the first server is a candidate server having the fewest total number of session connections among multiple candidate servers when receiving the session request of the first round of the first session, the multiple candidate servers being online servers in a target server cluster, the target server cluster being configured to run a speech conversation model corresponding to a model identifier carried in the session request, and the total number of session connections corresponding to any candidate server being used to represent a load of currently unprocessed sessions on any candidate server; Sending the first session request to the first server, and adding a first quantity to a preset quantity to obtain a second quantity, where the first quantity is the total number of session connections corresponding to the first server when the first session request is received, the preset quantity being used to represent the load of the first session in the first round, and the second quantity is the total number of session connections corresponding to the first server when the first session request is started to be processed; During the processing of the first session request, if a second session request for the first round of the second session carrying the model identifier is received, a second server with the least total number of session connections is determined based on the second number and the total number of session connections corresponding to other candidate servers, and the second server is used to process the second session request.

2. The method according to claim 1, characterized in that In a case where the first session request is a session request of a first round of the first session, determining the first server corresponding to the first session identifier of the first session includes: After receiving the session request of the first round of the first session, obtaining the total number of session connections corresponding to each candidate server in the multiple candidate servers according to the model identifier carried in the session request of the first round of the first session, From the total number of session connections corresponding to the multiple candidate servers, select the candidate server with the least total number of session connections as the first server.

3. The method according to claim 2, characterized in that After determining the first server corresponding to the first session identifier of the first session, the method further includes: The first session identifier and the first server identifier of the first server are stored in a routing table, where the routing table is used to store a correspondence between a session identifier and a server identifier of a server processing the session.

4. The method according to claim 3, characterized in that In a case where the first session request is a session request of a non-first round of the first session, determining the first server corresponding to the first session identifier of the first session includes: Based on the first session identifier, querying the routing table for a first server identifier corresponding to the first session identifier; A server corresponding to the first server identifier is determined as the first server.

5. The method according to claim 1, wherein Before adding the first quantity to the preset quantity to obtain the second quantity, the method further includes: determining whether the first server is capable of processing the first session request; In a case where the first server is capable of processing the first session request, the operation of adding the first number to the preset number to obtain the second number is performed.

6. The method according to claim 5, characterized in that The method further comprises: If the first server cannot process the first session request, determining a third server based on the first session identifier and a first preset algorithm, where the third server is a candidate server with the smallest load among the multiple candidate servers when the first session request is received, and the third server is configured to process the first session request and subsequent rounds of session requests for the first session; Sending the first session request to the third server.

7. The method according to claim 6, characterized in that Before determining the third server based on the first session identifier and the first preset algorithm, the method further includes: If the first session request includes the first session identifier, performing the operation of determining the third server based on the first session identifier and the first preset algorithm; If the first session request does not include the first session identifier, a second preset algorithm is used to determine a fourth server, and the first session request is sent to the fourth server, where the fourth server is the server with the smallest load among the multiple candidate servers when the first session request is received.

8. The method according to claim 1, characterized in that After determining the second server with the least total number of session connections based on the second number and the total number of session connections corresponding to other candidate servers, the method further includes: The second session identifier of the second session request and the second server identifier of the second server are stored in a routing table, where the routing table is used to store a correspondence between session identifiers of sessions and server identifiers of servers processing the sessions.

9. The method according to claim 1, characterized in that The method further comprises: After the first session request is processed, the preset number is subtracted from the third number to obtain a fourth number, where the third number is the sum of the total number of session connections corresponding to the first server when the first session request is processed and the preset number, and the fourth number is the total number of session connections corresponding to the first server when the first session request is processed.

10. The method according to any one of claims 1 to 9, characterized in that After sending the first session request to the first server, the method further includes: The model identifier and the server identifier of the target server are sent to a data analysis and monitoring system, so that the data analysis and monitoring system analyzes the target server.

11. The method according to any one of claims 1 to 9, characterized in that The method further comprises: If the load of the first server exceeds a preset load, the first server is taken offline; or If the load of the first server exceeds a preset load, the weight value of the first server is downgraded, and the weight value is used to reflect the current load of the first server.

12. The method according to any one of claims 1 to 9, characterized in that The method further comprises: If the response time of the first server within a preset time period exceeds a preset threshold, the first server is marked as unavailable, and unprocessed sessions on the first server are migrated.

13. The method according to any one of claims 1 to 9, characterized in that When the voice conversation model corresponding to the model identifier is a memory-intensive model, the target server cluster is a memory cluster; When the voice conversation model corresponding to the model identifier is a graphics processor (GPU) intensive model, the target service cluster is a GPU cluster.

14. An electronic device, characterized in that: The system comprises a processor and a memory; the memory stores at least one program code; the at least one program code is used to be called and executed by the processor to implement the method for processing a session request according to any one of claims 1 to 13.

15. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one computer program, and when the at least one computer program is executed by a processor, it can implement the method for processing a session request according to any one of claims 1 to 13.

16. A computer program product, characterized in that The computer program product includes a computer program, and when the computer program is executed by a processor, the method for processing a session request according to any one of claims 1 to 13 can be implemented.

Images