Dialogue inference system and dialogue request processing method based on large language model

Abstract

The invention provides a dialogue inference system based on a large language model and a dialogue request processing method, and relates to the technical field of artificial intelligence, working nodes of the dialogue inference system are in communication connection through a high-speed interconnection link, and on the basis, when a management node receives a dialogue request from a session, the dialogue request is processed through the high-speed interconnection link. According to the load condition of the working node in the system, when the working node storing the historical dialogue data of the session is in a busy state, the dialogue request can be distributed to other working nodes, and the working node receiving the dialogue request can send the dialogue request to the other working nodes through the high-speed interconnection link. The historical dialogue data is quickly acquired from the working node storing the historical dialogue data, and the dialogue request is executed based on the acquired historical dialogue data. Therefore, available storage resources and available computing resources of the system are fully utilized, the historical dialogue data are prevented from being recalculated, the waiting time delay of the dialogue request is reduced, and the reasoning efficiency of the system is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence (AI), in particular to a dialogue reasoning system based on a large language model and a dialogue request processing method. BACKGROUND

[0002] In a large language model reasoning scenario, a session refers to a series of context-related interaction processes between a user and a dialogue reasoning system within a continuous period of time. In a session involving multiple rounds of dialogue, for any round of dialogue, the dialogue reasoning system stores the relevant data of a token in the form of a key-value pair (KV) each time the large language model reasons out a token. In this way, the dialogue reasoning system can reuse the tokens reasoned out in the historical dialogue when a new round of dialogue occurs, thereby accelerating text generation.

[0003] In related technologies, a dialogue reasoning system uses multiple worker nodes to achieve load balancing. For the same session, the token-related data reasoned out in the historical dialogue is stored in the same worker node, such as worker node A. Correspondingly, dialogue requests from the same session are assigned to the worker node A, so that the worker node A can perform dialogue reasoning based on the locally stored historical dialogue data.

[0004] However, as the number of dialogue rounds of a session increases, the token-related data reasoned out in the historical dialogue also increases. Since the storage resources of the worker node A are limited, a part of the token-related data needs to be discarded to release storage space. As a result, when the worker node A receives a new dialogue request, it needs to recalculate the discarded token-related data, resulting in a large computational overhead and increasing the latency of the dialogue request. SUMMARY

[0005] The present application provides a dialogue reasoning system based on a large language model and a dialogue request processing method, which can reduce the latency of dialogue requests while avoiding the recalculation of historical dialogue data.

[0006] In a first aspect, a dialogue reasoning system based on a large language model is provided, which is applied to a multi-round dialogue reasoning scenario based on a large language model LLM. The system includes a management node and multiple worker nodes, and each worker node is connected through a high-speed interconnection link.

[0007] The first worker node is configured to store historical dialogue data of a session, and the historical dialogue data includes data generated by reasoning at least one round of historical dialogue of the session through a large language model.

[0008] The management node is configured to receive a first dialogue request of a session, and send the first dialogue request to a second working node if the first working node is in a busy state, the first dialogue request being a dialogue request of a subsequent dialogue round of at least one historical dialogue round;

[0009] The second working node is configured to receive the first dialogue request, and obtain historical dialogue data stored on the first working node through the high-speed interconnection link; obtain input of the large language model based on the first dialogue request and the historical dialogue data, perform dialogue inference by using the large language model, and obtain dialogue data corresponding to the first dialogue request.

[0010] In the above method, when the management node receives a dialogue request from a session, the dialogue request can be distributed to other working nodes if the working node storing the historical dialogue data of the session is in a busy state according to the load condition of the working nodes in the system. In addition, the working node receiving the dialogue request can quickly obtain the historical dialogue data from the working node storing the historical dialogue data through the high-speed interconnection link, and execute the dialogue request based on the obtained historical dialogue data. In this way, the available storage resources and available computing resources of the system are fully utilized, the historical dialogue data is not recalculated, the latency of the dialogue request is reduced, and the inference efficiency of the system is improved.

[0011] In some embodiments, the memory of the working node includes double data rate memory (DDR) and high bandwidth memory (HBM); the first working node is configured to store the historical dialogue data in the HBM of the first working node, and unload the historical dialogue data to the DDR of the first working node if the HBM memory of the first working node is insufficient.

[0012] In some embodiments, the second working node is configured to store the historical dialogue data and the dialogue data corresponding to the first dialogue request in the HBM of the second working node, and unload the historical dialogue data and the dialogue data corresponding to the first dialogue request to the DDR of the second working node if the HBM memory of the second working node is insufficient.

[0013] In the above manner, in the case where the memory of the working node includes DDR and HBM, the working node can unload data in the HBM to the DDR to expand the storage space. Since the working nodes in the present application are connected by high-speed interconnection links, data stored in the DDR or the HBM of the working node can be transmitted to other working nodes in time for use by the other working nodes.

[0014] In some embodiments, the management node is further configured to determine the second worker node based on load information of the at least one worker node, the load information indicating at least one of a computing load and a storage load of the worker node, the at least one worker node referring to the worker nodes in the plurality of worker nodes except the first worker node.

[0015] In this way, the management node fully considers the load conditions of the worker nodes in the system when determining the second worker node, ensuring that the second worker node has the ability to execute the first dialogue request and improving the inference efficiency of the system.

[0016] In some embodiments, the worker node includes a host and an acceleration chip, and the acceleration chip of the first worker node and the acceleration chip of the second worker node are connected through a high-speed interconnection link.

[0017] In some embodiments, the first worker node is configured to store the historical dialogue data in the acceleration chip of the first worker node, and in the case that the acceleration chip of the first worker node is insufficient, unload the historical dialogue data to the host of the first worker node.

[0018] In some embodiments, if the historical dialogue data is stored in the host of the first worker node, the first worker node is further configured to load the historical dialogue data from the host of the first worker node to the acceleration chip of the first worker node, and transmit the historical dialogue data to the second worker node through the acceleration chip of the first worker node; or,

[0019] If the historical dialogue data is stored in the acceleration chip of the first worker node, the first worker node is further configured to transmit the historical dialogue data to the second worker node through the acceleration chip of the first worker node.

[0020] In this way, the high-speed interconnection link between the worker nodes in the dialogue inference system is utilized, so that the worker nodes can use the acceleration chip to transmit data through the high-speed interconnection link interface according to the demand, and the host memory and the acceleration chip memory of different worker nodes are connected. Therefore, whether the worker node stores data in the host memory or the acceleration chip memory, the data can be transmitted to other worker nodes in time for use by other worker nodes.

[0021] In some embodiments, the second worker node is configured to store the obtained historical dialogue data to the acceleration chip of the second worker node; or store the obtained historical dialogue data to the host of the second worker node.

[0022] In this way, when the second worker node subsequently receives a new dialogue request from the same session, it does not need to recalculate the data generated in the first dialogue inference process.

[0023] In some embodiments, the second working node is further configured to store the dialogue data corresponding to the first dialogue request when the second working node has sufficient memory; or, when the second working node has insufficient memory, transmit the historical dialogue data and the dialogue data corresponding to the first dialogue request to the third working node via a high-speed interconnect link.

[0024] In this way, the high-speed interconnection links between the various working nodes are utilized, enabling the second working node to promptly transmit historical dialogue data and data generated by the current round of dialogue inference to other working nodes when memory is insufficient, without discarding data, thus improving the system resource utilization rate.

[0025] In some embodiments, the first dialogue request instructs dialogue reasoning on the first dialogue of the session, and if the second working node transmits historical dialogue data and the dialogue data corresponding to the first dialogue request to the third working node via a high-speed interconnect link.

[0026] The management node is also used to receive the second dialogue request of the session and send the second dialogue request to the third worker node. The second dialogue request is a dialogue request for the subsequent dialogue round of the first dialogue.

[0027] In this way, after the second working node transmits the historical dialogue data and the data generated by the current round of dialogue inference to the third working node, the subsequent management node can send new dialogue requests for the same session to the third working node, thereby improving the efficiency of dialogue request processing.

[0028] Secondly, a dialogue request processing method is provided, applied to a dialogue reasoning system based on a large language model. This system includes a management node and multiple worker nodes, which are connected via a high-speed interconnect link. The method is executed by the management node and includes:

[0029] Receive the first dialogue request of the session, which includes at least one round of historical dialogue. The first dialogue request is a dialogue request for the subsequent dialogue rounds of at least one round of historical dialogue.

[0030] If the first working node is busy, the first dialogue request will be sent to the second working node. The first working node stores the historical dialogue data of the session, which includes data generated when reasoning about at least one round of historical dialogue through a large language model.

[0031] A notification message is sent to the second working node, instructing the second working node to retrieve the historical dialogue data stored on the first working node via a high-speed interconnect link.

[0032] In some embodiments, the method further includes: determining a second working node based on load information of at least one working node, wherein the load information indicates at least one of the computational load and storage load of the working node, and at least one working node refers to a working node other than the first working node among a plurality of working nodes.

[0033] In some embodiments, the first dialogue request instructs dialogue reasoning on the first dialogue of the session. If the second working node transmits historical dialogue data and the dialogue data corresponding to the first dialogue request to the third working node via a high-speed interconnect link, the method further includes:

[0034] The second dialogue request is received and sent to the third worker node. The second dialogue request is a dialogue request for a subsequent dialogue round of the first dialogue.

[0035] Thirdly, a dialogue request processing method is provided, applied to a dialogue reasoning system based on a large language model. This system includes a management node and multiple worker nodes, which are connected via a high-speed interconnect link. The method includes:

[0036] The management node receives the first dialogue request of the session. If the first working node is busy, it sends the first dialogue request to the second working node. The first working node stores the historical dialogue data of the session. The historical dialogue data includes the data generated when reasoning about at least one round of historical dialogue of the session through the large language model. The first dialogue request is a dialogue request for the subsequent dialogue rounds of at least one round of historical dialogue.

[0037] The second working node receives the first dialogue request and obtains the historical dialogue data stored on the first working node through the high-speed interconnection link; based on the first dialogue request and the historical dialogue data, it obtains the input of the large language model, uses the large language model to perform dialogue reasoning, and obtains the dialogue data corresponding to the first dialogue request.

[0038] In some embodiments, the method further includes:

[0039] The management node determines a second working node based on the load information of at least one working node. The load information indicates at least one of the computing load and storage load of the working node. At least one working node refers to a working node other than the first working node among a plurality of working nodes.

[0040] In some embodiments, the method further includes:

[0041] If the second working node has sufficient memory, it stores the dialogue data corresponding to the first dialogue request; or, if the second working node has insufficient memory, it transmits the historical dialogue data and the dialogue data corresponding to the first dialogue request to the third working node via a high-speed interconnect link.

[0042] In some embodiments, the first dialogue request instructs dialogue reasoning on the first dialogue of the session, and if the second working node transmits historical dialogue data and the dialogue data corresponding to the first dialogue request to the third working node via a high-speed interconnect link; the method further includes:

[0043] The management node receives the second dialogue request of the session and sends it to the third worker node. The second dialogue request is a dialogue request for a subsequent dialogue round of the first dialogue.

[0044] Fourthly, a dialogue request processing apparatus is provided, including at least one functional unit for implementing the functions of a management node in a system as provided in the first aspect or any possible implementation thereof.

[0045] Fifthly, a dialogue request processing apparatus is provided, including at least one functional unit for implementing the functions of a working node in the system as provided in the first aspect or any possible implementation thereof.

[0046] In a sixth aspect, a computing cluster is provided, comprising a management device and multiple computing devices, wherein the management device is used to implement the functions of a management node in the system provided by the first aspect or any possible implementation thereof, and the computing devices are used to implement the functions of a working node in the system provided by the first aspect or any possible implementation thereof.

[0047] Seventhly, this application provides a computer program product that, when run on a computing cluster, enables the computing cluster to perform the functions of the system provided by the first aspect or any possible implementation thereof. The computer program product can be a software installation package, which can be downloaded and executed on the computing cluster when the functions of the aforementioned system are required.

[0048] Eighthly, this application provides a computer-readable storage medium for storing at least one piece of program code, which implements the functions of the system provided by the first aspect or any possible implementation thereof. The storage medium includes, but is not limited to, volatile memory, such as random access memory, and non-volatile memory, such as flash memory, hard disk drive (HDD), and solid-state drive (SSD). Attached Figure Description

[0049] Figure 1This is a schematic diagram of a related technology;

[0050] Figure 2 This is a schematic diagram of an implementation environment provided in an embodiment of this application;

[0051] Figure 3 This is a schematic diagram of the hardware structure of a management device provided in an embodiment of this application;

[0052] Figure 4 This is a schematic diagram of the structure of a computing cluster provided in an embodiment of this application;

[0053] Figure 5 This is a schematic diagram illustrating a dialogue reasoning system implementing multi-turn dialogue reasoning, as provided in an embodiment of this application.

[0054] Figure 6 This is a schematic diagram of the architecture of a dialogue reasoning system provided in an embodiment of this application;

[0055] Figure 7 This is a flowchart of a dialogue request processing method provided in an embodiment of this application;

[0056] Figure 8 This is a schematic diagram illustrating data transmission based on a high-speed interconnect link, provided in an embodiment of this application.

[0057] Figure 9 This is a schematic diagram of a dialogue request processing method provided in an embodiment of this application;

[0058] Figure 10 This is a schematic diagram of the structure of a dialogue request processing device provided in an embodiment of this application;

[0059] Figure 11 This is a schematic diagram of another dialogue request processing device provided in an embodiment of this application. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings. It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, historical dialogue data and dialogue requests involved in this application were obtained under fully authorized conditions.

[0061] To facilitate understanding, the key terms and concepts involved in this application will be explained below.

[0062] Artificial intelligence (AI) models are a class of mathematical algorithm models that use machine learning concepts to solve practical problems. Typically, AI models include a large number of parameters and calculation formulas (or calculation rules).

[0063] Acceleration chips, also known as acceleration cards, acceleration devices, or accelerators, are a type of specialized hardware device or computer system designed to accelerate computation in AI scenarios. In the embodiments of this application, acceleration chips may be, for example, graphics processing units (GPUs), neural network processing units (XPUs), intelligent processing units (IPUs), tensor processing units (TPUs), domain-specific architecture (DSA) chips, and so on, but are not limited to these.

[0064] Large language models (LLMs) are AI language processing models trained on massive amounts of text data. LLMs are typically trained on vast amounts of text data from diverse sources, including internet web pages, books, news articles, academic papers, and social media posts. Illustratively, LLMs are used to build dialogue reasoning systems that can understand user questions and generate accurate answers. Through learning from vast amounts of knowledge, LLMs can answer questions across various domains, including history, science, technology, and culture. For example, in the field of intelligent customer service, dialogue reasoning systems can automatically answer user inquiries, improving the efficiency and quality of customer service.

[0065] In the context of large language model reasoning, a session refers to a series of context-related interactions between a user and a dialogue reasoning system over a continuous period of time. In some scenarios, a session includes multi-turn dialogues, which refer to multiple rounds of interactive communication, or a process consisting of multiple consecutive dialogue rounds. It should be understood that a session typically involves multiple cycles of questioning and answering between the user and the dialogue reasoning system, where each round's answer is generated based on the previous interaction history. For example, in the field of intelligent customer service, the entire communication process between a user and intelligent customer service is a session, and the user's description of the problem may involve multiple rounds of dialogue, which together constitute the entire session.

[0066] A token is the smallest semantic unit represented by a vector. This smallest semantic unit can be a character, word, phrase, or similar entity. A sequence of tokens serves as the input to a large language model. For example, in a large language model reasoning scenario, if the question is "tell a wonderful adventure story," the large language model breaks this question down into multiple tokens, such as "tell," "a," "wonderful," "of," "adventure," and "story." Each token carries specific semantic information and is the basic unit for the large language model to understand the question.

[0067] In multi-turn dialogue reasoning, in order to maintain the semantic consistency of the conversation context and ensure accurate understanding of the input of the current dialogue turn, the large language model will refer to the input and output of the historical dialogue in the same conversation when generating the answer for the current dialogue turn. That is, the input of any turn in a multi-turn dialogue is: the word sequence input in the historical dialogue, the word sequence output in the historical dialogue, and the word sequence input by the user in this turn of dialogue.

[0068] Key-value (KV) caching is crucial because large language model inference involves extensive vector computation. When processing any lexical unit, the large language model generates intermediate data—key and value vectors—which are then used in subsequent inference processes. Therefore, in each round of dialogue, after the large language model infers a lexical unit, the dialogue inference system stores the related data of that lexical unit in key-value (KV) pairs. This allows the dialogue inference system to reuse lexical units inferred from previous dialogues in a new round without recalculation, thus accelerating text generation.

[0069] The application scenarios and implementation environment of this application are described below.

[0070] This application is applied to multi-turn dialogue reasoning scenarios based on Large Language Model (LLM).

[0071] In related technologies, during a single round of dialogue in a multi-turn dialogue reasoning scenario, each time a large language model infers a token, the dialogue reasoning system stores the relevant data of that token in the form of key-value pairs (KV). In this way, in a new round of dialogue, the dialogue reasoning system can reuse the tokens inferred from the previous dialogue, thereby accelerating text generation.

[0072] Indicatively, for reference Figure 1 , Figure 1 This is a schematic diagram of a related technology. For example... Figure 1As shown, the dialogue inference system uses N worker nodes to achieve load balancing, where N is a positive integer. For the same session, the token-related data inferred from its historical dialogues (i.e., the historical dialogue data KV cache) is stored on the same worker node. For example, the historical dialogue data of sessions S1 and S2 are both stored on worker node 1, while the historical dialogue data of session S3 is stored on worker node 2. Based on this, when the management node of the dialogue inference system receives a dialogue request Q2 (question 2) from session S2, since the historical dialogue data of session S2 is stored on worker node 1, the dialogue inference system will assign the dialogue request Q2 to worker node 1 for execution to facilitate inference based on the historical dialogue data of session S2. However, as the number of dialogue rounds increases, the amount of token-related data inferred from the historical dialogues also increases, while the storage resources of a single worker node are limited. Often, it is necessary to discard some token-related data to free up storage space. For example, in the diagram, worker node 1 may discard historical dialogue data of session S2 when storage space is insufficient, such as discarding the key-value pairs (KV) of the tokens corresponding to Q1 (question 1) and A1 (answer 1). As a result, when worker node 1 receives a new dialogue request Q2 from session S2, it needs to recalculate the KV of the discarded tokens, resulting in a large computational overhead. Furthermore, worker node 1 needs to wait for the KV of the discarded tokens to be calculated before it can execute the dialogue reasoning for Q2, which greatly increases the latency of waiting for dialogue requests.

[0073] Based on this, this application provides a dialogue reasoning system based on a large language model, which interconnects the various working nodes in the dialogue reasoning system with high-speed interconnect links. By utilizing the high-speed interconnect links between working nodes, different dialogue requests from the same session can be flexibly scheduled, making full use of the system's available storage and computing resources. While avoiding recalculation of historical dialogue data, the latency of waiting for dialogue requests is reduced, thereby effectively improving the system's reasoning efficiency.

[0074] The following is for reference. Figures 2 to 4 The implementation environment of this application will be described.

[0075] Figure 2 This is a schematic diagram of an implementation environment provided in an embodiment of this application. For example... Figure 2 As shown, the implementation environment includes a client 100 and a dialogue reasoning system 200 based on a large language model (hereinafter referred to as the dialogue reasoning system 200). The client 100 is directly or indirectly connected to the dialogue reasoning system 200 through a wireless network or a wired network.

[0076] Client 100 is used to provide users with services such as intelligent dialogue. Illustratively, a user operates client 100 by inputting a question (e.g., in voice, text, or video format, which is not limited in this application), triggering client 100 to send a dialogue request to dialogue inference system 200. For example, if a user inputs the question "How is the weather today?", client 100 sends a dialogue request to dialogue inference system 200, which instructs the system to infer an answer to the question "How is the weather today?". Client 100 can be deployed on the user's device, such as an application server, desktop computer, laptop, mobile phone, etc., which is not limited in this application.

[0077] The dialogue reasoning system 200 is a distributed computing system with storage and computing capabilities. Illustratively, the dialogue reasoning system 200 receives a dialogue request from the client 100, performs dialogue reasoning based on the question indicated by the dialogue request, runs a large language model to generate an answer to the question, and feeds the answer back to the client 100. The client 100 can create a session based on the user's input question to maintain contextual information, better understand the user's intent in subsequent interactions, and provide coherent responses. For example, in the field of intelligent customer service, when a user begins to communicate with intelligent customer service, the client 100 is triggered to generate the first dialogue request and create a session, denoted as S1 (i.e., session identifier), and sends the dialogue request to the dialogue inference system 200. The dialogue inference system 200 performs dialogue inference based on the dialogue request, generates an answer, and feeds the answer back to the client 100. Then, the user asks further questions based on the answer. The entire series of interactions is contained in session S1 until the user ends the consultation or the session S1 times out and is automatically closed after a period of no interaction. During this process, the dialogue inference system 200 can locate the historical dialogue data of the session indicated by the dialogue request to realize dialogue inference.

[0078] Schematic illustration: The dialogue reasoning system 200 includes multiple nodes, which are interconnected. Each node is a device with both computing and storage capabilities, such as a server or desktop computer, but this application is not limited to these. A node can be an independent physical server, a server cluster consisting of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. A node can also be a virtualized node, such as a virtual machine or container, in which case the virtual machine or container can be deployed on at least one computing device (e.g., a server).

[0079] In this embodiment, the multiple nodes in the dialogue inference system 200 are divided into a management node 201 and multiple worker nodes 202 according to the functions they perform. The worker nodes 202 are connected via high-speed interconnect links. For example, the high-speed interconnect links can be Huawei Cache Coherent System (HCCS), RDMA over Converged Ethernet (RoCE), NVIDIA Link, Compute Express Link (CXL), Universal Chiplet Interconnect Express (UCIe), Cache Coherent Interconnect for Accelerators (CCIX), etc., and this application is not limited to these. Indicatively, the management node 201 receives dialogue requests from the client 100 and distributes the dialogue requests to the worker nodes 202. The worker nodes 202 receive the dialogue requests and run a large language model to implement dialogue inference.

[0080] Furthermore, each working node 202 includes a host and at least one acceleration chip. The host and acceleration chip are connected via a peripheral component interconnect express (PCIe) link, and interact with each other via the PCIe link. Schematic, the host of working node 202 receives a dialogue request sent by management node 201 and controls the acceleration chip to run a large language model to achieve dialogue reasoning. The host is, for example, the host of a computing device, and its memory is, for example, dynamic random access memory (DRAM), double data rate (DDR), etc., but this application is not limited to these. The acceleration chip is, for example, a GPU, XPU, IPU, TPU, DSA chip, etc., and the acceleration chip's memory is, for example, high bandwidth memory (HBM), etc., but this application is not limited to these. Schematic, the acceleration chips of each working node 202 are connected via a high-speed interconnect link.

[0081] In some embodiments, the working node 202 includes multiple acceleration chips, each of which can be connected to each other via a high-speed interconnect link. This allows different acceleration chips to quickly access each other's memory through the high-speed interconnect link, achieving efficient data transfer between different acceleration chips. That is, in the dialogue inference system 200, any two acceleration chips can quickly access each other's memory through a high-speed interconnect link, achieving efficient data transfer between different acceleration chips.

[0082] It should be noted that the above Figure 2 The number of nodes in the client 100 and the dialogue reasoning system 200 shown is only illustrative. The number of nodes in the client 100 and the dialogue reasoning system 200 may be more or less, and this application embodiment does not limit this.

[0083] Furthermore, the aforementioned wireless or wired networks utilize standard communication technologies and / or protocols. These networks are typically Transmission Control Protocol / Internet Protocol (TCP / IP) networks within data center networks, as well as RDMA networks such as RoCE networks and InfiniBand (IB) networks, etc., without limitation. In other embodiments, customized and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0084] The structure of the nodes in the aforementioned dialogue reasoning system 200 is described below.

[0085] This application provides a management device that can be configured as the management node 201 in the aforementioned dialogue reasoning system 200. (See reference...) Figure 3 , Figure 3 This is a schematic diagram of the hardware structure of a management device provided in an embodiment of this application. Figure 3 As shown, the management device 300 includes a memory 301, a processor 302, a communication interface 303, and a bus 304. The memory 301, processor 302, and communication interface 303 are interconnected via the bus 304.

[0086] Memory 301 refers to a device for storing data, which may be a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code having an instruction or data structure form and accessible by a computer, but not limited thereto. Illustratively, memory 301 is used to store at least a piece of program code. When the program code stored in memory 301 is executed by processor 302, processor 302 performs the steps executed by the management node in the following method embodiments.

[0087] The processor 302 may be a network processor (NP), a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or an integrated circuit used to control the execution of the program of the present application. The processor 302 may be a single-core processor or a multi-core processor. The number of processors 302 may be one or more.

[0088] The communication interface 303 uses a transceiver module, such as a transceiver, to enable communication between the management device 300 and other devices or communication networks. For example, data can be acquired through the communication interface 303.

[0089] The memory 301 and the processor 302 can be set separately or integrated together.

[0090] Bus 304 may include a pathway for transmitting information between various components of management device 300 (e.g., memory 301, processor 302, communication interface 303).

[0091] This application also provides a computing cluster, which includes a management device and multiple computing devices. (See reference...) Figure 4 , Figure 4 This is a schematic diagram of the structure of a computing cluster provided in an embodiment of this application. Figure 4 As shown, the computing cluster includes a management device and multiple computing devices. Each computing device includes memory, a processor, a communication interface, an acceleration chip, and a bus. The computing devices are used to implement the steps performed by the worker nodes in the method embodiments described below. In some embodiments, the devices in the computing cluster can be connected to each other via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. For example, the management device and the computing devices are connected via a network. Specifically, they are connected to the network through the communication interfaces in each device.

[0092] Based on the above introduction to the application scenarios and implementation environment of this application, the following section introduces the method for implementing multi-turn dialogue reasoning in the dialogue reasoning system provided by this application.

[0093] For ease of understanding, please refer to the following: Figure 5 and Figure 6 The principles of this application will be introduced.

[0094] Figure 5 This is a schematic diagram illustrating a dialogue reasoning system implementing multi-turn dialogue reasoning, as provided in an embodiment of this application. For example... Figure 5 As shown, the dialogue reasoning system includes a management node and multiple worker nodes. The management node and the N worker nodes (N is a positive integer) are connected via communication links, and the worker nodes are connected to each other via high-speed interconnect links. Each worker node includes a host and an acceleration chip. The number of acceleration chips in a worker node can be one or more, which is not limited in this application. It should be understood that the storage space of the worker nodes can be jointly provided by the host memory and the acceleration chip memory. The figure illustrates this using DDR host memory and HBM acceleration chip memory as an example.

[0095] In this embodiment, both the host and acceleration chip of the worker node are capable of storing historical dialogue data (i.e., KV cache). Indicatively, when the acceleration chip of the worker node runs a large language model to perform dialogue reasoning, it stores the historical dialogue data generated during the reasoning process in the acceleration chip's HBM (Hardware Memory). If the acceleration chip's memory is insufficient, the worker node can offload the historical dialogue data to the host memory to expand storage space.

[0096] Taking sessions S1, S2, and S3 in the diagram as examples, the historical dialogue data for sessions S1 and S2 is stored in the host memory (DDR) of worker node 1 (server 1), while the historical dialogue data for session S3 is stored in the host memory (DDR) of worker node 2 (server 2). In this scenario, this application provides a load balancing mechanism. Because the worker nodes in the dialogue inference system are connected via high-speed interconnect links, when the management node receives a dialogue request from a session, it can allocate the request to an idle worker node based on the load of each worker node in the system, provided the worker node storing the historical dialogue data for that session is busy. Furthermore, the worker node receiving the dialogue request can quickly retrieve the historical dialogue data from the worker node storing the historical dialogue data for that session via the high-speed interconnect link and execute the dialogue request based on the retrieved historical dialogue data. This not only avoids recalculating the historical dialogue data but also reduces the latency of waiting for dialogue requests, effectively improving the system's inference efficiency.

[0097] For example, taking dialogue request Q2 of session S2 in the diagram as an example, when the management node receives a new round of dialogue request Q2 from session S2, it determines, based on the load of each worker node in the system, that worker node 1 is currently busy and worker node 2 is currently idle. Based on this, it assigns dialogue request Q2 to worker node 2 for execution. Worker node 2 receives dialogue request Q2, obtains the historical dialogue data of session S2 (KV of the tokens corresponding to Q1 and A1) from worker node 1 through the high-speed interconnect link, and executes dialogue request Q2 based on the obtained historical dialogue data. (It should be understood that when worker node 1 is currently idle, the management node sends dialogue request Q2 to worker node 1 for execution; this will not be elaborated further here.) In some scenarios, when worker node 2's storage space is insufficient, worker node 2 can also transmit the dialogue data generated from executing dialogue request Q2 to other worker nodes through the high-speed interconnect link to improve the system's resource utilization.

[0098] based on Figure 5 The load balancing mechanism shown below is for reference. Figure 6 This paper introduces the functions of the dialogue reasoning system provided in this application. Figure 6 This is a schematic diagram of the architecture of a dialogue reasoning system provided in an embodiment of this application. Figure 6 As shown, the dialogue inference system provides load balancing function 601 and dialogue request execution function 602. The load balancing function 601 is implemented by the management node of the dialogue inference system, and the dialogue request execution function 602 is implemented by the worker nodes of the dialogue inference system.

[0099] Schematic, the load balancing function 601 provided by the management node includes a receiving module 6011, a load balancing module 6012, and a sending module 6013. The receiving module 6011 receives session requests; the load balancing module 6012 determines the worker node to execute the current session request based on the load status of multiple worker nodes in the system; and the sending module 6013 sends the session request to the worker node determined by the load balancing module 6012. In some embodiments, the load balancing function 601 further includes a storage module 6014 for storing load information of each worker node, etc., which is not limited in this application.

[0100] For any given worker node, the dialogue request execution function 602 provided by that worker node includes a receiving module 6021, an execution module 6022, a sending module 6023, and a storage module 6024. The receiving module 6021 receives dialogue requests sent by the management node and obtains historical dialogue data stored on other worker nodes via a high-speed interconnect link. The execution module 6022 executes the dialogue request, that is, it runs a large language model to infer a round of dialogue and generates a response for that round. The sending module 6023 sends the generated response to the management node and sends locally stored historical dialogue data to other worker nodes. The storage module 6024 stores historical dialogue data and the large language model, etc., which are not limited in this application.

[0101] Furthermore, the functional division of dialogue reasoning systems is not limited to... Figure 6 The content shown can be configured with more functions according to user needs in practical applications. Moreover, the various functions provided by the dialogue reasoning system can be allocated to the host and / or acceleration chip of the node as needed. For example, the host and acceleration chip of the working node can work together to implement the above-mentioned dialogue request execution function 602, etc. This application does not limit this.

[0102] The following is for reference. Figure 7 The illustrated embodiment describes a method for processing dialogue requests in the dialogue reasoning system provided in this application.

[0103] Figure 7 This is a flowchart of a dialogue request processing method provided in an embodiment of this application. Figure 7 As shown, the method is applied to a dialogue reasoning system based on a large language model. The dialogue reasoning system includes a management node and multiple working nodes, which are connected to each other through a high-speed interconnection link. Schematically, the method includes the following steps 701 to 707.

[0104] 701. The management node receives the first dialogue request of the session. The session includes at least one round of historical dialogue. The first dialogue request is a dialogue request for the subsequent dialogue rounds of the at least one round of historical dialogue.

[0105] In this embodiment, the dialogue inference system communicates with the client. The user, by operating the client, inputs a question (e.g., in voice, text, or video format, which is not limited in this application), triggering the client to send a dialogue request to the dialogue inference system. In this embodiment, based on at least one round of historical dialogue in the session, the implementation method of the dialogue inference system for reasoning about a new round of dialogue in the session is described. That is, based on the historical dialogue data of the session already stored in the dialogue inference system, the management node of the dialogue inference system receives the first dialogue request for the session. This first dialogue request is a dialogue request for a subsequent round of dialogue following at least one round of historical dialogue. Schematic, the first dialogue request instructs the inference of the first dialogue of the session using a large language model; the first dialogue is also a subsequent round of dialogue following at least one round of historical dialogue. For example, the first dialogue request includes the user-input question "How's the weather today?", which is the question in the first dialogue. The first dialogue request instructs the inference of the question "How's the weather today?" using a large language model to generate an answer.

[0106] Additionally, the historical dialogue data includes data generated during reasoning of at least one round of historical dialogue using a large language model, specifically the key-value pairs (KV) of lexical tokens generated during this process. For example, before the management node receives the first dialogue request, the dialogue inference system has already performed two rounds of dialogue inference. The data generated during the first round of dialogue inference includes the input lexical sequence Q1 and the output lexical sequence A1 of the large language model, while the data generated during the second round includes the input lexical sequence Q2 and the output lexical sequence A2. Accordingly, the historical dialogue data stored by the dialogue inference system includes Q1, A1, Q2, and A2.

[0107] In some embodiments, within the dialogue inference system, multiple rounds of dialogue within the same session share the same session identifier, and at least one round of historical dialogue is a completed dialogue with the same session identifier as the first dialogue. For example, when a user initiates a dialogue for the first time, the client creates a session and generates a session identifier. Subsequent dialogue requests sent by the client to the dialogue inference system based on this session carry this session identifier, allowing the dialogue inference system to identify the session to which a received dialogue request belongs based on the session identifier in the dialogue request. Accordingly, in this step, the first dialogue request carries the session identifier of the session.

[0108] 702. If the first working node is busy, the management node determines the second working node. The first working node stores the historical dialogue data of the session.

[0109] In this embodiment, the working node storing the historical dialogue data of the session is referred to as the first working node. The first working node can be any one of multiple working nodes. It should be understood that the first working node and the second working node are different working nodes. The first working node being in a busy state includes at least one of the following: the computing resources of the first working node are occupied; the memory of the first working node is insufficient (or the remaining storage space is insufficient).

[0110] Schematic illustration: For any given worker node, its working state includes a busy state and an idle state. The management node can send an inquiry message to the first worker node to determine whether its working state is busy. Alternatively, the first worker node can send a notification message to the management node every first set of time intervals (a preset interval, set based on business requirements, for example, 1 second) to inform it of its working state. This application does not limit the specifics of this method. It should be noted that the method for measuring the working state can be set according to business requirements. For example, a worker node being in a busy state can be defined as the utilization rate of the acceleration chip reaching 80%; or, the memory usage reaching 80%; or, the utilization rate of the acceleration chip reaching 70% and the memory usage reaching 70%, etc. This application does not limit the specifics of this method.

[0111] In some embodiments, the management node stores management information indicating the session to which the dialogue data stored by each worker node in the dialogue inference system belongs. For example, the management information is in list form, including the node identifier of each worker node and the session identifier of the corresponding session. Illustratively, after receiving a first dialogue request, the management node, based on the management information and the session indicated by the first dialogue request, determines the first worker node storing the historical dialogue data of that session. If the first worker node is busy, a second worker node other than the first worker node is determined from among the multiple worker nodes.

[0112] In some embodiments, the second working node refers to a working node in an idle state. Illustratively, the management node determines the second working node by: determining the idle second working node based on the working state of at least one working node in the system. It should be understood that at least one working node refers to any working node in the system other than the first working node. The management node obtains the working state of a working node in the same way as the first working node, and therefore will not be repeated. This process means that the management node can quickly determine the idle second working node based on the working state of nodes in the system. In other scenarios, if multiple working nodes in the system are all in an idle state, the management node can determine the second working node based on the communication distance between the idle working node and the first working node, thereby improving the speed at which the subsequent second working node obtains historical dialogue data from the first working node. This application does not limit this.

[0113] In other embodiments, the second working node refers to a working node whose load meets the criteria. Indicatively, the management node determines the second working node by: the management node determining the second working node based on the load information of at least one working node in the system. It should be understood that at least one working node referred to herein refers to working nodes other than the first working node in the system. For any working node, the load information of that working node indicates at least one of the computing load and storage load. The computing load refers to the computing resource utilization of the working node, such as through CPU utilization, acceleration chip load (e.g., acceleration chip utilization, temperature, etc.), task queue length, computing time, response time, etc., and is not limited thereto. The storage load refers to the storage resource utilization of the working node, such as through storage capacity utilization, memory usage, disk I / O rate, data transfer volume, etc., and is not limited thereto. In some embodiments, the management node sends an inquiry message to at least one working node to obtain the load information of each working node, or each working node sends load information to the management node at a second time interval (a preset time interval, set based on business requirements, for example, the second time interval is 2 seconds), and this application does not limit this. This process involves the management node determining a suitable second worker node based on the load status of the nodes in the system. For example, if worker node A's load information indicates a task queue length of 0 and a storage capacity utilization rate of 50%, while worker node B's load information indicates a task queue length of 0 and a storage capacity utilization rate of 90%, then worker node A is selected as the second worker node. Through this method, the management node fully considers the load status of each worker node in the system when determining the second worker node, ensuring that the second worker node is capable of executing the first dialogue request and improving the system's inference efficiency.

[0114] It should be noted that the above method for determining the second working node is only an example and does not constitute a limitation of this application. In practical applications, the strategy for determining the second working node can be configured according to business needs.

[0115] 703. The management node sends the first dialogue request to the second worker node.

[0116] 704. The second working node receives the first dialogue request.

[0117] 705. The second working node obtains the historical dialogue data stored on the first working node through a high-speed interconnect link.

[0118] In this embodiment, after receiving the first dialogue request, the second working node obtains the historical dialogue data stored on the first working node via a high-speed interconnect link. In some embodiments, the management node carries the node information of the first working node in the first dialogue request, so that the second working node knows that the historical dialogue data is stored on the first working node. Alternatively, the management node sends a notification message to the second working node, instructing the second working node to obtain the historical dialogue data stored on the first working node via the high-speed interconnect link. For example, the notification message includes the node information of the first working node. Alternatively, the second working node locally stores relevant information about the data stored by each working node, and after receiving the first dialogue request, the second working node determines the first working node storing the historical dialogue data of that session based on the session indicated by the first dialogue request. It should be understood that there can be multiple ways for the second working node to know that the first working node stores historical dialogue data; these are merely illustrative examples and do not constitute a limitation of this application. In practical applications, configurations can be made according to business needs.

[0119] Schematic illustration: A second working node sends a data transmission request to a first working node via a high-speed interconnect link. This data transmission request indicates the transmission of historical dialogue data of a session. For example, the data transmission request carries a session identifier so that the first working node can locate the historical dialogue data of that session. The first working node receives the data transmission request and transmits the historical dialogue data to the second working node via the high-speed interconnect link. It should be understood that a working node includes a host and an acceleration chip. The first working node transmitting historical dialogue data to the second working node via the high-speed interconnect link means that the first working node transmits the historical dialogue data to the network interface card (NIC) of the acceleration chip on the second working node via the high-speed interconnect link. It should be noted that the number of acceleration chips in a working node can be one or more. When the first working node includes multiple acceleration chips, the multiple acceleration chips can cooperate to realize the transmission of historical dialogue data to improve data transmission efficiency. Alternatively, a specific acceleration chip (e.g., an acceleration chip currently in an idle state) can realize the transmission of historical dialogue data to save communication resources, etc. This application does not limit this.

[0120] Based on the foregoing description of the implementation environment of this application, the high-speed interconnect link used in this application can be, for example, HCCS, RoCE, NvLink, CXL, UCIe, CCIX, etc., and this application is not limited to these. For example, taking RoCE as an example of a high-speed interconnect link, the second working node sends a data transmission request to the first working node. Based on the data transmission request, the first working node sends historical dialogue data to the second working node through the RoCE connection. The data transmission can be packet-based, streaming, or block-based, depending on the size and nature of the data and the RoCE configuration; this application does not limit this. As another example, taking HCCS as an example of a high-speed interconnect link, the second working node sends a data transmission request to the first working node. Based on the data transmission request, the first working node sends historical dialogue data to the second working node through a data transmission channel (such as a direct memory access DMA channel) between the first and second working nodes. The data can be transmitted as a continuous data stream or divided into appropriately sized data packets for transmission as needed. It should be understood that this is only an example. In practical applications, the transmission of historical dialogue data from the first working node to the second working node can be achieved according to the specific configuration of the high-speed interconnection link. This application does not limit this.

[0121] Furthermore, as described above regarding the worker nodes, both the host and acceleration chip of the worker node are capable of storing historical dialogue data. Accordingly, the first worker node can store historical dialogue data on its acceleration chip or on its host. In some embodiments, if the first worker node's acceleration chip runs out of memory, it can offload the historical dialogue data to its host. This expands the storage space.

[0122] The following describes the process by which a second working node obtains historical dialogue data stored on a first working node via a high-speed interconnect link, taking different locations on the first working node as examples, including any of the following scenarios:

[0123] Scenario 1: Historical dialogue data is stored on the host of the first working node. In this case, the first working node loads the historical dialogue data from its host to its acceleration chip, and then transmits the historical dialogue data to the second working node via the acceleration chip.

[0124] Scenario 2: Historical dialogue data is stored in the acceleration chip of the first working node. In this case, the first working node transmits the historical dialogue data to the second working node through its acceleration chip.

[0125] For the second working node, it can store the acquired historical dialogue data in its own acceleration chip or in its host machine; this application does not limit this choice. For example, if the second working node is idle when acquiring historical dialogue data and can immediately execute the first dialogue request, it can store the historical dialogue data in the acceleration chip for timely use. Alternatively, if the second working node is executing other dialogue requests when acquiring historical dialogue data, it can store the historical dialogue data in the host machine to avoid occupying the acceleration chip's memory, and then load the historical dialogue data into the acceleration chip when executing the first dialogue request later.

[0126] As can be seen from step 705 above, the dialogue request processing method provided in this application utilizes the high-speed interconnection link between various working nodes in the dialogue inference system, enabling the working nodes to transmit data using the network card on the acceleration chip through the high-speed interconnection link interface as needed, connecting the host memory and acceleration chip memory of different working nodes. For example, data can be transmitted from the host memory of working node A to the acceleration chip memory of working node B, or vice versa.

[0127] Indicatively, for reference Figure 8 , Figure 8 This is a schematic diagram illustrating data transmission based on a high-speed interconnect link, as provided in an embodiment of this application.

[0128] like Figure 8 As shown in Figure (a), worker node A and worker node B are connected via a high-speed interconnect link. Worker node A can transfer data from the host memory (DDR) to the accelerator chip memory (HBM) via a PCIe link. Then, the accelerator chip's network interface card (NIC) transfers the data to the accelerator chip memory (HBM) of worker node B via the high-speed interconnect link. This achieves interconnection between the host memory and the accelerator chip memory of the worker nodes.

[0129] like Figure 8 As shown in Figure (b), worker node A and worker node B are connected via a high-speed interconnect link. Worker node A can transfer data from the host memory (DDR) to the accelerator chip memory (HBM) via a PCIe link. Then, the accelerator chip's network card transfers the data to the accelerator chip memory (HBM) of worker node B via the high-speed interconnect link. Finally, the accelerator chip of worker node B offloads the data to the host memory (DDR). This achieves interconnection between host memory and host memory of worker nodes.

[0130] The above Figure 8 The example shown is for illustration only. The memory of the acceleration chip between working nodes can also be interconnected, which will not be elaborated here.

[0131] 706. The second working node obtains the input of the large language model based on the first dialogue request and historical dialogue data, and uses the large language model to perform dialogue reasoning to obtain the dialogue data corresponding to the first dialogue request.

[0132] This step is also the process by which the second working node executes the first dialogue request. The dialogue data corresponding to the first dialogue request includes the response to the first dialogue. Indicatively, the second working node sends the response back to the management node, which then sends the response back to the user. In some embodiments, the dialogue data corresponding to the first dialogue request also includes the key-value pairs (KV) of the tokens generated during dialogue inference.

[0133] 707. The second working node stores the dialogue data corresponding to the first dialogue request.

[0134] In this embodiment, the second working node stores the dialogue data corresponding to the first dialogue request in its own memory, such as the memory of the accelerator chip or the host memory; this application does not limit the specific storage location. In this way, when the second working node subsequently receives a new dialogue request from the same session, it does not need to recalculate the data generated during the first dialogue inference process. Illustratively, the second working node stores the dialogue data corresponding to the first dialogue request in the accelerator chip's memory; if the accelerator chip's memory is insufficient, the dialogue data corresponding to the first dialogue request is offloaded to the host memory.

[0135] In some embodiments, when the second worker node lacks memory, it transmits historical dialogue data and the dialogue data corresponding to the first dialogue request to the third worker node via a high-speed interconnect link. That is, if the second worker node executes the first dialogue request and its local memory (including host memory and accelerator chip memory) is insufficient, it can transmit historical dialogue data and the data generated by the current round of dialogue inference to other worker nodes via a high-speed interconnect link, without discarding data, thus improving system resource utilization. The third worker node is, for example, a worker node with sufficient storage space, meaning it has enough remaining memory to store the data from the second worker node. Based on this, the management node can receive the second dialogue request of the session and send it to the third worker node. The second dialogue request is a dialogue request for a subsequent round of the first dialogue. Illustratively, the second dialogue request indicates that the second dialogue of the session is inferred using a large language model. That is, after the second worker node transmits the historical dialogue data and the data generated by the current round of dialogue inference to the third worker node, the subsequent management node can send new dialogue requests for the same session to the third worker node, improving the efficiency of dialogue request processing. Of course, if the third working node is busy, the new dialogue request can be sent to the fourth working node for execution according to the same process as steps 702 to 707 above, which will not be elaborated here.

[0136] The following is for reference. Figure 9 The dialogue request processing method shown in steps 701 to 707 above will be described by way of example. Figure 9 This is a schematic diagram of a dialogue request processing method provided in an embodiment of this application. Figure 9 As shown, this dialog request processing method includes the following steps:

[0137] Step 1: The management node receives a dialogue request from session S1. The dialogue request includes a session identifier and a question, instructing the question to generate an answer by reasoning through a large language model.

[0138] Step 2: The management node determines the working node that stores the historical dialogue data of session S1 as the first working node based on the session identifier indicated by the dialogue request.

[0139] Step 3: The management node determines the working status of the first working node. If the first working node is busy, proceed to step 3-A below; if the first working node is idle, proceed to step 3-B below.

[0140] Step 3-A: The management node sends the dialogue request to the second working node, which is in an idle state. The second working node receives the dialogue request and obtains the historical dialogue data of session S1 from the first working node through the high-speed interconnect link. After loading the dialogue request and historical dialogue data into the acceleration chip, the second working node runs the large language model to realize dialogue reasoning.

[0141] Step 3-B: The management node sends the dialogue request to the first working node. The first working node receives the dialogue request, loads the historical dialogue data from the host memory to the acceleration chip memory, and runs the large language model to realize dialogue reasoning.

[0142] Steps 2 and 3 above are the process by which the dialogue reasoning system achieves load balancing through the management node. That is, the management node flexibly schedules the dialogue requests of the session according to the load of each working node in the system, making full use of the available storage and computing resources of the system, and effectively improving the reasoning efficiency of the system while avoiding recalculation of historical dialogue data.

[0143] Step 4: For the working node that executes the dialogue request, the working node generates the key-value pairs of the input and output sequences of the large language model during the inference process. If the working node has sufficient local memory, the dialogue data generated when executing the dialogue request is stored in the working node. If the working node does not have sufficient local memory, the historical dialogue data and the dialogue data generated when executing the dialogue request are transferred to other working nodes, such as the third working node.

[0144] In summary, this application provides a dialogue request processing method applied to a dialogue inference system based on a large language model. Specifically, it involves a load balancing mechanism. Since the various worker nodes in the dialogue inference system are connected via high-speed interconnect links, when the management node receives a dialogue request from a session, it can allocate the request to an idle worker node based on the load of each worker node in the system. If the worker node storing the historical dialogue data for that session is busy, the request will be assigned to it. Furthermore, the worker node receiving the dialogue request can quickly retrieve the historical dialogue data from the worker node storing the historical dialogue data for that session via the high-speed interconnect link and execute the dialogue request based on the retrieved historical dialogue data. This not only avoids recalculating historical dialogue data but also reduces the latency of dialogue request waiting, effectively improving the system's inference efficiency.

[0145] Based on the above Figure 7 As illustrated in the method embodiment, this application also provides a dialogue request processing apparatus for implementing the functions of the management node in the aforementioned dialogue reasoning system. (Illustratively, refer to...) Figure 10 , Figure 10 This is a schematic diagram of the structure of a dialogue request processing device provided in an embodiment of this application, as shown below. Figure 10 As shown, the device includes a receiving unit 1001 and a transmitting unit 1002.

[0146] The receiving unit 1001 is used to receive the first dialogue request of the session, the session includes at least one round of historical dialogue, and the first dialogue request is the dialogue request of the subsequent dialogue round of at least one round of historical dialogue.

[0147] The sending unit 1002 is used to send the first dialogue request to the second working node if the first working node is in a busy state. The first working node stores the historical dialogue data of the session, which includes data generated when reasoning about at least one round of historical dialogue through a large language model.

[0148] The sending unit 1002 is also configured to: send a notification message to the second working node, the notification message instructing the second working node to obtain the historical dialogue data stored on the first working node through the high-speed interconnection link.

[0149] In some embodiments, the apparatus further includes a determining unit, configured to:

[0150] Based on the load information of at least one working node, a second working node is determined. The load information indicates at least one of the computing load and storage load of the working node. At least one working node refers to a working node other than the first working node among a plurality of working nodes.

[0151] In some embodiments, the first dialogue request instructs dialogue reasoning on the first dialogue of the session. If the second working node transmits historical dialogue data and the dialogue data corresponding to the first dialogue request to the third working node via a high-speed interconnect link, the sending unit 1002 is further configured to:

[0152] The second dialogue request is received and sent to the third worker node. The second dialogue request is a dialogue request for a subsequent dialogue round of the first dialogue.

[0153] Based on the above Figure 7 As illustrated in the method embodiment, this application also provides another dialogue request processing apparatus for implementing the functions of the working nodes in the above-described dialogue reasoning system. (Illustratively, refer to...) Figure 11 , Figure 11 This is a schematic diagram of another dialogue request processing device provided in an embodiment of this application, such as... Figure 11 As shown, the device includes a receiving unit 1101, an acquisition unit 1102, and an inference unit 1103.

[0154] The receiving unit 1101 is used to receive the first dialogue request of the session, the session includes at least one round of historical dialogue, and the first dialogue request is a dialogue request for the subsequent dialogue round of at least one round of historical dialogue.

[0155] The acquisition unit 1102 is used to acquire the historical dialogue data of the session stored on the first working node through a high-speed interconnection link. The historical dialogue data includes data generated when reasoning about at least one round of historical dialogue through a large language model.

[0156] The reasoning unit 1103 is used to obtain the input of the large language model based on the first dialogue request and historical dialogue data, and to perform dialogue reasoning using the large language model to obtain the dialogue data corresponding to the first dialogue request.

[0157] In some embodiments, the device further includes a storage unit for:

[0158] If memory is sufficient, store the dialogue data corresponding to the first dialogue request; or, if memory is insufficient, transmit the historical dialogue data and the dialogue data corresponding to the first dialogue request to the third working node via a high-speed interconnect link.

[0159] In some embodiments, the memory of the working node includes double-speed memory (DDR) and high-bandwidth memory (HBM); a storage unit is used to store historical dialogue data and dialogue data corresponding to the first dialogue request in the HBM, and, in the event that the HBM memory is insufficient, to offload the historical dialogue data and dialogue data corresponding to the first dialogue request to the DDR.

[0160] The aforementioned device fully utilizes the system's available storage and computing resources, avoids recalculating historical dialogue data, reduces latency during dialogue request waiting, and improves the system's reasoning efficiency.

[0161] Of course, the above Figure 10 or Figure 11 The apparatus shown can also include other functional units to implement the functions involved in the management node or working node in the above method embodiments. In practical applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the apparatus can be divided into different functional units to complete all or part of the functions described above. In addition, the apparatus provided in the above embodiments and the dialogue request processing method embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0162] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with substantially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor does it limit the quantity or execution order. It should also be understood that although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms. These terms are merely used to distinguish one element from another. For example, without departing from the various examples described, a first working node can be referred to as a second working node, and similarly, a second working node can be referred to as a first working node. Both a first working node and a second working node can be working nodes, and in some cases, they can be separate and distinct working nodes.

[0163] In this application, the term "at least one" means one or more, and the term "multiple" means two or more. For example, multiple working nodes means two or more working nodes.

[0164] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0165] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of program structure information. This program structure information includes one or more program instructions. When these program instructions are loaded and executed on a computing device, the processes or functions according to the embodiments of this application are generated, in whole or in part.

[0166] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0167] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A dialogue reasoning system based on a large language model, characterized in that, The system includes a management node and multiple working nodes, which are connected to each other via a high-speed interconnect link. The first working node is used to store the historical dialogue data of the session, which includes data generated when reasoning about at least one round of historical dialogue of the session through a large language model. The management node is used to receive the first dialogue request of the session. If the first working node is busy, it sends the first dialogue request to the second working node. The first dialogue request is a dialogue request for the subsequent dialogue round of the at least one round of historical dialogue. The second working node is used to receive the first dialogue request and to obtain the historical dialogue data stored on the first working node through the high-speed interconnection link; The input to the large language model is obtained based on the first dialogue request and the historical dialogue data. The large language model is then used to perform dialogue reasoning to obtain the dialogue data corresponding to the first dialogue request.

2. The system according to claim 1, characterized in that, The memory of the working node includes double-speed memory (DDR) and high-bandwidth memory (HBM); The first working node is configured to store the historical dialogue data in its HBM and, in the event that the HBM memory of the first working node is insufficient, to offload the historical dialogue data to its DDR.

3. The system according to claim 2, characterized in that, The second working node is configured to store the historical dialogue data and the dialogue data corresponding to the first dialogue request in the HBM of the second working node, and, in the event that the HBM memory of the second working node is insufficient, to offload the historical dialogue data and the dialogue data corresponding to the first dialogue request to the DDR of the second working node.

4. The system according to any one of claims 1 to 3, characterized in that, The management node is further configured to determine the second working node based on the load information of at least one working node, wherein the load information indicates at least one of the computing load and storage load of the working node, and the at least one working node refers to a working node other than the first working node among the plurality of working nodes.

5. The system according to any one of claims 1 to 4, characterized in that, The working node includes a host and an acceleration chip, and the acceleration chip of the first working node and the acceleration chip of the second working node are connected to each other through the high-speed interconnect link.

6. The system according to claim 5, characterized in that, The first working node is configured to store the historical dialogue data in its acceleration chip and, in the event of insufficient memory in the acceleration chip of the first working node, to offload the historical dialogue data to the host of the first working node.

7. The system according to claim 6, characterized in that, If the historical dialogue data is stored in the host of the first working node, the first working node is further configured to load the historical dialogue data from the host of the first working node to the acceleration chip of the first working node, and transmit the historical dialogue data to the second working node through the acceleration chip of the first working node. or, If the historical dialogue data is stored in the acceleration chip of the first working node, the first working node is further configured to transmit the historical dialogue data to the second working node through the acceleration chip of the first working node.

8. The system according to any one of claims 5 to 7, characterized in that, The second working node is used to store the acquired historical dialogue data in the acceleration chip of the second working node; or, to store the acquired historical dialogue data in the host of the second working node.

9. The system according to any one of claims 1 to 8, characterized in that, The second working node is further configured to store the dialogue data corresponding to the first dialogue request when the second working node has sufficient memory; or, when the second working node has insufficient memory, transmit the historical dialogue data and the dialogue data corresponding to the first dialogue request to the third working node through the high-speed interconnection link.

10. The system according to claim 9, characterized in that, The first dialogue request indicates dialogue reasoning for the first dialogue of the session. If the second working node transmits the historical dialogue data and the dialogue data corresponding to the first dialogue request to the third working node through the high-speed interconnection link; The management node is also configured to receive a second dialogue request of the session and send the second dialogue request to the third working node, wherein the second dialogue request is a dialogue request for a subsequent dialogue round of the first dialogue.

11. A method for processing dialogue requests, characterized in that, An application is made to a dialogue reasoning system based on a large language model. The system includes a management node and multiple worker nodes, which are interconnected via high-speed interconnect links. The method is executed by the management node and includes: Receive a first dialogue request for a session, the session including at least one round of historical dialogue, the first dialogue request being a dialogue request for a subsequent round of the at least one round of historical dialogue; If the first working node is busy, the first dialogue request is sent to the second working node. The first working node stores the historical dialogue data of the session, which includes data generated when reasoning about the at least one round of historical dialogue through a large language model. A notification message is sent to the second working node, the notification message instructing the second working node to obtain the historical dialogue data stored on the first working node through the high-speed interconnect link.

12. The method according to claim 11, characterized in that, The method further includes: The second working node is determined based on the load information of at least one working node, wherein the load information indicates at least one of the computing load and storage load of the working node, and the at least one working node refers to a working node other than the first working node among the plurality of working nodes.

13. The method according to claim 11 or 12, characterized in that, The first dialogue request instructs dialogue reasoning on the first dialogue of the session. If the second working node transmits the historical dialogue data and the dialogue data corresponding to the first dialogue request to the third working node via the high-speed interconnect link, the method further includes: The second dialogue request of the session is received and sent to the third working node. The second dialogue request is a dialogue request for a subsequent dialogue round of the first dialogue.

14. A method for processing dialogue requests, characterized in that, An application is made to a dialogue reasoning system based on a large language model, the system comprising a management node and multiple worker nodes, the worker nodes being interconnected via a high-speed interconnect link, the method comprising: The management node receives the first dialogue request of the session. If the first working node is busy, it sends the first dialogue request to the second working node. The first working node stores the historical dialogue data of the session. The historical dialogue data includes data generated when reasoning about at least one round of historical dialogue of the session through a large language model. The first dialogue request is a dialogue request for the subsequent dialogue rounds of the at least one round of historical dialogue. The second working node receives the first dialogue request and obtains the historical dialogue data stored on the first working node through the high-speed interconnection link; it obtains the input of the large language model based on the first dialogue request and the historical dialogue data, performs dialogue reasoning using the large language model, and obtains the dialogue data corresponding to the first dialogue request.

15. The method according to claim 14, characterized in that, The method further includes: The management node determines the second working node based on the load information of at least one working node, wherein the load information indicates at least one of the computing load and storage load of the working node, and the at least one working node refers to a working node other than the first working node among the plurality of working nodes.

16. The method according to claim 14 or 15, characterized in that, The method further includes: If the second working node has sufficient memory, it stores the dialogue data corresponding to the first dialogue request; or, if the second working node has insufficient memory, it transmits the historical dialogue data and the dialogue data corresponding to the first dialogue request to the third working node through the high-speed interconnect link.

17. The method according to claim 16, characterized in that, The first dialogue request instructs dialogue reasoning on the first dialogue of the session. If the second working node transmits the historical dialogue data and the dialogue data corresponding to the first dialogue request to the third working node via the high-speed interconnect link; the method further includes: The management node receives the second dialogue request of the session and sends the second dialogue request to the third working node. The second dialogue request is a dialogue request for a subsequent dialogue round of the first dialogue.

18. A computing cluster, characterized in that, The computing cluster includes a management device and multiple computing devices. The management device is used to implement the functions of a management node in the system as described in any one of claims 1 to 10. The computing devices are used to implement the functions of a working node in the system as described in any one of claims 1 to 10.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store at least one piece of program code, which, when executed by a computing cluster, enables the computing cluster to perform the functions of the system as described in any one of claims 1 to 10.

20. A computer program product, characterized in that, When the computer program product is run on a computing cluster, the computing cluster enables the system to perform the functions of any one of claims 1 to 10.

Images